Understanding the Confusion Matrix
The cover image is a confusion matrix illustration generated by ChatGPT and modified using Canva. The prompt for ChatGPT was: “A visually engaging illustration of a confusion matrix used in machine learning, with labeled sections for True Positive, False Positive, True Negative, and False Negative. The matrix is displayed on a futuristic digital interface with a blue neon glow. The background features abstract neural network connections and data flow lines, symbolizing artificial intelligence and deep learning. The image is designed with a professional and modern aesthetic.” 。
Introduction
In machine learning, we can build various models, ranging from simple linear regression to complex neural network models. These models can be applied to different fields, such as image recognition, language generation, and financial forecasting. Each model has its advantages and applicable scenarios. For example, linear regression is suitable for fitting and analyzing datasets with target variables, while deep learning can handle complex nonlinear datasets. However, how do we assess the quality of a model? Simply relying on accuracy may not be sufficient to evaluate model performance, especially in cases with imbalanced class distributions. Therefore, we use Confusion Matrix to analyze the performance of classification models.
What is Accuracy?
Accuracy is one of the most intuitive evaluation metrics, simply representing the proportion of correct predictions made by the model relative to the total number of samples. However, relying solely on accuracy can lead to misleading evaluations, especially in cases where the class distribution is highly imbalanced. For example:
Though the accuracy is 99.6%, it seems high, but in reality, only 10 out of the 50 sick people were diagnosed as positive, meaning that $\frac{50 - 10}{50} = \frac{4}{5} = 80\%$ of the sick people were missed, which is quite alarming!
To address the issue of misleading accuracy, we can use a more detailed analysis tool — Confusion Matrix, which helps us with a more thorough evaluation.
What is a Confusion Matrix?
A confusion matrix is a table format evaluation metric that shows the relationship between the predicted results of a classification model and the actual labels. For binary classification problems, the confusion matrix is typically presented as a $2 \times 2$ matrix, as follows:
| Actual Positive | Actual Negative | |
|---|---|---|
| Predicted Positive | True Positive (TP) | False Positive (FP) (Type I Error) |
| Predicted Negative | False Negative (FN) (Type II Error) | True Negative (TN) |
True Positive (TP): The number of times the model correctly predicted a positive result.
False Positive (FP): The number of times the model predicted a positive result for a negative instance.
False Negative (FN): The number of times the model predicted a negative result for a positive instance.
True Negative (TN): The number of times the model correctly predicted a negative result.
Type I and Type II Errors
In statistics and machine learning, we typically focus on two types of errors:
Type I Error
Also known as False Positive (FP), it occurs when the model incorrectly classifies a negative instance as positive. For example, in medical testing, a healthy patient is misdiagnosed as having a disease.
Type II Error
Also known as False Negative (FN), it occurs when the model incorrectly classifies a positive instance as negative. For example, a patient who actually has a disease is misdiagnosed as healthy.
The impact of Type I and Type II errors depends on the application context. For instance, in fraud detection, a Type I error could lead to a legitimate transaction being mistakenly flagged as fraudulent, while a Type II error could result in an actual fraudulent transaction going unnoticed. Generally, Type II errors are considered more severe, especially in medical diagnoses, where missing a disease could lead to worsened conditions.
Derived Metrics from the Confusion Matrix
Through the confusion matrix, we can calculate various evaluation metrics to obtain more information than just accuracy.
| Total population = $P + N$ | Predicted positive | Predicted negative | Informedness, Bookmaker informedness ($BM$) = $TPR + TNR - 1$ | Prevalence threshold ($PT$) = $\frac{\sqrt{TPR \times FPR} - FPR}{TPR - FPR}$ |
|---|---|---|---|---|
| Actual positive ($P$) | True positive ($TP$) | False negative ($FN$) | True positive rate ($TPR$), Recall, Sensitivity ($SEN$) = $\frac{TP}{P}$ = $1 - FNR$ | False negative rate ($FNR$) = $\frac{FN}{P}$ = $1 - TPR$ |
| Actual negative ($N$) | False positive ($FP$) | True negative ($TN$) | False positive rate ($FPR$) = $\frac{FP}{N}$ = $1 - TNR$ | True negative rate ($TNR$) = $\frac{TN}{N}$ = $1 - FPR$ |
| Prevalence = $\frac{P}{P + N}$ | Positive predictive value ($PPV$) = $\frac{TP}{TP + FP}$ = $1 - FDR$ | False omission rate ($FOR$) = $\frac{FN}{TN + FN}$ = $1 - NPV$ | Positive likelihood ratio ($LR+$) = $\frac{TPR}{FPR}$ | Negative likelihood ratio ($LR−$) = $\frac{FNR}{TNR}$ |
| Accuracy ($ACC$) = $\frac{TP + TN}{P + N}$ | False discovery rate ($FDR$) = $\frac{FP}{TP + FP}$ = $1 - PPV$ | Negative predictive value ($NPV$) = $\frac{TN}{TN + FN}$ = $1 - FOR$ | Markedness ($MK$) = $PPV + NPV - 1$ | Diagnostic odds ratio ($DOR$) = $\frac{LR+}{LR−}$ |
| Balanced accuracy ($BA$) = $\frac{TPR + TNR}{2}$ | $F_1$ score = $\frac{2 \times PPV \times TPR}{PPV + TPR}$ = $\frac{2 \times TP}{2 \times TP + FP + FN}$ | Fowlkes–Mallows index ($FM$) = $\sqrt{PPV \times TPR}$ | Matthews correlation coefficient ($MCC$) = $\sqrt{TPR \times TNR \times PPV \times NPV}$ $- \sqrt{FNR \times FPR \times FOR \times FDR}$ | Threat score ($TS$), Critical success index ($CSI$), Jaccard index = $\frac{TP}{TP + FN + FP}$ |
The above is taken from Wikipedia.
The calculations for these metrics are as follows:
Accuracy
$$ Accuracy = \frac{TP + TN}{TP + TN + FP + FN} $$ The confusion matrix can also be used to calculate accuracy, which measures the proportion of overall correct predictions. However, like with accuracy, it can be misleading in imbalanced datasets.
Precision
$$ Precision = \frac{TP}{TP + FP} $$ Also known as Positive Predictive Value (PPV), precision represents the proportion of positive predictions that are actually true positives. It indicates how accurate the model is when it predicts a positive outcome.
Recall
$$ Recall = \frac{TP}{TP + FN} $$ Also known as Sensitivity or True Positive Rate (TPR), recall represents the proportion of actual positives that are correctly identified by the model. It measures how well the model detects true positive instances.
F-Score
$$ F_\beta = (1 + \beta^2) \times \frac{Precision \times Recall}{(\beta^2 \times Precision) + Recall} $$ The F-score, also known as the F-measure, takes both Precision and Recall into account and is often used to evaluate the accuracy of an algorithm. In this formula, $\beta$ is a weight parameter. As $\beta \to 0$, the F-score becomes equivalent to Precision, and as $\beta \to \infty$, the F-score becomes equivalent to Recall.
When $\beta = 1$, it represents a special case of the F-score, where Precision and Recall are weighted equally. In other words, the $F_1$ score is the harmonic mean of Precision and Recall, and it is particularly useful for evaluating models when there is a trade-off between Precision and Recall.
$$ F_1 = 2 \times \frac{Precision \times Recall}{Precision + Recall} = \frac{2TP}{2TP + FP + FN} $$
Calculation process:
Numerator: $$ Precision \times Recall = \left( \frac{TP}{TP + FP} \right) \times \left( \frac{TP}{TP + FN} \right) = \frac{TP^2}{(TP+FP)(TP+FN)} $$
Denominator: $$ Precision + Recall = \left( \frac{TP}{TP + FP} \right) + \left( \frac{TP}{TP + FN} \right) = \frac{2TP^2 + TP \times FN + TP \times FP}{(TP+FP)(TP+FN)} $$
Thus, $$ F_1 = 2 \times \frac{Precision \times Recall}{Precision + Recall} = 2 \times \frac{\frac{TP^2}{(TP+FP)(TP+FN)}}{\frac{2TP^2 + TP \times FN + TP \times FP}{(TP+FP)(TP+FN)}} = \frac{2TP}{2TP + FP + FN} $$
Specificity
$$ Specificity = \frac{TN}{TN + FP} $$ Also known as True Negative Rate (TNR), specificity is the counterpart to Recall. It represents the proportion of actual negatives that are correctly identified by the model.
When to Use a Confusion Matrix?
When dealing with imbalanced classes:
When one class dominates the dataset, accuracy alone may overestimate the performance of the model.When misclassification costs are important:
In applications such as medical diagnoses or fraud detection, the consequences of false positives and false negatives may differ, requiring a more detailed analysis of errors.In multiclass classification problems:
The confusion matrix can be extended to multiclass classification, creating an $N \times N$ matrix to analyze the misclassifications across different classes.
Conclusion
The confusion matrix is a very useful evaluation tool that helps analyze the performance of classification models from multiple perspectives. By calculating derived metrics, we can assess the model’s performance more comprehensively and adjust the model based on specific needs.
References
李航. (2022). 機器學習聖經：最完整的統計學習方法. (First Edition). 深智數位股份有限公司.
Type I and type II errors. (March 12, 2025). Wikipedia, The Free Encyclopedia. Retrieved on March 22, 2025 from https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
F-score. (March 21, 2025). Wikipedia, The Free Encyclopedia. Retrieved on March 22, 2025 from https://en.wikipedia.org/wiki/F-score
Confusion matrix. (February 28, 2025). Wikipedia, The Free Encyclopedia. Retrieved on March 22, 2025 from https://en.wikipedia.org/wiki/Confusion_matrix
Related Posts
