Articles
Posts Tags Categories Random
Partners
About
About Author Portfolio Terms of Use Privacy Policy Time Travel Donate
More
Settings Funny Tools Useful Websites Student Works Message Board Recent Comments

Contents

Methods for Selecting Models

 Included in Machine Learning Statistical Analysis
 Published   Updated    799 words   4 minutes   views
https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/Methods%20for%20Selecting%20Models/cover%20image.png

Introduction

How can we evaluate and compare multiple models, and select the best one for statistical analysis or machine learning? Simple models may fail to fit the data well, while overly complex models may suffer from “overfitting”, reducing their predictive performance on future data.

Thus, we urgently need a criterion to compare models and strike a balance between goodness of fit, parsimony, and predictive power. To address this, statisticians have proposed various model selection criteria, among which the two most well-known are the Akaike Information Criterion and the Bayesian Information Criterion.

Akaike Information Criterion

The Akaike Information Criterion (AIC) was proposed by Japanese statistician Hirotugu Akaike (赤池弘次) in the 1970s as a general model selection method that balances model fit and simplicity.

AIC is based on the log-likelihood function, with an added penalty term to account for model complexity and prevent overfitting caused by too many parameters. The formula is:

$$ \text{AIC} = -2\ln(L) + 2k, $$

where $\ln L$ is the log-likelihood of the model, $L = f(y \mid \hat{\theta})$ is the likelihood under the maximum likelihood estimate, $y$ is the observed data, $\theta$ is the parameter vector, and $\hat{\theta}$ is its estimate. The penalty term $2k$ is linear in the number of parameters $k$ (the model’s degrees of freedom).

Bayesian Information Criterion

The Bayesian Information Criterion (BIC) was introduced by Gideon E. Schwarz in 1978. It is similar in form to AIC but imposes a stricter penalty on model complexity.

BIC also builds on the log-likelihood function but increases the penalty as the sample size grows. Its formula is:

$$ \text{BIC} = -2\ln(L) + k \ln(n), $$

where $\ln L$ is the log-likelihood, $k$ is the number of model parameters, and $n$ is the sample size. Compared to AIC’s linear penalty $2k$, BIC uses $\ln(n)$ as a multiplier, which emphasizes simplicity more strongly as $n$ increases.

In theory, BIC can be viewed as a Bayesian model comparison method. It assumes the true model is among the candidates and approximates a Bayes factor. Therefore, BIC tends to favor models with fewer parameters and is appropriate when the true model is believed to be among the candidate set.

Compare with AIC and BIC

Both AIC and BIC aim to balance model fit and simplicity. AIC uses a fixed penalty term $2k$, while BIC uses a sample-size-dependent penalty $k\ln(n)$, which increases with larger datasets.

As a result, BIC penalizes complexity more heavily than AIC and is less likely to accept complex models.

When to Use

AIC

  • Assumption: Does not assume the true model is among the candidates.
  • Advantages: Lighter penalty, more tolerant of small samples, captures more variables, better predictive performance. Minimizes average prediction error as sample size grows.
  • Disadvantages: Not consistent, may not choose the true model even if it’s in the candidate set. May overfit with small samples.

BIC

  • Assumption: Assumes the true model is among the candidate models.
  • Advantages: Consistent, selects the true model as the sample size approaches infinity.
  • Disadvantages: Heavier penalty may lead to underfitting with small samples, reducing predictive accuracy.

Summary

CriterionModel PreferencePredictive PowerConsistencySmall Sample BehaviorBest Use Case
AICMore complex modelsHigherNoTends to overfitPrediction, machine learning
BICSimpler modelsMore conservativeYesMay underfitModel fit, variable selection
Tip
For both AIC and BIC, smaller values are better. A lower score indicates a better trade-off between fit and simplicity.

Extended Criterion

Corrected Akaike Information Criterion

The Corrected Akaike Information Criterion (AICc) is an adjusted version of AIC for small sample sizes. When the sample size $n$ is not large relative to the number of parameters $k$, AIC tends to underestimate the penalty for model complexity. AICc corrects this with an additional term:

$$ \begin{align*} \text{AICc} & = \text{AIC} + \frac{2k(k + 1)}{n - k - 1} \\ & = -2\ln(L) + 2k + \frac{2k(k + 1)}{n - k - 1}. \end{align*} $$

When $n \gg k$, AICc is nearly the same as AIC. In small-sample scenarios, AICc effectively reduces the risk of choosing overly complex models. AICc is widely used in regression and time series models.

Conclusion

There are many model selection criteria. This article introduces only the most fundamental and widely used ones. Each criterion evaluates different aspects of a model and may lead to different choices. Therefore, selecting an appropriate model selection criterion based on your specific needs, and finding the best model for your context is a crucial step in the modeling process.

References

Updated on 2025-07-20 16:31 UTC+0800   5bff11bd
Creative Commons License
 Model, Selection, AIC, BIC, Machine Learning, Statistical Analysis, Statistics
Back | Home