Information Criteria

Information criteria are model-comparison tools that penalize model complexity while rewarding goodness of fit.

Why they are useful

They provide a compact way to compare competing models when the question is not simply whether one null hypothesis should be rejected, but which model achieves the best balance between explanatory power and parsimony.

Common examples

• Akaike Information Criterion (AIC)
• Bayesian Information Criterion (BIC)

Different criteria emphasize prediction-vs-parsimony trade-offs differently. AIC is often interpreted as favoring predictive adequacy, while BIC tends to penalize complexity more strongly as sample size increases.

Compact formulas

• AIC = 2k - 2 ln(L)
• BIC = ln(n)k - 2 ln(L)

where k is the number of estimated parameters, L is the likelihood, and n is the sample size.

Interpretation

Information criteria are relative measures. An AIC or BIC value is not meaningful in isolation; what matters is the difference between models fit to the same data. Lower values indicate the preferred model under the assumptions of the criterion.

Relation to hypothesis testing

Information criteria answer a different question from a hypothesis test. Rather than asking whether the data are incompatible with a null model, they compare multiple candidate models and reward simpler explanations when fit is similar.

See also: probabilistic model selection, Hypothesis test, test statistic, Statistical inference, MOC Statistics and Inference