Probabilistic Model Selection

Probabilistic model selection compares candidate models by how well each accounts for the observed data under an explicit probabilistic framework, balancing fit against complexity. The output is a ranking on a continuous scale, not a verdict.

Why the question differs from hypothesis testing

Hypothesis testing asks whether the data is sufficiently incompatible with a null model to reject it. That is a question about one model, and rejecting it says nothing about what to believe instead — a rejected null leaves every alternative equally unaddressed.

Model selection asks which of several plausible models is better supported. This is usually the question people actually have. Nobody genuinely wants to know whether temperature is unrelated to elevation; they want to know whether a linear relationship, a piecewise one, or one including a seasonal interaction describes the data better.

Framing it this way also avoids the trap Amrhein and colleagues describe: a continuous comparison does not naturally collapse into a verdict that then travels without its evidence.

The tools, and their limits

AIC and BIC are the standard criteria, both penalising complexity so that adding parameters must earn its keep. Lower is better for either, and the two answer subtly different questions — AIC targets predictive performance, BIC targets identifying a true model under stronger assumptions — so they can rank models differently and disagreement is informative rather than a problem to resolve.

Three things worth holding onto:

• The criteria are comparative, not absolute. A low AIC means better than the alternatives considered. If every candidate is a poor description of the data, one of them still wins.
• Differences have scale. A gap of one or two is weak evidence; a gap of ten is strong. Ranking without reporting the differences discards most of the information, and it is the most common way this method gets used as though it were a test.
• Comparisons require the same data. Models fitted to different observations, or with different transformations of the response, are not comparable however similar the numbers look.

The failure mode is treating the lowest score as a decision. It is evidence of a particular strength, and reporting it as such is the whole point of preferring this framing.

See also: statistical inference, information criteria, hypothesis testing, and the statistics map.