Probability distribution informed extensions for classical model selection methods: moving beyond the "Rule of 2"

Model selection is a task often undertaken by statisticians and non-statisticians alike. The goal of model selection may be prediction of future outcomes, description of a data-generating process, exploration of a relationship within the data, or some combination thereof. To serve these goals, various methods of performing model selection have been developed in the past.

One such method of model selection involves the use of likelihood-based information criteria such as the Akaike information criterion (AIC), Bayesian information criterion (BIC), and corrected Akaike information criterion (AICc). These criteria help one to identify a model that offers the best balance of conformity to the data and complexity, and can be used in a variety of different ways. A rule of thumb for using likelihood-based information criteria is the “Rule of 2,” which prescribes that one should not discount models that possess a criterion value within 2 units of the minimum criterion value among all candidate models. When applying this rule, one can often achieve good efficacy in model selection.

However, this “Rule of 2” possesses limited theoretical backing. In this thesis, the distributional properties of likelihood-based information criteria are investigated with the goal of producing methods of model selection using likelihood-based information criteria that have sound backing in probability and distribution theory. Using distributional results, a new model selection algorithm to select a model from a collection of candidates is presented, along with a novel goodness-of-fit test for linear models. The efficacy of these methods is demonstrated in simulation studies, and additionally, an example application involving public health data is presented.