Item parameterization and ability estimation: considerations for Bayesian multidimensional adaptive testing

Adaptive testing is a popular test format that selects test questions, or items, for an examinee based on how that examinee performed on previous items. There are many design choices that can affect how well an examinee’s ability is measured. This dissertation consists of two articles that investigate adaptive testing design choices when the test is meant to measure multiple abilities.

The first article investigates how to model an item’s difficulty and its capacity to discriminate between high- and low-performing examinees; these statistics assist in determining which items should be administered. Two models are compared, one that requires treatment of examinee answers as either correct or incorrect and one that assumes examinees with different ability levels will be attracted to different item response options. Results indicate improved measurement of examinee abilities can be obtained by using the latter model, but these results may depend on additional design choices.

The second article investigates a Bayesian method of measuring abilities throughout a test. Bayesian methods require the specification of a distribution of plausible ability values; these values are combined with an examinee’s item responses to obtain an estimate of the examinee’s ability. In this study, the variability of this distribution is manipulated so that increasingly larger values are permissible. Results indicate that the degree of variability may have less impact on the measurement of ability than previously suggested.

Findings from these studies showcase the challenges in building accurate adaptive tests for multiple abilities and have implications for how these tests are constructed.