An investigation into item calibration in multidimensional multistage testing

In recent years, multistage tests (MST) have gained popularity for their adaptive module-level approach, where modules are clusters of items preassembled to fulfill specific content and statistical criteria. In a typical MST, all test-takers receive an identical set of items during the initial routing stage. Their performance at this stage determines which module they proceed to for the second stage, where modules are customized to their interim ability levels, estimated after the routing stage. There is currently a research gap regarding the performance of item calibration methods within the context of multidimensional MST (MMST), where each item measures two or more dimensions simultaneously. This study seeks to address this gap by investigating the performance of three calibration methods (concurrent calibration, fixed item parameter calibration, and concurrent calibration with multiple panels) under an MMST. Additionally, it aims to identify the factors that impact the performance of these methods.

Through computer simulation, the measurement precision achieved by the aforementioned calibration methods across various conditions is assessed. Factors examined included sample size (12,000, 24,000), correlation among dimensions (0.2, 0.5, 0.8), and number of items in the routing stage (8 items, 16 items). Across 36 simulation conditions, each replicated 10 times, it is found that while the calibration methods have minimal impact on item and ability parameter estimates, correlation among dimensions significantly influences parameter estimation. Additionally, sample size and routing stage length have notable effects on estimating item discrimination parameters. These findings could inform researchers and practitioners in the MMST domain, guiding future investigations and enhancing understanding of calibration method performance.