Online calibration for pretest items in computerized adaptive testing

Online calibration embeds pretesting into operational computerized adaptive test (CAT) to collect calibration samples for pretest items. Different sampling strategy can be applied in online calibration, such as random sampling design and adaptive sampling design. In this dissertation, the simulation study was conducted to evaluate different designs of online calibration sampling method with regards to pretesting performance of pretest items.

In the study, the initialization strategy as well as the accuracy of initialization in adaptive sampling design were manipulated. In total, five online calibration sampling designs were investigated, including random sampling design, two adaptive sampling with provisional estimates designs, and two adaptive sampling with two-stage designs. The interaction between online calibration sampling design and total sample size was also considered into investigation. Eight different calibration sample sizes, representing small, medium, and large samples, were applied to each sampling design.

Findings showed that random sampling design produced high pretesting completion rate and relatively good absolute error (AB), standard error (SE), root mean square error (RMSE), and average coverage of 95% confidence interval (CI) for item parameter estimates. For adaptive sampling design, overall, the initialization strategy of provisional estimates outperformed two-stage design. Compared to random sampling design, adaptive sampling design tended to yield smaller RMSE when calibration sample sizes were relatively small and when pretest items had very low or very high difficulty. The results of the study help with decisions of practitioners on online calibration design and implementation.