The impact of item parameter drift on the precision of online calibration in computer adaptive test

With the rapid advancement of technology, the traditional testing format as paper-pencil has gradually been replaced by the computer adaptive format in K-12 education and continuing education. In the setting of computer adaptive testing, each examinee is tested and scored on different test forms which are tailored to the examinee. To immediately release the test score to examinees, it is required to obtain the psychometric statistics of each question prior to the test administration. In addition, newly developed questions which need piloting may be embedded into the adaptive test so that their psychometric statistics could be obtained and used in future tests. Therefore, potential changes/drifts in the statistics of existing questions could lead to inaccurate test scores as well as inaccurate statistics of new questions.

This paper investigated the impact of such drifts on the accuracy of the statistics of new items across five factors. In particular, the study compared two methods (random method vs. D-Tp method) which were used to obtain the statistics of the new questions. The results showed that each method could have its own advantage depending on which type of drifts occurred to the statistics of old questions. If the changes happened to the most difficult questions, the D-Tp method was preferred. Otherwise, the random method was preferred under the other studied conditions. It is recommended for a testing program to be aware of what types of changes are potentially undergoing with the statistics of existing questions and select the appropriate method to obtain the statistics of new items accordingly.