Classification consistency and accuracy indices for simple structure multidimensional item response theory model

In many large-scale testing programs, examinees are frequently categorized into different performance levels. These classifications are then used to make high-stakes decisions about examinees in contexts such as in licensure, certification, and educational assessments. Numerous approaches to estimating the consistency and accuracy of this classification process have been developed under the CTT and UIRT frameworks. However, the multidimensional framework, particularly on the composite theta score metric, remains less explored.

This dissertation was designed to explore the estimation of classification consistency and accuracy indices for composite summed and theta scores within the SS-MIRT framework. To achieve this goal, five prevalent approaches from the UIRT framework have been extended to the SS-MIRT context, including the Lee, Rudner, Guo, Bayesian EAP, and Bayesian MCMC approaches.

Results showed that all five adapted approaches demonstrated commendable performance, with the estimation of classification indices exhibiting significant consistency in magnitude and pattern across different conditions. It was also found that approaches using the MLE estimator generally showed higher ABIAS and RMSE, but lower SE compared to those employing the EAP estimator. Regarding the score metric, approaches applied to the composite summed score metric typically resulted in higher ABIAS, SE, and RMSE than those for the composite theta score metric. For methods to obtain marginal indices, the D and M methods yielded nearly identical outcomes, whereas the P method showed variations. The last interesting finding is that an increase in the correlation between dimensions and test length generally led to a decrease in ABIAS, SE, and RMSE.