Investigating the effects of missing data treatments on item response theory vertical scaling

The purpose of this study was to examine if different ways of handling missing data affects the item parameter estimates, the construction of a vertical scale, and the subsequent consequences for interpreting students’ growth. An empirical study was conducted with the verbal test and quantitative test of a large-scale ability test for grades 4 through 12.

Five commonly used methods for handling missing responses were investigated: deleting cases with missing responses, scoring them as incorrect (IN), scoring them as if they were not-presented (NP), treating missing responses in the middle of the test as incorrect and at end of the test as not-reached (INNR), and assigning missing responses a partial score (BN). In addition, three multiple imputation methods, a more recent approach for handling missing responses were investigated. The effect of missing data treatments were investigated under conditions when the calibration was conducted separately for each grade as well as when the calibration was conducted concurrently across grades. In addition, the impact of missing data was examined when three proficiency estimators including EAP, MLE, and QD was used to estimate the student’s ability. The vertical scale was evaluated based on student growth, the variability of students’ ability within each grade, and standardized growth. The item parameter estimates were evaluated based on the summary statistics. Lastly, the practical impact was investigated based on how students’ raw scores convert to different scale scores.

The results showed that different ways of handling missing responses affect the resulting item parameter estimates and vertical scales. In general, IN produced items with higher discrimination and difficulty but lower pseudo-guessing compared to other missing data treatments. IN also produced higher mean ability estimates and larger growth. Two of the multiple imputation methods tended to perform similarly to INNP and NP. Missing data had a greater impact when the calibration was done separately for each grade than when the calibration was done concurrently for all grades. Moreover, missing data had a larger effect when the student’s ability was estimated based on MLE than EAP or QD. The impact of missing data was larger on low and high item difficulty estimates than items with middle range difficulty estimates, and yielded differences in students’ scale scores especially for those who were at the extreme ends.