Journal article
The Impact of Operational Scoring Experience and Additional Mentored Training on Raters' Essay Scoring Accuracy
Applied measurement in education, Vol.33(3), pp.210-222
07/02/2020
DOI: 10.1080/08957347.2020.1750404
Abstract
Rater training is essential in ensuring the quality of constructed response scoring. Most of the current knowledge about rater training comes from experimental contexts with an emphasis on short-term effects. Few sources are available for empirical evidence on whether and how raters become more accurate as they gain scoring experiences or what long-term effects training can have. In this study, we addressed this research gap by tracking how the accuracies of new raters change through experience and by examining the impact of an additional training session on their accuracies in scoring calibration and monitoring essays. We found that, on average, raters' accuracy improved with scoring experience and that individual raters differed in their accuracy trajectories. The estimated average effect of the training was an approximately six percent increase in the calibration essay accuracy. On the other hand, we observed a smaller impact on the monitoring essay accuracy. Our follow-up analysis showed that this differential impact of the additional training on the calibration and monitoring essay accuracy could be accounted for by successful gatekeeping through calibration.
Details
- Title: Subtitle
- The Impact of Operational Scoring Experience and Additional Mentored Training on Raters' Essay Scoring Accuracy
- Creators
- Ikkyu Choi - Educational Testing ServiceEdward W. Wolfe - Educational Testing Service
- Resource Type
- Journal article
- Publication Details
- Applied measurement in education, Vol.33(3), pp.210-222
- DOI
- 10.1080/08957347.2020.1750404
- ISSN
- 0895-7347
- eISSN
- 1532-4818
- Publisher
- Taylor & Francis
- Number of pages
- 13
- Language
- English
- Date published
- 07/02/2020
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985123940502771
Metrics
1 Record Views