Journal article
Essay Selection Methods for Adaptive Rater Monitoring
Applied psychological measurement, Vol.41(1), pp.60-79
01/01/2017
DOI: 10.1177/0146621616672855
PMID: 29881078
Abstract
Constructed-response items are commonly used in educational and psychological testing, and the answers to those items are typically scored by human raters. In the current rater monitoring processes, validity scoring is used to ensure that the scores assigned by raters do not deviate severely from the standards of rating quality. In this article, an adaptive rater monitoring approach that may potentially improve the efficiency of current rater monitoring practice is proposed. Based on the Rasch partial credit model and known development in multidimensional computerized adaptive testing, two essay selection methods-namely, the D-optimal method and the Single Fisher information method-are proposed. These two methods intend to select the most appropriate essays based on what is already known about a rater's performance. Simulation studies, using a simulated essay bank and a cloned real essay bank, show that the proposed adaptive rater monitoring methods can recover rater parameters with much fewer essay questions. Future challenges and potential solutions are discussed in the end.
Details
- Title: Subtitle
- Essay Selection Methods for Adaptive Rater Monitoring
- Creators
- Chun Wang - University of MinnesotaTian Song - PearsonZhuoran Wang - University of MinnesotaEdward Wolfe - Educational Testing Service
- Resource Type
- Journal article
- Publication Details
- Applied psychological measurement, Vol.41(1), pp.60-79
- DOI
- 10.1177/0146621616672855
- PMID
- 29881078
- NLM abbreviation
- Appl Psychol Meas
- ISSN
- 0146-6216
- eISSN
- 1552-3497
- Publisher
- Sage
- Number of pages
- 20
- Grant note
- Pearson R&D Research Grant
- Language
- English
- Date published
- 01/01/2017
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985123694502771
Metrics
2 Record Views