Journal article
Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use
Journal of educational measurement, Vol.46(4), pp.371-389
12/01/2009
DOI: 10.1111/j.1745-3984.2009.00088.x
Abstract
In this study, we describe a framework for monitoring rater performance over time. We present several statistical indices to identify raters whose standards drift and explain how to use those indices operationally. To illustrate the use of the framework, we analyzed rating data from the 2002 Advanced Placement English Literature and Composition examination, employing a multifaceted Rasch approach to determine whether raters exhibited evidence of two types of differential rater functioning over time (i.e., changes in levels of accuracy or scale category use). Some raters showed statistically significant changes in their levels of accuracy as the scoring progressed, while other raters displayed evidence of differential scale category use over time.
Details
- Title: Subtitle
- Monitoring Rater Performance Over Time: A Framework for Detecting Differential Accuracy and Differential Scale Category Use
- Creators
- Carol M. Myford - University of Illinois ChicagoEdward W. Wolfe
- Resource Type
- Journal article
- Publication Details
- Journal of educational measurement, Vol.46(4), pp.371-389
- DOI
- 10.1111/j.1745-3984.2009.00088.x
- ISSN
- 0022-0655
- eISSN
- 1745-3984
- Publisher
- Wiley
- Number of pages
- 19
- Language
- English
- Date published
- 12/01/2009
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985123700002771
Metrics
5 Record Views