Journal article
A Family of Rater Accuracy Models
Journal of applied measurement, Vol.16(2), pp.153-160
2015
PMID: 26075664
Abstract
Engelhard (1996) proposed a rater accuracy model (RAM) as a means of evaluating rater accuracy in rating data, but very little research exists to determine the efficacy of that model. The RAM requires a transformation of the raw score data to accuracy measures by comparing rater-assigned scores to true scores. Indices computed based on raw scores also exist for measuring rater effects, but these indices ignore deviations of rater-assigned scores from true scores. This paper demonstrates the efficacy of two versions of the RAM (based on dichotomized and polytomized deviations of rater-assigned scores from true scores) to two versions of raw score rater effect models (i.e., a Rasch partial credit model, PCM, and a Rasch rating scale model, RSM). Simulated data are used to demonstrate the efficacy with which these four models detect and differentiate three rater effects: severity, centrality, and inaccuracy. Results indicate that the RAMs are able to detect, but not differentiate, rater severity and inaccuracy, but not rater centrality. The PCM and RSM, on the other hand, are able to both detect and differentiate all three of these rater effects. However, the RSM and PCM do not take into account true scores and may, therefore, be misleading when pervasive trends exist in the rater-assigned data.
Details
- Title: Subtitle
- A Family of Rater Accuracy Models
- Creators
- Edward W. Wolfe - Iowa City, IA, United StatesHong Jiao - Iowa City, IA, United StatesTian Song - Iowa City, IA, United States
- Resource Type
- Journal article
- Publication Details
- Journal of applied measurement, Vol.16(2), pp.153-160
- PMID
- 26075664
- NLM abbreviation
- J Appl Meas
- ISSN
- 1529-7713
- Number of pages
- 8
- Language
- English
- Date published
- 2015
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985123699002771
Metrics
1 Record Views