Journal article
Evaluating Rater Accuracy in Rater-Mediated Assessments Using an Unfolding Model
Educational and psychological measurement, Vol.76(6), pp.1005-1025
12/01/2016
DOI: 10.1177/0013164415621606
PMCID: PMC5965606
PMID: 29795898
Abstract
The number of performance assessments continues to increase around the world, and it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study describes an unfolding model for examining rater accuracy. Accuracy is defined as the difference between observed and expert ratings. Dichotomous accuracy ratings (0 = inaccurate, 1 = accurate) are unfolded into three latent categories: inaccurate below expert ratings, accurate ratings, and inaccurate above expert ratings. The hyperbolic cosine model (HCM) is used to examine dichotomous accuracy ratings from a statewide writing assessment. This study suggests that HCM is a promising approach for examining rater accuracy, and that the HCM can provide a useful interpretive framework for evaluating the quality of ratings obtained within the context of rater-mediated assessments.
Details
- Title: Subtitle
- Evaluating Rater Accuracy in Rater-Mediated Assessments Using an Unfolding Model
- Creators
- Jue Wang - University of GeorgiaGeorge Engelhard - University of GeorgiaEdward W. Wolfe - Pearson (United States)
- Resource Type
- Journal article
- Publication Details
- Educational and psychological measurement, Vol.76(6), pp.1005-1025
- DOI
- 10.1177/0013164415621606
- PMID
- 29795898
- PMCID
- PMC5965606
- NLM abbreviation
- Educ Psychol Meas
- ISSN
- 0013-1644
- eISSN
- 1552-3888
- Publisher
- Sage
- Number of pages
- 21
- Grant note
- Pearson
- Language
- English
- Date published
- 12/01/2016
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985123698902771
Metrics
2 Record Views