Journal article
When raters disagree, then what: examining a third-rating discrepancy resolution procedure and its utility for identifying unusual patterns of ratings
Journal of applied measurement, Vol.3(3), pp.300-324
2002
PMID: 12147915
Abstract
The purpose of this study was to examine a procedure for identifying and resolving discrepancies in ratings. We sought to determine to what extent the third-rater adjudication procedure employed in scoring the Test of Spoken English (TSE) successfully identified all anomalous ratings. We analyzed data from the April 1997 TSE scoring session using FACETS, a rating scale analysis computer program. The results suggest that, while it is important for an assessment program to identify cases in which there is obvious disagreement in the ratings assigned and have a policy to resolve those disagreements, implementing a discrepancy resolution procedure is not sufficient in and of itself for quality control monitoring. Often times, there are other anomalous ratings that discrepancy resolution procedures may miss. Fit analysis can provide a valuable adjunct to a discrepancy resolution procedure, flagging suspect rating profiles in need of expert review before a final score report is issued.
Details
- Title: Subtitle
- When raters disagree, then what: examining a third-rating discrepancy resolution procedure and its utility for identifying unusual patterns of ratings
- Creators
- Carol M Myford - University of Illinois ChicagoEdward W Wolfe
- Resource Type
- Journal article
- Publication Details
- Journal of applied measurement, Vol.3(3), pp.300-324
- PMID
- 12147915
- NLM abbreviation
- J Appl Meas
- ISSN
- 1529-7713
- Language
- English
- Date published
- 2002
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985123935902771
Metrics
2 Record Views