Journal article
Human rater monitoring with automated scoring engines
Psychology science, Vol.61(2), pp.127-148
04/01/2019
Abstract
In this study, we focus on the measurement errors associated with individual raters for rater accuracy. [...]using the resulting estimates from the two sets of analyses that utilize HE and AE, a rater was considered accurate/inaccurate if his/her rater-specific measurement error variance (ar) estimate is significantly lower/higher than the fixed value of HE or AE (1). [...]for future study, it may be interesting to explore the use of these models to larger data sets to examine the findings in a similar context, or to experiment with more reasonable and realistic criteria for rater inaccuracy and rater centrality. Fixed-effects rater location estimates were used for rater severity/leniency, and rater-specific measurement error variances were used for rater accuracy/inaccuracy. [...]the fixed values used for hypothesis testing are meaningful because they provide the basis for statistical comparison of individual HRs against each target score with respect to severity and accuracy. The present study rests on one real data set based on the fully crossed design, which was experimental and ideal enough to estimate multiple rater effects for individuals. Because most of the automated scoring systems, including the one used in this study, are datadriven (from specific data sets) statistical procedures that maximize the predictive accuracy of the outcome variables, AE scores as a reference might not be stable enough to estimate multiple rater effects.
Details
- Title: Subtitle
- Human rater monitoring with automated scoring engines
- Creators
- Hyo ShinEdward WolfeMark Wilson
- Resource Type
- Journal article
- Publication Details
- Psychology science, Vol.61(2), pp.127-148
- ISSN
- 2190-0493
- eISSN
- 2190-0507
- Publisher
- PABST Science Publishers
- Language
- English
- Date published
- 04/01/2019
- Academic Unit
- Psychological and Quantitative Foundations
- Record Identifier
- 9985134747302771
Metrics
1 Record Views