Human Scoring with Automated Scoring in Mind

Edward W. Wolfe

doi:10.1201/9781351264808-4

Back

Book chapter

Human Scoring with Automated Scoring in Mind

Edward W. Wolfe

Handbook of Automated Scoring, pp.49-68

Statistics in the social and behavioral sciences series, CRC Press

2020

DOI: 10.1201/9781351264808-4

View Online

Abstract

In this chapter we identify important issues in human scoring that should be considered by automated scoring teams. Specifically, we provide an overview of the terminology relevant to human scoring, the process used in large-scale scoring projects, provide a conceptual framework for depicting research regarding human scoring, and present some illustrative examples of research that represent aspects of that framework. Our presentation emphasizes the issues and ideas that automated scoring teams should consider and investigate in their efforts to obtain the highest-quality ratings for a reasonable cost. The automated scoring team must also consider how they will monitor and document the quality of the assigned scores and how scoring leaders who oversee the human scoring project will provide feedback to raters. The enterprise of training raters is important because raters may introduce errors into the assigned scores if they apply the scoring criteria in idiosyncratic ways due to inadequate training. The inputs and outputs of human scoring processes are helpful reference points for thinking about these processes or systems. Researchers also study the characteristics of responses that are being scored for evidence that they influence the scoring process outputs. Much of the research of this relationship has employed feature analysis of the responses as predictors of rating quality. Feature analysis is the linguistic analysis of the response content through the automated computation and subsequent statistical analysis of variables derived through natural language processing.

Scores Assigned

Analytic Scoring Criteria

Target Standard Deviation

Scoring Criteria

Inexperienced Raters

Common Rater Effect

Target Scores

Item Response Theory

Rater Effects

Influence Rater Accuracy

Rater Attitude

Scoring Task

Human Scoring

Scoring Leaders

Scoring Project

Rating Speed

Unsupervised Machine Learning

Observed Rater Effect

TOEFL iBT

Consensus Scores

Latent Trait Measurement

Scoring Process

Rating Quality

Total Scoring Time

Details

Title: Subtitle: Human Scoring with Automated Scoring in Mind
Creators: Edward W. Wolfe
Resource Type: Book chapter
Publication Details: Handbook of Automated Scoring, pp.49-68
Series: Statistics in the social and behavioral sciences series
DOI: 10.1201/9781351264808-4
Publisher: CRC Press; Boca Raton, FL
Alternative title: Human Scoring with Automated Scoring in Mind
Language: English
Date published: 2020
Academic Unit: Psychological and Quantitative Foundations
Record Identifier: 9985134649502771

Metrics

1 Record Views