Logo image
Back
Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model
Preprint   Open access

Comparing Human and AI Rater Effects Using the Many-Facet Rasch Model

Hong Jiao, Dan Song and Won-Chan Lee
ArXiV.org
Cornell University
05/23/2025
DOI: 10.48550/arxiv.2505.18486
url
https://doi.org/10.48550/arxiv.2505.18486View
Preprint (Author's original)This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

Large language models (LLMs) have been widely explored for automated scoring in low-stakes assessment to facilitate learning and instruction. Empirical evidence related to which LLM produces the most reliable scores and induces least rater effects needs to be collected before the use of LLMs for automated scoring in practice. This study compared ten LLMs (ChatGPT 3.5, ChatGPT 4, ChatGPT 4o, OpenAI o1, Claude 3.5 Sonnet, Gemini 1.5, Gemini 1.5 Pro, Gemini 2.0, as well as DeepSeek V3, and DeepSeek R1) with human expert raters in scoring two types of writing tasks. The accuracy of the holistic and analytic scores from LLMs compared with human raters was evaluated in terms of Quadratic Weighted Kappa. Intra-rater consistency across prompts was compared in terms of Cronbach Alpha. Rater effects of LLMs were evaluated and compared with human raters using the Many-Facet Rasch model. The results in general supported the use of ChatGPT 4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet with high scoring accuracy, better rater reliability, and less rater effects.
Computer Science - Computation and Language Computer Science - Learning

Details

Metrics

82 Record Views
Logo image