Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning

Giovanni Zimotti

doi:10.1007/s40692-026-00399-w

Back

Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning

Journal article

Open access

Peer reviewed

Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning

Giovanni Zimotti

Journal of computers in education (the official journal of the Global Chinese Society for Computers in Education)

06/30/2026

DOI: 10.1007/s40692-026-00399-w

Appears in UI Libraries Support Open Access

Files and links (1)

url

https://doi.org/10.1007/s40692-026-00399-wView

Published (Version of record) Open Access

Abstract

This study investigates the use of local Large Language Models (LLMs): Gemma, LLama, Mistral Small, and Phi, in assessing Spanish second language (L2) writing. While commercial models like ChatGPT have already been used for this task, they present privacy risks and reproducibility challenges. To evaluate local alternatives to ChatGPT, human raters were used to assess a third-semester assignment, evaluating 100 paragraphs to provide a baseline of comparison for the local LLMs' assessment. Results showed varying performance: while some models approached the human baseline; others were inaccurate and inconsistent. On average, they performed better using a holistic rubric than an analytic one. A third experiment, averaging the top 3 models: Llama3.3:70b, Gemma2:9b, Mistral-Small:22b, achieved the highest agreement (r = 0.88), surpassing inter-human rating. A separate experiment testing RAG and fine-tuning techniques on the smallest model: Llama3.1:8b, showed no significant improvements with RAG but substantial gains in accuracy and consistency with fine tuning. Findings suggest that local LLMs show promise for Spanish L2 writing assessment. Local LLMs are preferable to proprietary ones because they protect privacy and reduce risks of data exposure, they may better support replicable workflows and may offer practical advantages for long term adoption.

Spanish L2 writing assessment

Automated writing evaluation

Second language assessment

Local large language models

Human-AI rating agreement

Generative AI in language education

Fine-tuning

Reproducible AI workflows

Computer-assisted language learning

Educational data privacy

UIOWA OA Agreement

Details

Title: Subtitle: Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning
Creators: Giovanni Zimotti - University of Iowa, Spanish and Portuguese
Resource Type: Journal article
Publication Details: Journal of computers in education (the official journal of the Global Chinese Society for Computers in Education)
DOI: 10.1007/s40692-026-00399-w
ISSN: 2197-9987
eISSN: 2197-9995
Publisher: Springer Nature
Number of pages: 23
Language: English
Electronic publication date: 06/30/2026
Academic Unit: Spanish and Portuguese
Record Identifier: 9985182815602771

Metrics

1 Record Views

See more details