SimGrade: Using Code Similarity Measures for More Accurate Human Grading

Sonja Johnson-Yu; Nicholas Bowman; Mehran Sahami; Chris Piech

doi:10.48550/arxiv.2403.14637

Back

SimGrade: Using Code Similarity Measures for More Accurate Human Grading

Preprint

Open access

SimGrade: Using Code Similarity Measures for More Accurate Human Grading

Sonja Johnson-Yu, Nicholas Bowman, Mehran Sahami and Chris Piech

arXiv.org

Cornell University

02/19/2024

DOI: 10.48550/arxiv.2403.14637

Files and links (1)

url

https://doi.org/10.48550/arxiv.2403.14637View

Preprint (Author's original)This preprint has not been evaluated by subject experts through peer review. Preprints may undergo extensive changes and/or become peer-reviewed journal articles. Open Access

Abstract

While the use of programming problems on exams is a common form of summative assessment in CS courses, grading such exam problems can be a difficult and inconsistent process. Through an analysis of historical grading patterns we show that inaccurate and inconsistent grading of free-response programming problems is widespread in CS1 courses. These inconsistencies necessitate the development of methods to ensure more fairer and more accurate grading. In subsequent analysis of this historical exam data we demonstrate that graders are able to more accurately assign a score to a student submission when they have previously seen another submission similar to it. As a result, we hypothesize that we can improve exam grading accuracy by ensuring that each submission that a grader sees is similar to at least one submission they have previously seen. We propose several algorithms for (1) assigning student submissions to graders, and (2) ordering submissions to maximize the probability that a grader has previously seen a similar solution, leveraging distributed representations of student code in order to measure similarity between submissions. Finally, we demonstrate in simulation that these algorithms achieve higher grading accuracy than the current standard random assignment process used for grading.

Computer Science - Computers and Society

Details

Title: Subtitle: SimGrade: Using Code Similarity Measures for More Accurate Human Grading
Creators: Sonja Johnson-Yu
Nicholas Bowman
Mehran Sahami
Chris Piech
Resource Type: Preprint
Publication Details: arXiv.org
Publisher: Cornell University; Ithaca, New York
DOI: 10.48550/arxiv.2403.14637
eISSN: 2331-8422
Language: English
Date posted: 02/19/2024
Academic Unit: Educational Policy and Leadership Studies; Public Policy Center (Archive); Center for Social Science Innovation
Record Identifier: 9984577123802771

Metrics

2 Record Views