Submitted

Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark

Zhiqi Yu, Xingping Liu, Haobin Mao, Mingshuo Liu, Long Chen, Jack Xin, and Yifeng Yu

Submitted

arXiv   Bibtex

ABSTRACT:

Grading in large undergraduate STEM courses often yields
minimal feedback due to heavy instructional workloads. We present a
large-scale empirical study of AI grading on real, handwritten single-
variable calculus work from a major U.S. public research
university. Using OCR-conditioned large language models with
structured, rubric-guided prompting, our system produces scores and
formative feedback for thousands of free-response quiz submissions
from nearly 800 students. In a setting with no single ground-truth
label, we evaluate performance against official teaching-assistant
grades, student surveys, and independent human review, finding strong
alignment with TA scoring and a large majority of AI-generated
feedback rated as correct or acceptable across quizzes.  Beyond
calculus, this setting highlights core challenges in OCR-conditioned
mathematical reasoning and partial-credit assessment. We analyze key
failure modes, propose practical rubric- and prompt- design
principles, and introduce a multi-perspective evaluation protocol for
reliable, real-course deployment. Building on the dataset and
evaluation framework developed here, we outline a standardized
benchmark for AI grading of handwritten mathematics to support
reproducible comparison and future research.