One-Line Summary
This work asks how faithfully LLM-based educational assessment agrees with human raters, and argues that a single agreement number is misleading. It proposes an uncertainty-aware, multi-metric framework that reports human–machine agreement across several complementary metric families—together with the uncertainty around each estimate—so that the reliability of LLM scoring can be judged honestly rather than optimistically.
Background & Motivation
Large language models are increasingly used to grade and evaluate student work in educational settings, from scoring short answers and essays to rating rubric dimensions. Because human expert grading is expensive and slow, LLM-based assessment is attractive as a scalable alternative. But before an LLM can replace or assist a human rater, we must know how much its judgments actually agree with human judgments—and how much we can trust that agreement estimate itself.
Much prior work reports human–machine agreement using a single headline statistic, often raw percentage agreement or a single correlation. This is problematic for two reasons. First, raw agreement does not account for the agreement expected by chance, which can make weak models look deceptively strong. Second, a point estimate hides the uncertainty in the measurement: with limited samples, ordinal rating scales, and imbalanced grade distributions, an agreement score can swing substantially, and reporting it without confidence bounds gives a false sense of precision.
Key Challenge: A single agreement score is a fragile summary of a complex phenomenon. Different metric families capture different aspects of agreement:
- Chance correction matters: Raw percentage agreement inflates results because some agreement occurs by chance; chance-adjusted measures such as Cohen's/Fleiss' kappa correct for this.
- Ordinal structure matters: Educational grades are typically ordered, so a disagreement of one level should be penalized less than a large disagreement—motivating weighted kappa (e.g., quadratic weighted kappa).
- Uncertainty matters: Any agreement estimate carries sampling and rater-related uncertainty, which must be quantified and reported rather than assumed away.
Proposed Method
The paper frames the evaluation of LLM-based assessment as a measurement problem and builds an evaluation protocol that is deliberately multi-metric and uncertainty-aware. Rather than optimizing a model to hit one target number, it treats human–machine agreement as something to be characterized carefully from several complementary angles.
1
Collect Paired Human and Machine Ratings
For an educational assessment task, gather ratings from human experts and from an LLM-based scorer over the same items, producing paired judgments that can be compared directly. This pairing is the basis for every agreement statistic that follows.
2
Report Multiple Agreement Metric Families
Instead of a single score, compute agreement across several metric families that capture distinct properties: raw/percentage agreement as a baseline, chance-adjusted agreement (kappa-type coefficients) to remove agreement expected by chance, and weighted agreement (e.g., weighted kappa) to respect the ordinal structure of educational grades so that near-misses are penalized less than gross disagreements.
3
Quantify Uncertainty Around Each Estimate
Attach a measure of uncertainty to every agreement estimate—for example, confidence intervals—so that the stability of each number is visible. This turns a single point estimate into an interval that reflects how much the result could vary given the data, guarding against over-interpreting a lucky or unlucky sample.
4
Interpret Agreement Jointly, Not in Isolation
Read the metrics together as a profile rather than cherry-picking the most favorable one. When chance-adjusted and weighted metrics agree and their uncertainty is small, confidence in the LLM scorer is warranted; when they diverge or their intervals are wide, the framework flags that the LLM's agreement with humans is not yet trustworthy for that assessment context.
Key Points
- Single scores mislead: Judging LLM-based educational assessment by one agreement number—especially raw percentage agreement—can substantially overstate how well the model tracks human raters.
- Chance-adjusted measures are essential: Kappa-type coefficients remove the agreement expected by chance and give a more honest picture of genuine human–machine concordance.
- Ordinal structure should be respected: Weighted agreement measures acknowledge that educational grades are ordered, so being off by one level is not the same as being off by many.
- Uncertainty must be reported: Every agreement estimate should come with an uncertainty range; a wide interval signals that the result should not be trusted as a precise measurement.
- Multi-metric profiles enable honest decisions: Reporting several metric families together, each with its uncertainty, lets practitioners decide when an LLM scorer is reliable enough to deploy and when it is not.
Why It Matters
As LLMs move into real classrooms and testing pipelines, the stakes of getting evaluation right are high. If we overstate how well an LLM agrees with human graders, we risk deploying automated assessment that quietly misgrades students, with consequences for feedback, placement, and fairness. An uncertainty-aware, multi-metric view makes the limits of LLM scoring explicit instead of hiding them behind a single flattering number.
By treating human–machine agreement as a measurement to be characterized—chance-corrected, ordinally weighted, and reported with its uncertainty—this work offers educators and researchers a more trustworthy basis for deciding where LLM-based assessment can responsibly support or substitute for human raters, and where human oversight remains necessary.
Benchmark
Domain LLM