One-Line Summary
A comparative study of rubric design strategies for using a large language model as an automatic judge (LLM-as-a-Judge) of Korean-language responses — examining how choices such as holistic versus analytic rubrics, coarse versus fine-grained criteria, and different scoring scales affect the reliability and human alignment of automatic judgments in Korean.
Background & Motivation
LLM-as-a-Judge has become a widely adopted paradigm for evaluating open-ended text generation, replacing costly and slow human evaluation with an LLM that scores or ranks candidate responses. Because a judge LLM is prompted with an evaluation rubric that defines what constitutes a good response, the quality of its judgments depends heavily on how that rubric is designed.
Most existing rubric-design practices and empirical findings originate from English-centric evaluation, where the relationship between rubric strategy and judgment reliability has been studied more extensively. It remains unclear whether the same design choices transfer to Korean, a language with different morphological structure, honorific and politeness conventions, and stylistic norms that a judge must account for when assessing fluency, appropriateness, and helpfulness.
Key Challenge: The reliability of LLM-as-a-Judge depends heavily on rubric design, yet how rubric strategies affect the quality and human alignment of automatic judgments in Korean is understudied. Practitioners building Korean evaluation pipelines have little empirical guidance on which rubric strategy to choose.
This paper directly compares alternative rubric strategies under a controlled setting in order to identify which choices yield more reliable, consistent, and human-aligned judgments when an LLM evaluates Korean responses.
Dimensions of Rubric Design
A rubric strategy is not a single knob but a combination of several design dimensions, each of which may interact with the characteristics of Korean:
| Dimension | Option A | Option B |
|---|
| Structure | Holistic (single overall judgment) | Analytic (per-criterion scores) |
| Granularity of criteria | Coarse (few broad criteria) | Fine-grained (many specific criteria) |
| Scoring scale | Narrow (e.g., binary or 3-point) | Wide (e.g., 5- or 10-point) |
| Reasoning | Score directly | Explain rationale before scoring |
| Criterion specification | Generic descriptions | Detailed, language-aware descriptions |
Proposed Method
The study frames rubric design as a controlled comparison, systematically varying rubric strategies while holding the judge model, prompts, and evaluated responses fixed, so that any change in judgment quality can be attributed to the rubric strategy itself.
1
Holistic vs. Analytic Rubrics
The first axis contrasts a holistic rubric, in which the judge produces a single overall quality judgment for a Korean response, against an analytic rubric, in which the judge scores several distinct sub-criteria (such as fluency, factual/contextual appropriateness, and helpfulness) that are then aggregated. This comparison tests whether decomposing evaluation into explicit criteria makes the judge more consistent and better aligned with human evaluators, or whether it introduces additional noise for Korean.
2
Coarse vs. Fine-Grained Criteria
The second axis varies the granularity of the evaluation criteria — from a small number of broad criteria to a larger set of narrowly defined ones. Finer criteria can capture subtle aspects of Korean quality (e.g., appropriate honorific usage or natural phrasing) but may also overwhelm the judge or dilute its focus. The study examines the trade-off between descriptive coverage and judgment stability.
3
Scoring Scale Design
The third axis compares different score scales, ranging from coarse binary or few-point scales to wider multi-point scales. Wider scales offer more resolution but can reduce inter-judgment consistency and human agreement if the judge cannot reliably distinguish adjacent levels. The comparison identifies which scale granularity best balances discriminative power against reliability for Korean responses.
4
Criterion Specification and Rationale
Beyond structure and scale, the study varies how explicitly each criterion is described in the rubric — from generic instructions to detailed, language-aware descriptions — and whether the judge is asked to explain its reasoning before scoring. These prompt-level choices are evaluated for their effect on the stability and human alignment of Korean judgments.
5
Reliability and Alignment Analysis
For each rubric strategy, the resulting judgments are assessed for reliability (consistency of scores across repeated or parallel judgments) and human alignment (agreement with human preferences on Korean responses). Comparing strategies along these two axes reveals which rubric designs produce trustworthy automatic evaluation in Korean.
Controlled Comparison:
- Fixed judge and inputs: The same judge model and the same set of Korean responses are used across all rubric conditions, isolating the effect of rubric design.
- Parallel prompts: Rubric variants share a common prompt scaffold so that differences reflect the rubric strategy rather than incidental prompt wording.
- Human reference: Human judgments on Korean responses serve as the reference against which each rubric strategy's alignment is measured.
Key Points
- Rubric strategy matters: The choice of rubric structure, criterion granularity, and scoring scale meaningfully changes how reliable and human-aligned an LLM judge is when evaluating Korean responses — it is not a neutral implementation detail.
- Holistic vs. analytic trade-off: Decomposing evaluation into explicit sub-criteria (analytic) can improve interpretability and alignment, but a holistic judgment can be simpler and more stable in some settings; the study characterizes when each is preferable for Korean.
- Granularity has a sweet spot: Overly fine-grained criteria risk overwhelming the judge, while overly coarse criteria miss important aspects of Korean quality — suggesting an intermediate granularity is often most reliable.
- Scale granularity affects consistency: Wider scoring scales provide more resolution but can lower consistency and human agreement when the judge cannot reliably separate adjacent levels.
- Language-aware rubrics help: Rubrics that explicitly account for Korean-specific properties (such as honorifics and natural phrasing) and that elicit rationale before scoring tend to yield more stable, human-aligned judgments.
- Practical guidance for Korean: The comparison offers concrete, empirically grounded guidance for practitioners designing LLM-as-a-Judge evaluation pipelines for Korean-language systems.
Why It Matters
As LLM-based systems for Korean proliferate, reliable automatic evaluation becomes essential for development, benchmarking, and deployment. LLM-as-a-Judge offers a scalable alternative to human evaluation, but its trustworthiness hinges on rubric design — a factor that has been studied primarily in English.
By systematically comparing rubric strategies specifically for Korean, this work fills an important gap: it provides evidence about which design choices lead to judgments that are both consistent and aligned with human preferences in Korean. This helps practitioners avoid brittle evaluation setups and build more trustworthy Korean evaluation pipelines.
Broader Implication: Automatic evaluation is only as reliable as the rubric that drives it. Establishing rubric strategies tailored to Korean is a necessary step toward fair, reproducible, and human-aligned benchmarking of Korean language technologies, and toward extending best practices in LLM-as-a-Judge beyond English-centric settings.
Benchmark
Multilingual