A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays
Pith reviewed 2026-06-29 22:09 UTC · model grok-4.3
The pith
LLM judges converge on majority human reading and ignore minority interpretation on ambiguous rubric cells
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
When the rubric does not prescribe how to grade a correct final answer that omits a decisive statutory citation, the human panel splits between a B/C majority at scores 6-8 and examiner A at scores 1-2. Of 26 LLMs, 22 score in or near the B/C band, three occupy the silent middle gap, and only GPT-5.4 Nano approaches A's band without consistently scoring within it. No LLM reproduces the minority reading on the contested cells. The B/C cluster spans every model size, vendor, and price tier tested, while an instrumented three-LLM anchor sub-panel reaches alpha 0.77 against the human panel's alpha of 0.36.
What carries the argument
The identical-inputs protocol that supplies the exact same question, official grading regulation, gold answer, and candidate answer to both the three human examiners and the 26-LLM panel for direct comparison of score distributions on contested cells.
If this is right
- A benchmark that selects its LLM judge by maximizing agreement with a human reference panel will inherit the asymmetry toward the majority reading by construction.
- The high LLM-panel alpha of 0.77 versus human-panel alpha of 0.36 reflects systematic convergence on one reading rather than balanced reproduction of both.
- The B/C-direction cluster includes models from every size, vendor, and price tier tested.
- An instrumented three-LLM anchor sub-panel carries determinism probes and input ablations yet still inherits the majority bias.
Where Pith is reading between the lines
- LLM judges may systematically narrow the range of acceptable legal interpretations even when the rubric permits multiple coherent readings.
- The protocol could be extended to other domains with ambiguous scoring rules to test whether one-sided LLM convergence is specific to legal essays.
- Methods that deliberately encourage LLMs to sample from minority human bands on contested cells could be developed and measured against this baseline.
- Stability metrics alone do not guarantee that an LLM judge captures the full distribution of valid human judgments.
Load-bearing premise
The three human examiners represent the full spectrum of valid interpretations permitted by the rubric, and the observed split reflects genuine rubric ambiguity rather than examiner error or idiosyncratic judgment.
What would settle it
Finding even one LLM that consistently assigns scores in examiner A's 1-2 band on all five contested cells would falsify the claim that zero LLMs reproduce the minority human reading.
Figures
read the original abstract
Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling as evidence of judge stability. We test both assumptions on the Thai bar examination through an identical-inputs protocol: three Bar Council-trained examiners (A, B, C) and a 26-LLM judge panel score the same 15 cross-graded answers from the same four inputs (question, official Bar Council grading regulation, gold answer, candidate answer). The headline finding is asymmetric. On 10 of 15 cells where the rubric prescribes both axes, all 29 raters converge in a tight band: panel agreement is universal. On the remaining 5 cells where the rubric does not prescribe how to grade a correct final answer that omits a decisive statutory citation, the human panel splits between two coherent readings (B/C majority at the upper rubric band, score 6-8; A minority at the lower band, score 1-2). The LLM judge population does not split symmetrically: 22 of 26 LLMs score in or near B/C's contested band, 3 sit in the regulation-silent middle gap, and only 1 (GPT-5.4 Nano) approaches A's band without consistently scoring within it. Zero LLMs in our 26-judge panel reproduce the minority human reading on the contested cells. The B/C-direction cluster spans every model size, vendor, and price tier we tested. An instrumented three-LLM anchor sub-panel (Claude 4.6 Opus, Gemini 3.1 Pro, GPT-5.4 Pro) carries determinism probes, input ablations, and bootstrap CIs, and reaches anchor panel $\alpha = 0.77$ on the 15 cells against human-panel $\alpha = 0.36$. The high LLM-panel $\alpha$ reflects systematic convergence on the majority reading rather than balanced reproduction of both readings; a benchmark that selects its LLM judge by maximising agreement with a human reference panel will inherit this asymmetry by construction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports results from an identical-inputs protocol in which three Bar Council examiners and a 26-LLM panel independently score the same 15 Thai bar-exam free-form answers against the official grading regulation and gold answer. It finds near-universal convergence on 10 cells where the rubric is prescriptive, but a 2-1 human split on the remaining 5 cells (where the rubric is silent on penalizing omission of a decisive statutory citation); 22 LLMs cluster with the majority human band, three occupy the middle gap, and none reproduce the minority (examiner A) reading. An instrumented three-LLM anchor sub-panel yields higher internal consistency (α=0.77) than the human panel (α=0.36), which the authors attribute to systematic convergence on the majority reading.
Significance. If the central empirical pattern holds, the work demonstrates that LLM-judge panels can systematically favor one of two rubric-permitted human interpretations when the grading regulation is under-specified, implying that any benchmark that selects judges by maximizing agreement with a human reference panel will inherit this asymmetry by construction. The identical-inputs design, determinism probes, input ablations, and bootstrap CIs on the anchor panel constitute concrete methodological strengths that allow direct comparison of observed score distributions.
major comments (1)
- [discussion of the 5 contested cells and the 2-1 human split] The claim that zero of the 26 LLMs reproduce the minority human reading on the 5 contested cells presupposes that examiner A's lower-band scores (1-2) constitute a legitimate, rubric-permitted alternative rather than an idiosyncratic threshold. This assumption rests on the rubric's silence regarding omission of a decisive statutory citation when the final answer is otherwise correct, together with the observed 2-1 split. With only three human raters, however, the data cannot distinguish genuine ambiguity from examiner error or personal judgment; no additional validation (expanded rater pool, inter-examiner rationale transcripts, or explicit rubric clauses) is described that would confirm both bands are defensible under the full Bar Council regulation. This is load-bearing for the headline asymmetry result.
Simulated Author's Rebuttal
We thank the referee for the careful reading and for identifying a key interpretive assumption in our analysis of the contested cells. We address the major comment below.
read point-by-point responses
-
Referee: The claim that zero of the 26 LLMs reproduce the minority human reading on the 5 contested cells presupposes that examiner A's lower-band scores (1-2) constitute a legitimate, rubric-permitted alternative rather than an idiosyncratic threshold. This assumption rests on the rubric's silence regarding omission of a decisive statutory citation when the final answer is otherwise correct, together with the observed 2-1 split. With only three human raters, however, the data cannot distinguish genuine ambiguity from examiner error or personal judgment; no additional validation (expanded rater pool, inter-examiner rationale transcripts, or explicit rubric clauses) is described that would confirm both bands are defensible under the full Bar Council regulation. This is load-bearing for the headline asymmetry result.
Authors: We agree that n=3 prevents us from definitively separating rubric ambiguity from individual examiner judgment. The manuscript's interpretation rests on (i) the official regulation's explicit silence on penalizing omission of a decisive statutory citation when the final answer is otherwise correct and (ii) the fact that the 2-1 split occurred among three Bar Council-trained examiners rather than random disagreement. We do not claim the split proves equal validity of both readings; we claim it demonstrates observable human variation on an under-specified point. The central empirical pattern—that 22 LLMs cluster with the majority band, three occupy the middle gap, and none consistently reproduce examiner A's band—remains informative even under a conservative reading that treats A's scores as idiosyncratic: it shows the LLM population does not sample the full range of observed human scores on these cells. We will add a limitations paragraph explicitly noting that the small human panel precludes strong claims about the source of the split. revision: partial
- We cannot supply an expanded rater pool, inter-examiner rationale transcripts, or new explicit clauses in the Bar Council regulation, as these lie outside the original study design and the fixed external materials provided by the Bar Council.
Circularity Check
No circularity: direct empirical comparison of observed scores
full rationale
The paper conducts an empirical study by having three human examiners and 26 LLMs score the same 15 essay cells under identical inputs (question, regulation, gold answer, candidate answer). It reports observed score distributions, splits (e.g., 2-1 human split on 5 cells), and agreement metrics such as Cronbach's alpha without any equations, fitted parameters, or derivations. No self-citations are invoked to justify core premises, no predictions are made from fitted inputs, and no ansatzes or uniqueness theorems are smuggled in. The central claims rest on raw inter-rater data rather than reducing to the paper's own inputs by construction.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption The official Bar Council grading regulation is the authoritative and complete source for scoring criteria.
- domain assumption The three Bar Council-trained examiners provide representative samples of expert human judgment.
Reference graph
Works this paper leans on
-
[1]
URLhttps://arxiv.org/abs/2109.09774. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated evaluation of retrieval augmented generation. In Nikolaos Aletras and Orphee De Clercq, editors,Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 1...
-
[2]
LLM Evaluators Recognize and Favor Their Own Generations
doi: https://doi.org/10.1111/jels.12413. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/ jels.12413. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. LEXTREME: A multi-lingual and multi-task benchmark for the legal domain. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association ...
work page internal anchor Pith review Pith/arXiv arXiv doi:10.1111/jels.12413 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.