A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

May combine two claims.

The 2014 NeurIPS experiment found that roughly half of the papers accepted by one program committee were rejected by the other; the 2021 replication reproduces the same disagreement on a five-times-larger submission pool.

“The 2014 NeurIPS experiment routed 166 papers to two independent program committees and found that roughly half of the papers accepted by one were rejected by the other; the 2021 replication reproduces the same disagreement on a five-times-larger submission pool”

• cites Cortes and Lawrence [2021] sourceDisputed

  Split into 2 atoms — 1 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Legitimate disagreement among qualified annotators on subjective tasks should be modelled as informative signal rather than treated as noise to average out.

“Recent NLP annotation scholarship [Plank, 2022] argues that legitimate disagreement among qualified annotators on subjective tasks should be modelled as informative signal rather than treated as noise to average out.”

• cites Plank, 2022 sourceSupported

May combine two claims.

Trained rater panels do not converge to a single true score, and agreement is raised through training and adjudication rather than existing a priori.

“trained rater panels do not converge to a single true score, and agreement is raised through training and adjudication rather than existing a priori.”

• cites Shaw and Weir, 2007Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ke and Ng, 2019 sourceCould not verify

  Section 4 (Evaluation; page unavailable) describes an agreement metric for evaluating automated essay scoring, and Section 6 (Concluding Remarks; page unavailable) discusses annotated corpora. Neither addresses whether trained rater panels converge on a true score or whether training and adjudication raise their agreement. Searches for rater training, inter-rater reliability, and adjudication found no passage supporting those claims. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Krippendorff's α and Cohen's weighted κ are the standard panel-level inter-rater stability metrics for ordinal scoring.

“Krippendorff's α [Krippendorff, 2011] and Cohen's weighted κ are the standard panel-level inter-rater stability metrics for ordinal scoring”

• cites Krippendorff, 2011Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Artstein and Poesio, 2008 sourcePartly supported

  Section 2.6 describes both α and weighted κ as measures that can account for different magnitudes of disagreement, and §2.6.1 says α applies to multiple coders. But the paper says weighted κ is defined for two coders, so it does not support calling both measures panel-level metrics. Section 5.2 also recommends α, rather than establishing both as standard for ordinal scoring; page numbers were not available in our copy. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

On 10 of the 15 cross-graded cells where the rubric prescribes both axes, all 29 raters converge in a tight band (panel agreement is universal).

“On 10 of 15 cells where the rubric prescribes both axes, all 29 raters converge in a tight band: panel agreement is universal.”

May combine two claims.

The LLM judge population does not split symmetrically on the contested cells: 22 of 26 LLMs score in or near B/C's contested band, 3 sit in the regulation-silent middle gap, and only 1 (GPT-5.4Nano) approaches A's band without consistently scoring within it.

“The LLM judge population does not split symmetrically: 22 of 26 LLMs score in or near B/C's contested band, 3 sit in the regulation-silent middle gap, and only 1 (GPT-5.4Nano) approaches A's band without consistently scoring within it.”

May combine two claims.

Claude 4.6 Opus and Gemini 3.1 Pro tie at 20/30, one point below the top human (21).

“Claude 4.6 Opus and Gemini 3.1 Pro tie at 20/30, one point below the top human (21)”

May combine two claims.

All three human cross-graders agree on correctness for the contested cells but split on the reasoning category.

“All three human cross-graders agree on correctness for the contested cells but split on the reasoning category.”

May combine two claims.

The B/C-direction cluster reaches panel α = 0.56 on the 15 cells; the single A-direction model achieves only r(A, Nano) = 0.22 across the 15-cell vector.

“The B/C-direction cluster reaches panel α = 0.56 on the 15 cells; the single A-direction model achieves only r(A, Nano) = 0.22 across the 15-cell vector.”

May combine two claims.

Examinee capability rank in Phase 1 does not predict judge cluster alignment in Phase 2: weaker Phase-1 examinees become more strongly B/C-aligned as judges, not less.

“examinee capability rank in Phase 1 does not predict judge cluster alignment in Phase 2 (Section 9): weaker Phase-1 examinees become more strongly B/C-aligned as judges, not less.”

May combine two claims.

The LLM judge population converges easily on the deterministic axis of the rubric and disagrees on the interpretive axis.

“the LLM judge population converges easily on the deterministic axis of the rubric and disagrees on the interpretive axis”

May combine two claims.

Within-LLM means are 0.71 (κ w) and 0.73 (ρ).

“against within-LLM means of 0.71 and 0.73”

Probably not a substantial claim: may not pass S3 (load-bearing).

The headline column means are 5.93, 6.13, 6.20 for Claude, Gemini, GPT-5.4Pro.

“and 5.93, 6.13, 6.20 for Claude, Gemini, GPT-5.4Pro”

Probably not a substantial claim: may not pass S3 (load-bearing).

The headline column means are 4.67, 6.33, 6.30 for A, B, C.

“the headline column means are 4.67, 6.33, 6.30 for A, B, C”

May combine two claims.

Within-human means are 0.47 (κ w) and 0.52 (ρ).

“give within-human means of 0.47 and 0.52”

May combine two claims.

Mean cross-tribe Pearson sits below the mean within-tribe value of every other grader.

“mean cross-tribe Pearson sits below the mean within-tribe value of every other grader”

May combine two claims.

On Item C1, A scores 1 (unusable); B and C score 7 and 8 (good); the three LLM judges give 5.5/6/5.5 (acceptable).

“On Item C1, A scores 1 (unusable); B and C score 7 and 8 (good); the three LLM judges give 5.5/6/5.5 (acceptable).”

Classified as probably not a substantial claim.

The three contested Q2 cells are those where the candidate reaches the correct final answer on sub-question (a) but argues from Section 1023 alone, without naming Section 1023/1 as the decisive good-faith exception.

“The three contested Q2 cells are those where the candidate reaches the correct final answer on sub-question (a) but argues from Section 1023 alone, without naming Section 1023/1 as the decisive good-faith exception.”

May combine two claims.

The inverse correlation between examinee total and judge cluster strength is exploratory (n = 8) but consistently signed.

“the inverse correlation between examinee total and judge cluster strength is exploratory (n = 8) but consistently signed”

May combine two claims.

This is smaller than the 2.1× panel-α gap but in the same direction.

“smaller than the 2.1× panel-α gap but in the same direction”

May combine two claims.

Classifying each judge by its contested-cell mean on the 3 contested Q2 cells, 22 of 26 LLMs (85%) land in or near B/C's contested reading band (m_c ∈ [5,8]); 3 LLMs sit in the regulation-silent middle gap (3 < m_c < 5); only 1 LLM (GPT-5.4Nano, m_c = 2.33) approaches A's reading band, and even Nano does not consistently score within A's band of [1,2].

“Classifying each judge by its contested-cell mean (m c ) on the 3 contested Q2 cells, 22 of 26 LLMs (85%) land in or near B/C's contested reading band (m c ∈ [5,8]); 3 LLMs sit in the regulation-silent middle gap (3 < m c < 5); only 1 LLM (GPT-5.4Nano, m c = 2.33) approaches A's reading band, and even Nano does not consistently score within A's band of [1,2].”

May combine two claims.

The LLM mean matches or exceeds the human cohort mean on both Q2 and Q3.

“the LLM mean matches or exceeds the human cohort mean on both”

May combine two claims.

The strongest Phase-1 examinees become moderate-strength B/C-direction judges; none becomes an A-direction judge.

“the strongest Phase-1 examinees become moderate-strength B/C-direction judges; none becomes a A-direction judge”

May combine two claims.

Nano's overall 15-cell r A is only 0.22.

“which is why Nano's overall 15-cell r A is only 0.22”

May combine two claims.

The single A-direction model (Nano) is a closed-frontier-small OpenAI model that fails on multiple other axes.

“the single A-direction model (Nano) is a closed-frontier-small OpenAI model that fails on multiple other axes”

May combine two claims.

The Thai justifications produced by Mini consistently note that the candidate's reasoning is wrong or incomplete but the final conclusion is correct, and assign tier 7 on that basis alone.

“The Thai justifications produced by Mini consistently note that the candidate's reasoning is wrong or incomplete but the final conclusion is correct, and assign tier 7 (correct final answer, reasoning good) on that basis alone.”

May combine two claims.

The Thai justifications produced by Nano repeatedly state that the reasoning has major errors, name a specific omitted statutory citation, and drop the score below the regulation's mid-range under T4.

“The Thai justifications produced by Nano repeatedly state that the reasoning has major errors, name a specific omitted statutory citation, and drop the score below the regulation's mid-range under T4.”

May combine two claims.

The three certified human examiners rank 22, 24, and 27 of 29 by the same metric (B at r = 0.51; C at r = 0.55; A at r = 0.39).

“The three certified human examiners rank 22, 24, and 27 of 29 by the same metric (B at r = 0.51; C at r = 0.55; A at r = 0.39).”

May combine two claims.

Middle LLMs (n = 3) are Qwen 3.6 Max (m c = 4.67), Qwen 3.5 Plus (m c = 4.50), Haiku 4.5 (m c = 4.83).

“Qwen 3.6 Max (m c = 4.67), Qwen 3.5 Plus (m c = 4.50), Haiku 4.5 (m c = 4.83).”

May combine two claims.

On Item C1, A's score is 1.0 at the bottom of T4; B's score is 7.0 and C's score is 8.0 in T7; the three LLM judges are at the bottom of T6 / top of T5.

“A's 1.0 is at the bottom of T4; B's 7.0 and C's 8.0 are in T7; the three LLM judges are at the bottom of T6 / top of T5.”

Classified as probably not a substantial claim. May combine two claims.

The Thai justifications produced by Haiku 4.5 are coherent legal analyses of the right length and shape.

“The Thai justifications produced by Haiku 4.5 are coherent legal analyses of the right length and shape”

May combine two claims.

The strongest in-cluster judge across the 8-examinee panel is Grok 4.1 Fast, the lowest-scoring Phase-1 examinee at 8/30, which reaches ∆ = +0.36.

“The strongest in-cluster judge across the 8-examinee panel is Grok 4.1 Fast, the lowest-scoring Phase-1 examinee at 8/30, which reaches ∆ = +0.36.”

May combine two claims.

On the contested Q2 cells, Nano sits in the regulation's 0-1 band (T4), aligning with Judge A's stricter reading.

“On the contested Q2 cells, Nano sits in the regulation's 0-1 band (T4, correct final answer with unusable reasoning), aligning with Judge A's stricter reading.”

May combine two claims.

They reach ∆ = +0.14, +0.05, +0.07 as judges, moderate-to-weak cluster strength.

“They reach ∆ = +0.14, +0.05, +0.07 as judges, moderate-to-weak cluster strength.”

May combine two claims.

This is above the human panel α = 0.36 but below the B/C-direction-cluster panel α = 0.56.

“above the human panel α = 0.36 but below the B/C-direction-cluster panel α = 0.56”

Probably not a substantial claim: may not pass S3 (load-bearing).

Excluding Qwen 3.5 Plus, the correlation rises to Pearson = -0.90 over n = 7.

“Excluding the single anti-cluster outlier Qwen 3.5 Plus, the correlation rises to Pearson = -0.90 over n = 7.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

B and C credit a correct final answer with present-but-incomplete reasoning at the upper-mid band (5.5-8), where P withholds credit on the interpretive axis specifically.

“B and C credit a correct final answer with present-but-incomplete reasoning at the upper-mid band ( 5.5-8), where P withholds credit on the interpretive axis specifically.”

May combine two claims.

Gemma 4 26B-A4B is ahead of GPT-5.4Pro, GPT-5.5 Pro, and Gemini 3.1 Pro.

“ahead of GPT-5.4Pro, GPT-5.5 Pro, and Gemini 3.1 Pro”

May combine two claims.

Mini sits at the top of the B/C-direction band by mean but anti-correlates with B on cell-to-cell variation.

“Mini sits at the top of the B/C-direction band by mean but anti-correlates with B on cell-to-cell variation”

Probably not a substantial claim: may not pass S3 (load-bearing).

Pearson(exam total, ∆) = -0.34 over n = 8.

“Pearson(exam total, ∆) = -0.34 over n = 8”

May combine two claims.

B's mean is 6.33 with values across the full 4-10 range.

“B's mean is 6.33 with values across the full 4-10 range”

May combine two claims.

Nano scores 1.0 and 0.5 on two contested cells (consistent with A's reading) but 5.5 on the third (against A's reading).

“Nano scores 1.0 and 0.5 on two contested cells (consistent with A's reading) but 5.5 on the third (against A's reading)”

May combine two claims.

The disagreement is structured: A holds the strict reading on all three contested Q2 cells; B and C hold the upper-band reading on all three; the three anchor LLM judges hold the mid-band reading on all three.

“Second, the disagreement is structured: A holds the strict reading on all three contested Q2 cells; B and C hold the upper-band reading on all three; the three anchor LLM judges hold the mid-band reading on all three.”

May combine two claims.

On the three contested Q2 cells, human α = -0.34 and LLM α = 0.68, gap +1.02.

“Q2 | 5 | -0.34 | 0.68 | +1.02”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Across the 15 cross-graded answers, three recurring patterns are seen: cells where the three LLM judges sit above the majority-human pair (e.g. Items C4, C5), cells where they sit within the majority-human band (e.g. Items C2, C1), and cells where all six graders are within 1 point of each other (e.g. Items C6, C7, C3).

“Across the 15 cross-graded answers we see three recurring patterns: cells where the three LLM judges sit above the majority-human pair (e.g. Items C4, C5), cells where they sit within the majority-human band (e.g. Items C2, C1), and cells where all six graders are within 1 point of each other (e.g. Items C6, C7, C3).”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

The only examinee that deviates from the B/C direction at all is Qwen 3.5 Plus (11/30 in Phase 1), and it only reaches the middle gap (m_c = 4.50), not A direction.

“The only examinee that deviates from the B/C direction at all is Qwen 3.5 Plus (11/30 in Phase 1), and even Qwen 3.5 Plus only reaches the middle gap (m c = 4.50), not A direction.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Only 1 LLM (GPT-5.4Nano) approaches A's band, without consistently scoring within it.

“only 1 LLM (GPT-5.4Nano, orange diamond) approaches A's band, without consistently scoring within it.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

All 29 raters fall inside the horizontal band: panel agreement on the 12 non-contested cells is universal.

“All 29 raters fall inside the horizontal band: panel agreement on the 12 non-contested cells is universal.”

May combine two claims.

Pairwise Pearson on the 15 cross-graded cells gives B-C at r = 0.75 and the three LLM judges at r = 0.74-0.84, with B and C pairing with Claude and Gemini at r in [0.55, 0.76].

“Pairwise Pearson on the 15 cross-graded cells gives B-C at r = 0.75 and the three LLM judges at r = 0.74-0.84,with B and C pairing with Claude and Gemini at r in [0.55, 0.76].”

May combine two claims.

Eight of the 26 Phase-2 judges sat the exam as Phase-1 examinees on the same items, but the B/C-direction pattern holds on the 18 non-examinee LLMs too.

“Eight of the 26 Phase-2 judges sat the exam as Phase-1 examinees on the same items, but the B/C-direction pattern holds on the 18 non-examinee LLMs too.”

May combine two claims.

Dropping one rater at a time, the human-pair α ranges from 0.22 (drop B) to 0.76 (drop A); the LLM-pair α ranges from 0.73 (drop GPT) to 0.80 (drop Gemini).

“Dropping one rater at a time, the human-pair α ranges from 0.22 (drop B) to 0.76 (drop A); the LLM-pair α ranges from 0.73 (drop GPT) to 0.80 (drop Gemini).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The human-panel reading bands on the contested cells are A's band at [1, 2] and B/C's band at [6, 8].

“The vertical reference bands are the human-panel reading bands on the contested cells: A's band at [1, 2] (orange) and B/C's band at [6, 8] (blue).”

May combine two claims.

The within-judge ceiling at ≈ 0.94 exceeds the panel 0.77, so inter-model diversity reduces α rather than inflating it.

“the within-judge ceiling at ≈ 0.94 exceeds the panel 0.77, so inter-model diversity reduces α rather than inflating it.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Mean within-cell SD across seeds is 0.00 for Claude, 0.25 for Gemini, and 0.39 for GPT-5.4Pro, with a single GPT-5.4Pro cell at SD 1.16.

“Mean within-cell SD across seeds is 0.00, 0.25, and 0.39 respectively, with a single GPT-5.4Pro cell at SD 1.16.”

May combine two claims.

Claude 4.6 Opus scores 20/30 as an examinee and lands at B/C-direction m_c = 6.00 as a judge with ∆ = +0.14.

“Claude 4.6 Opus scores 20/30 as an examinee and lands at B/C-direction m c = 6.00 as a judge with ∆ = +0.14.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

The A-less pair (B, C) at α = 0.76 sits inside the LLM range; the every-LLM-dropped range [0.73, 0.80] rules out the inversion being carried by any specific model.

“The A-less pair (B, C) at α = 0.76 sits inside the LLM range; the every-LLM-dropped range [0.73, 0.80] rules out the inversion being carried by any specific model.”

May combine two claims.

On the 15 cross-graded cells, the three-LLM anchor panel achieves α = 0.77, while the three-human panel achieves α = 0.36, a gap of +0.41.

“All Questions | 15 | 0.36 | 0.77 | +0.41”

• Also stated in: § Additional information · plain

  “α (A, B, C) 0.36 [-0.09, 0.71] α (Cla, Gem, GPT) 0.77 [0.54, 0.91]”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The answer for Item C1 correctly identifies both the registered-but-unpublished director change on sub-question (a) and the postelection authority of the incoming directors on (b).

“The answer correctly identifies both the registered-but-unpublished director change on sub-question (a) and the postelection authority of the incoming directors on (b).”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

On the 12 non-contested cells (excluding the three contested Q2 cells), human α = 0.73 and LLM α = 0.79, so the panels are stability-equivalent.

“Removing the three contested Q2 cells alone yields human α = 0.73 and LLM α = 0.79 on the remaining 12 cells: the two panels are stability-equivalent off the regulation-silent boundary.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

On the contested cells, the human panel splits between A's band (1 examiner) and B/C's band (2 examiners), but the LLM judge population does not split symmetrically.

“On the contested cells, the human panel splits between A's band (1 examiner) and B/C's band (2 examiners), but the LLM judge population does not split symmetrically.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

The A-less pair (B, C) reaches α = 0.76, inside the LLM range (0.73-0.80).

“The A-less pair (B, C) reaches α = 0.76, inside the LLM range (0.73-0.80).”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All six grader scores on Item C1 are: A = 1.0, B = 7.0, C = 8.0, Claude 4.6 Opus = 5.5, Gemini 3.1 Pro = 6.0, GPT-5.4Pro = 5.5.

“All six grader scores on this cell are: A = 1.0,B = 7.0, C = 8.0, Claude 4.6 Opus = 5.5, Gemini 3.1 Pro = 6.0,GPT-5.4Pro = 5.5.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

The human panel is asymmetric; the LLM panel is symmetric.

“The human panel is asymmetric; the LLM panel is symmetric.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Judge A sits outside this cluster: mean r = 0.57 against the LLMs, 0.52 against the other two humans.

“Judge A sits outside this cluster: mean r = 0.57 against the LLMs, 0.52 against the other two humans.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The anonymised corpus (questions, gold answers, 150 essay answers, primary and cross-grade human scores, 1,164 span-anchored examiner comments) is publicly released at https://huggingface.co/datasets/airesearch/thai-bar-exam-judging under CC-BY 4.0.

“The anonymised corpus (questions, gold answers, 150 essay answers, primary and cross-grade human scores, 1,164 span-anchored examiner comments) is publicly released at https://huggingface.co/datasets/airesearch/ thai-bar-exam-judging under CC-BY 4.0.”

May combine two claims.

Pairwise κ w (humans 0.47, LLMs 0.71) and Spearman ρ (0.52 vs. 0.73) preserve direction but narrow the ratio.

“Pairwise κ w (humans 0.47, LLMs 0.71) and Spearman ρ (0.52 vs. 0.73) preserve direction but narrow the ratio.”

May combine two claims.

The direction is positive in ∼ 98% of resamples and survives every single-rater and single-cell removal except where the removal isolates the regulation-silent boundary itself.

“the direction is positive in ∼ 98% of resamples and survives every single-rater and single-cell removal except where the removal isolates the regulation-silent boundary itself.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Within-cell panel SD is 1.32 for humans and 0.71 for LLMs, a 1.87× ratio.

“Within-cell panel SD is 1.32 for humans and 0.71 for LLMs, a 1.87× ratio; part of the panel-α gap reflects narrower LLM spread rather than tighter consistency.”

May combine two claims.

The within-LLM mean is 1.5× the within-human mean under both coefficients, smaller than the 2.1× panel-α gap but in the same direction.

“The within-LLM mean is 1.5× the within-human mean under both coefficients, smaller than the 2.1× panel-α gap but in the same direction.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims. The quote only partly states this claim.

Withholding the gold answer reduces LLM α to 0.47, and withholding the regulation reduces it to 0.57, both above the human reference of 0.36.

“Baseline (full inputs) 0.77 Regulation withheld 0.57 Gold answer withheld 0.47”

• Also stated in: § Additional information · plain

  “Human reference (full inputs) 0.36”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Grok 4.1 Fast scores 8/30 as an examinee and lands at B/C-direction m_c = 6.50 as a judge with ∆ = +0.36, the strongest in-cluster pull in the eight-examinee panel.

“Grok 4.1 Fast scores 8/30 as an examinee, more than ten points lower, and lands at B/C-direction m c = 6.50 as a judge with ∆ = +0.36, the strongest in-cluster pull in the eight-examinee panel.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Dropping one cross-graded cell at a time, the human range is [0.23, 0.49] and the LLM range is [0.67, 0.82]; the two ranges are disjoint.

“Dropping one cross-graded cell at a time, the human range is [0.23, 0.49] and the LLM range is [0.67, 0.82]; the two ranges are disjoint.”

May combine two claims.

Excluding only the three contested Q2 cells (Items C1, C2, C3), the remaining 12 cells give human α = 0.73 and LLM α = 0.79: the panels are stability-equivalent on the non-contested subset.

“Excluding only the three contested Q2 cells (Items C1, C2, C3), the remaining 12 cells give human α = 0.73 and LLM α = 0.79: the panels are stability-equivalent on the non-contested subset.”

May combine two claims.

All 29 raters fall inside the horizontal band, meaning agreement on the 12 non-contested cells is universal.

“All 29 raters fall inside the horizontal band: agreement on the 12 non-contested cells is universal.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Leave-one-rater-out on the human panel gives α in [0.22, 0.76]; on the three-LLM anchor sub-panel it gives α in [0.73, 0.80].

“Leave-one-rater-out on the human panel gives α in [0.22, 0.76] (drop B to drop A); on the three-LLM anchor sub-panel it gives α in [0.73, 0.80].”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

On items where the interpretive axis is regulation-silent, the human panel splits between two coherent readings, but the LLM judge population overwhelmingly converges on the majority human reading.

“The structural finding is asymmetric: on items where both rubric axes are prescribed, panel agreement is universal across all 29 raters at α = 0.75; on items where the interpretive axis is regulation-silent, the human panel splits between two coherent readings, but the LLM judge population overwhelmingly converges on the majority human reading.”

May combine two claims.

Twenty-two of 26 LLMs land in or near B/C's reading band on the contested cells; only one LLM (GPT-5.4Nano) approaches the minority (A) reading without consistently scoring within it.

“Twenty-two of 26 LLMs land in or near B/C's reading band on the contested cells; only one LLM (GPT-5.4Nano) approaches the minority (A) reading without consistently scoring within it.”

May combine two claims.

The two examinees sit at opposite ends of the Phase-1 capability rank and converge on the same reading as judges, one of them more strongly.

“The two examinees sit at opposite ends of the Phase-1 capability rank and converge on the same reading as judges, one of them more strongly.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Within-judge cross-seed Krippendorff α is 0.998 for Claude, 0.945 for Gemini, and 0.877 for GPT-5.4Pro.

“Within-judge cross-seed Krippendorff α is 0.998 for Claude, 0.945 for Gemini, and 0.877 for GPT-5.4Pro.”

May combine two claims.

Judge A reads the omission of Section 1023/1 as pass/fail decisive and assigns the cell a 1; B and C read it as a partial-credit element and place the cell in the upper band at 7 and 8; the three LLM judges sit between the two positions at 5.5-6.0.

“Judge A reads this omission as pass/fail decisive and assigns the cell a 1; B and C read it as a partial-credit element and place the cell in the upper band at 7 and 8; the three LLM judges sit between the two positions at 5.5-6.0”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

Phase 1 total scores: human cohort mean is 12.04 and LLM cohort mean is 13.12.

“Humans (n = 42, gray dots) and 8 LLMs inserted blind (colored shapes) share a single jittered band; shading marks the conventional ≥ 15 pass zone; dotted lines are the human (12.04, blue) and LLM (13.12, red) cohort means.”

May combine two claims.

B and C place the answer in tier 3 (incorrect final answer, reasoning good, score 2-4); the three LLMs read the same bottom-line conclusion as close enough to the gold to enter the correct-answer tiers 5 or 6.

“B and C place the answer in tier 3 (incorrect final answer, reasoning good, score 2-4); the three LLMs read the same bottom-line conclusion as close enough to the gold to enter the correct-answer tiers 5 or 6.”

• AssumptionRests on an assumption: The three human examiners' scores on the 15 cross-graded cells constitute a valid reference standard for evaluating LLM judge performance.
• AssumptionRests on an assumption: The 15 cross-graded cells are representative of the full set of 150 essay answers for the purpose of comparing panel agreement.

May combine two claims.

On items where the interpretive axis is regulation-silent, the human panel splits into multiple coherent readings and the LLM judge population reproduces only the majority.

“On items where the interpretive axis is regulation-silent, the human panel splits into multiple coherent readings and the LLM judge population reproduces only the majority.”

Classified as probably not a substantial claim.

The rubric crosses final-answer correctness against reasoning quality.

“The rubric (Table 1) crosses final-answer correctness against reasoning quality.”

May combine two claims.

The data permits both framings: 'LLMs are at the median student level on Thai bar essays' and 'LLMs underperform the pass rate of human candidates'.

“the data permits both”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The 2014 grading regulation and the Civil and Commercial Code itself are public.

“The 2014 grading regulation and the Civil and Commercial Code itself are public.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

This reflects A's stricter interpretive reading on the T4-vs-T7 boundary.

“reflecting A's stricter interpretive reading on the T4-vs-T7 boundary”

May combine two claims.

Human cross-graders and LLM judges received the same four inputs and had no access to the primary grader's score, to each other's scores, to per-judge aggregates, or to the participant's identity.

“Human cross-graders and LLM judges thus received the same four inputs and had no access to the primary grader's score, to each other's scores, to per-judge aggregates, or to the participant's identity.”

May combine two claims.

The regulation overlaps and inverts: a correct final answer with unusable reasoning (T4: 0-1) scores strictly below an incorrect final answer with good reasoning (T3: 2-4), so the rubric is not a clean correctness split.

“the regulation overlaps and inverts: a correct final answer with unusable reasoning (T4: 0-1) scores strictly below an incorrect final answer with good reasoning (T3: 2-4), so the rubric is not a clean correctness split.”

Classified as probably not a substantial claim.

The regulation specifies an 8-tier ordinal scale crossing two axes: final-answer correctness against the gold answer (matches/differs), and reasoning quality (unusable/some/acceptable/good/very good).

“The regulation specifies an 8-tier ordinal scale crossing two axes: final-answer correctness against the gold answer (matches / differs), and reasoning quality (unusable / some / acceptable / good / very good).”

May combine two claims.

The variance is not a property of any specific triple.

“confirms that the variance is not a property of any specific triple”

May combine two claims.

The three examiners authored the questions and gold answers before any participant answer was collected, removing question and gold-answer contamination on the LLM side by construction.

“The three examiners authored the questions and the gold answers before any participant answer was collected; the instrument is therefore not on the open web at the time of the 2026 session, which removes question and gold-answer contamination on the LLM side by construction.”

Classified as probably not a substantial claim.

The judge prompt is identical across all 29 LLM judges.

“The judge prompt is identical across all 29 LLM judges.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Examiners do not undergo per-session calibration before grading; they apply the regulation under their own training and prior bar-grading experience.

“examiners do not undergo per-session calibration before grading; they apply the regulation under their own training and prior bar-grading experience”

Classified as probably not a substantial claim.

Two LLM-disadvantaged input ablations on the three-judge anchor panel measure dependence on the regulation and the gold answer respectively.

“Two LLM-disadvantaged input ablations on the three-judge anchor panel measure dependence on the regulation and the gold answer respectively”

Classified as probably not a substantial claim. May combine two claims.

The regulation prescribes tier categories but not the natural-language anchors that distinguish 'somewhat acceptable' from 'acceptable' reasoning.

“the regulation prescribes tier categories but not the natural-language anchors that distinguish”

Classified as probably not a substantial claim.

For each of the three anchor LLM judges, the 15 cross-graded cells were re-scored three times at temperature 0 with seeds 44, 45, 46, yielding 135 additional judgements that bound intra-judge stability under the same prompt.

“For each of the three anchor LLM judges, we additionally re-score the 15 cross-graded cells three times at temperature 0 with seeds 44, 45, 46, yielding 135 additional judgements that bound the intra-judge stability under the same prompt.”

May combine two claims.

For most fact patterns, the interpretive axis is rubric-determined in practice: the gold answer names the decisive Code sections, and examiners agree that an answer omitting a decisive section sits in the lower tiers and an answer naming all decisive sections sits in the upper tiers.

“For most fact patterns, the interpretive axis is rubric-determined in practice: the gold answer names the decisive Code sections, and examiners agree that an answer omitting a decisive section sits in the lower tiers and an answer naming all decisive sections sits in the upper tiers.”

Classified as probably not a substantial claim. May combine two claims.

The three examiners were not informed that any participant was an LLM; the blinding was full, not partial.

“The three examiners were not informed that any participant was an LLM; the blinding was full, not partial.”

Classified as probably not a substantial claim.

The score range for correct-answer-with-unusable-reasoning (T4) is 0-1.

“The score range for correct-answer-with-unusable-reasoning (T4) is 0-1”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The three LLMs were chosen at study start for vendor diversity and standardisation with prior NLP LLM-judge work, before any score was collected.

“The three LLMs were chosen at study start for vendor diversity and standardisation with prior NLP LLM-judge work, before any score was collected”

Probably not a substantial claim: may not pass S3 (load-bearing).

The score range for correct-answer-with-good-reasoning (T7) is 7-8.

“and for correct-answer-with-good-reasoning (T7) is 7-8”

May combine two claims.

The LLM population converges on the majority human reading and misses the minority human reading.

“The LLM population converges on the majority human reading and misses the minority human reading.”

Classified as probably not a substantial claim. May combine two claims.

The deterministic axis is grounded by the gold answer; the interpretive axis is judged without one.

“The deterministic axis is grounded by the gold answer; the interpretive axis is judged without one.”

Classified as probably not a substantial claim. May combine two claims.

The interpretive axis is grounded by examiner judgement under the same regulation; the regulation gives the tier names and score ranges, but does not specify how to map a particular fact-by-fact reasoning chain into 'unusable / some / acceptable / good / very good' on the correct-answer side.

“The interpretive axis is grounded by examiner judgement under the same regulation; the regulation gives the tier names and score ranges, but does not specify how to map a particular fact-by-fact reasoning chain into”

May combine two claims.

Pairwise averaging dilutes any single-rater outlier effect, which is why panel α widens the ratio relative to pairwise coefficients.

“Pairwise averaging dilutes any single-rater outlier effect, which is why panel α widens the ratio relative to pairwise coefficients.”

May combine two claims.

For a smaller set of fact patterns, the interpretive axis is regulation-silent: a candidate reaches the correct final conclusion via a reasoning chain that names some but not all of the gold answer's decisive sections, and the regulation does not specify whether the omission is a T4 or T7 phenomenon.

“For a smaller set of fact patterns, the interpretive axis is regulation-silent: a candidate reaches the correct final conclusion via a reasoning chain that names some but not all of the gold answer's decisive sections, and the regulation does not specify whether the omission is a T4 (correct, unusable) or T7 (correct, good) phenomenon.”

May combine two claims.

The gap is a property of one regulation-silent boundary, not of base-rate panel disagreement on bar-essay scoring.

“The gap is a property of one regulation-silent boundary, not of base-rate panel disagreement on bar-essay scoring.”

Classified as probably not a substantial claim.

The deterministic axis is grounded by an external artefact: the gold answer fixes which side of the matches-or-differs split the candidate sits on.

“The deterministic axis is grounded by an external artefact: the gold answer fixes which side of the matches-or-differs split the candidate sits on.”

May combine two claims.

Free-form essay rubrics combine a deterministic axis and an interpretive axis, and panel agreement degrades where the interpretive axis is regulation-silent.

“The structural framing is that free-form essay rubrics combine a deterministic axis and an interpretive axis, and panel agreement degrades where the interpretive axis is regulation-silent.”

Classified as probably not a substantial claim. May combine two claims.

LLM outputs were transcribed into the same answer-booklet format used by the human participants, all booklets were pseudonymised to sequence-identifier participant IDs, and the booklets were shuffled before grading.

“LLM outputs were transcribed into the same answer-booklet format used by the human participants, all booklets were pseudonymised to sequence-identifier participant IDs, and the booklets were shuffled before grading.”

Classified as probably not a substantial claim.

The stability comparison is a comparison of scoring functions on identical information.

“The stability comparison in the rest of the paper is therefore a comparison of scoring functions on identical information.”

May combine two claims.

The numeric score alone does not recover the tier: a score of 1 could be T2 or T4; a score of 5 could be T5 or T6.

“the numeric score alone does not recover the tier: a score of 1 could be T2 (incorrect, somewhat acceptable reasoning) or T4 (correct, unusable reasoning); a score of 5 could be T5 or T6.”

Classified as probably not a substantial claim.

The bar exam is an attractive testbed because four quantities can be held identical across raters: the question, the official Bar Council grading regulation, the gold answer, and the participant's free-form essay.

“The bar exam is an attractive testbed because four quantities can be held identical across raters: the question, the official Bar Council grading regulation, the gold answer, and the participant's free-form essay.”

Classified as probably not a substantial claim.

The LLM-system-prompt was held constant across all eight LLMs.

“the prompt was held constant across all eight LLMs.”