CoChat
Sign Up
PreprintCited by 0Adversarial Robustness in Machine LearningEthics and Social Impacts of AITopic Modeling

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Open access
Dev, Sunishchal, Sloan, Andrew, Kavner, Joshua, Kong, Nicholas, Sandler, Morgan·2026·arXiv
DOI 10.48550/arxiv.2603.05399arXiv 2603.05399
Open this paper in CoChat — free accountOpen PDF

Abstract

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

AI snapshot

Written from the paper’s full text.

TL;DR

The Judge Reliability Harness (JRH) is an open-source library that generates validation suites to stress-test LLM judges on binary, ordinal, and agentic evaluation tasks. Across four judges and four benchmarks, no judge is uniformly reliable, with reliability varying sharply by task, perturbation type, and model.

Key findings

  • No evaluated judge is uniformly reliable across benchmarks. Judges are generally robust on FORTRESS and HarmBench but more fragile on Persuade; the paper attributes Persuade fragility partly to its multi-class ordinal scoring (score ∈ [1, 6]) versus binary judgments elsewhere.
  • Formatting perturbations often cause larger reliability drops than semantic perturbations. With the exception of FORTRESS, autograders achieve their lowest mean scores on format invariance tests, suggesting typos or layout changes may disrupt judges more than semantic changes.
  • On AgentHarm, agent perturbation scores were 87.5% for GPT-4o, Gemini 2.5 Pro, and Llama 4 Maverick, while Claude Opus 4.5 scored 68.75%. On agent positives, Claude Opus 4.5, GPT-4o, and Llama 4 Maverick tied at 93.75%, while Gemini 2.5 Pro scored 75%; GPT-4o and Llama 4 Maverick had the highest overall accuracy at 0.906.
  • Cost-reliability trade-offs are nontrivial: Llama 4.1 Maverick 17B tends to be the most reliable judge while running at a fraction of the cost. Table 1 reports overall cost per accuracy point of 0.0010 for Llama 4 Maverick 17B, 0.0223 for Claude Sonnet 4.5, 0.0196 for GPT-4o, and 0.0080 for Gemini 2.5 Pro.

Methods

  • JRH runs four stages: load and normalize a seed dataset; generate and validate perturbed items; evaluate the judge on generated samples; compute and aggregate reliability metrics. Tests include label flip, format invariance, semantic paraphrase, verbosity bias, stochastic stability, synthetic ordinal, and agentic perturbation/positives, with optional human-in-the-loop review.
  • The study evaluates four judges—GPT-4o, Claude Sonnet 4.5, Llama Maverick 4.1 17B, and Gemini 2.5 Pro—across four benchmarks: FORTRESS, Persuade, HarmBench, and AgentHarm. Claude Opus 4.5 replaces Sonnet 4.5 on AgentHarm due to observed inconsistencies in Sonnet’s reasoning and structured score.
  • Benchmarks were down-sampled: 10 manually selected samples for FORTRESS, Persuade, and HarmBench, and 16 randomly selected samples for AgentHarm stratified by eight harm categories. GPT-4o-mini produced benchmark responses and perturbations, Gemini 3 Pro validated generated samples, and GPT-4o-mini served as fallback validator when Gemini 3 Pro raised safety errors. Metrics include fraction agreement with target labels; for Persuade, CCC, Pearson’s ρ, Spearman’s ρs, and MAE are also reported.

Limitations

  • The paper states these are preliminary experiments and that benchmarks were down-sampled due to substantial monetary costs; for example, running eight tests across ten samples requires 80 API calls, and downsampling also keeps manual review manageable.
  • The paper states AgentHarm relied more heavily on human-in-the-loop review due to agent complexity and safety guardrails in the perturbation model (GPT-4o); 14 of 16 agent perturbation transcripts had one or more messages modified, while only 2 of 16 agent positives transcripts were modified in HITL.
  • Evident from the text: results are based on small curated subsets (10 samples for three benchmarks; 16 for AgentHarm), so the reported reliability patterns may not generalize. The paper also notes that assessing semantic proximity of judge responses to ground truth in Persuade remains future work.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.05399

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

5 of 8 cited sources checked · 4 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    With the exception of Fortress, autograders struggle and achieve their lowest mean scores on format invariance tests.

    “With the exception of Fortress, we observe that autograders struggle and achieve their lowest mean scores on format invariance tests”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT-4o and Llama 4 Maverick achieved the highest overall accuracy score of 0.906 on AgentHarm.

    “As detailed in Table 3, GPT-4o and Llama 4 Maverick achieved the highest scores with an overall accuracy score of 0.906.”
    Quote in the paperSubstantial claim
  • Findingown data

    For PERSUADE, Pearson's ρ = 0.901 and MAE = 0.48 for Claude Sonnet 4.5.

    “For PERSUADE, we obtain Pearson's ρ = 0.901 and M AE = 0.48 for Claude Sonnet 4.5”
    Quote in the paperNot a substantial claim
  • Findingown data

    This behavior was not observed in Opus 4.5 or Haiku 4.5 (or any other tested model).

    “This behavior was not observed in Opus 4.5 or Haiku 4.5 (or any other tested model).”
    Quote in the paperSubstantial claim
  • Findingown data

    No judge evaluated is uniformly reliable across benchmarks using the harness.

    “No judge that we evaluated is uniformly reliable across benchmarks using our harness.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 12
  • May combine two claims. 21

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Doddapaneni et al. (2024) introduce the FBI meta-evaluation benchmark, where targeted perturbations that degrade factuality, instruction following, long form coherence, or reasoning reveal that evaluator LLMs often fail to detect quality drops.

    “Doddapaneni et al. (2024) introduce the FBI meta-evaluation benchmark, where targeted perturbations that degrade factuality, instruction following, long form coherence, or reasoning reveal that evaluator LLMs often fail to detect quality drops.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Doddapaneni et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Benchmarks such as MT-Bench and Chatbot Arena show that powerful judges like GPT-4 can approach expert level agreement on preference judgments.

    “Benchmarks such as MT-Bench (Bai et al., 2024) and Chatbot Arena (Chiang et al., 2024) show that powerful judges like GPT-4 can approach expert level agreement on preference judgments”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Bai et al., 2024 source

      Section 4.5 (Human Evaluation) and Table 5 report 87% agreement between GPT-4 and human experts, compared with 80% agreement among experts. But the study evaluates 1-to-10 ratings of multi-turn dialogues, not preference judgments, and the cited paper is MT-Bench-101 rather than MT-Bench. Page numbers are unavailable in our copy. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Chiang et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Frameworks such as G-Eval obtain strong correlations with human ratings on summarization and dialog through structured rubrics and chain of thought prompting.

    “while frameworks such as G-Eval (Liu et al., 2023) obtain strong correlations with human ratings on summarization and dialog through structured rubrics and chain of thought prompting.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liu et al., 2023 source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Chehbouni et al. (2025) argue that enthusiasm for LLM judges has outpaced rigorous scrutiny of their validity and reliability, and that key assumptions about their use as measurement instruments remain under tested.

    “Chehbouni et al. (2025) use tools from measurement theory to argue that enthusiasm for LLM judges has outpaced rigorous scrutiny of their validity and reliability, and that key assumptions about their use as measurement instruments remain under tested.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chehbouni et al., 2025 source

  • May combine two claims.

    Findingown data

    For PERSUADE, Pearson's ρ = 0.960 and MAE = 0.23 for GPT-4o.

    “ρ = 0.960 and M AE = 0.23 for GPT-4o”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Persuade benchmark, Claude Sonnet 4.5 shows the highest volatility (standard deviation of 17.18%) while Gemini 2.5 Pro exhibits the lowest (standard deviation of 11.10%).

    “In the Persuade benchmark, Claude Sonnet 4.5 shows the highest volatility (standard deviation of 17.18%) while Gemini 2.5 Pro exhibits the lowest (standard deviation of 11.10%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    LLM judges are robust in the FORTRESS and HarmBench benchmarks, while more fragile on the Persuade benchmark.

    “We observe that LLM judges are robust in the FORTRESS and HarmBench benchmarks, while more fragile on the Persuade benchmark.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On the agent positives test, Claude Opus 4.5, GPT-4o, and Llama 4 Maverick tie for highest score of 93.75% while Gemini 2.5 Pro scores only 75%.

    “On the agent positives test, Claude Opus 4.5, GPT-4o, and Llama 4 Maverick tie for highest score of 93.75% while Gemini 2.5 Pro scores only 75%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Llama 4.1 Maverick 17B tends to be the most reliable judge while running at a fraction of the cost of other judges tested.

    “It will also summarize key quantitative outcomes, indicating that Llama 4.1 Maverick 17B tends to be the most reliable judge while running at a fraction of the cost of other judges tested.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Some judges struggle to detect subtle violations inserted into multi-turn transcripts (high false-negative rates), while others overpenalize corrected transcripts (high false-positive rates).

    “Some judges struggle to detect subtle violations inserted into multi-turn transcripts (high false-negative rates), while others overpenalize corrected transcripts (high false-positive rates).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the agent perturbation test, GPT-4o, Gemini 2.5 Pro, and Llama 4 Maverick each achieved a score of 87.5% while Claude Opus 4.5 achieved a score of 68.75%.

    “For the agent perturbation test, GPT-4o, Gemini 2.5 Pro, and Llama 4 Maverick demonstrate strong robustness, each achieving a score of 87.5% while Claude Opus 4.5 achieves a score of 68.75%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Smaller or moderately sized open models (e.g., Llama 4.1 Maverick 17B) may outperform or match the reliability of premium frontier judges while operating at significantly lower inference cost.

    “Our preliminary experiments show that smaller or moderately sized open models (e.g., Llama 4.1 Maverick 17B) may outperform or match the reliability of premium frontier judges while operating at significantly lower inference cost as seen in Table 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For PERSUADE, Pearson's ρ = 0.935 and MAE = 0.34 for Gemini 2.5 Pro.

    “ρ = 0.935 and M AE = 0.34 for Gemini 2.5 Pro”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For PERSUADE, Pearson's ρ = 0.953 and MAE = 0.29 for Llama Maverick 4.

    “ρ = 0.953 and M AE = 0.29 for Llama Maverick 4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For FORTRESS, Persuade, and HarmBench, the LLM-generated perturbations appeared to satisfy the intended reliability tests upon human review, so no human edits were required before accepting the modified samples.

    “For FORTRESS, Persuade, and HarmBench, we did not require human edits to the LLM-generated perturbations before accepting the modified samples because, upon human review, they appeared to satisfy the intended reliability tests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The paper evaluates four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and finds meaningful variation in performance across models and perturbation types.

    “We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The semantic paraphrase test shows the most consistent robustness across each test with the lowest accuracy score of 40% achieved by Gemini 2.5 Pro in the Persuade Benchmark.

    “Generally, the semantic paraphrase test shows the most consistent robustness across each test with the lowest accuracy score of 40% achieved by Gemini 2.5 Pro in the Persuade Benchmark.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In HarmBench, Claude shows the lowest standard deviation (11.13%) while Gemini shows the highest (17.17%).

    “This pattern is reversed in HarmBench, where Claude shows the lowest standard deviation (11.13%) while Gemini shows the highest (17.17%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Formatting perturbations produce larger reliability drops than semantic perturbations; models are more robust to semantic level perturbations/differences than to formatting variations.

    “suggesting that models are more robust to semantic level perturbations/differences than to formatting variations.”
    • Also stated in: § 6 DISCUSSION · plain
      “Formatting perturbations produce larger reliability drops than semantic perturbations.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    JRH generates reliability tests that measure grading accuracy via label flipped responses, invariance to formatting and paraphrasing, susceptibility to verbosity bias, stochastic stability under repeated sampling, and calibration across an ordinal grading scale.

    “JRH generates reliability tests that measure grading accuracy via label flipped responses, invariance to formatting and paraphrasing, susceptibility to verbosity bias, stochastic stability under repeated sampling, and calibration across an ordinal grading scale.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ye et al. identify a wide range of biases in LLM judges, including position and verbosity biases, and propose the CALM framework to quantify their impact.

    “Ye et al. identify a wide range of biases in LLM judges, including position and verbosity biases, and propose the CALM framework to quantify their impact.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ye et al.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The gap between the central role of model judges and the limited tools available to characterize their reliability makes it difficult for practitioners and decision makers to understand how much confidence to place in AI evaluation results.

    “This gap between the central role of model judges and the limited tools available to characterize their reliability makes it difficult for practitioners and decision makers to understand how much confidence to place in AI evaluation results.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The reliability of judge system configuration (including the LLM judge model, rubric, and prompt templates) is rarely evaluated and measured in a systematic manner or reported alongside benchmark evaluation results.
    • Rests on an assumption: Point estimates of agreement with human raters on small validation sets provide limited assurance about how a judge will respond to realistic variations in inputs, such as changes in formatting, paraphrasing, verbosity, or sampling parameters.

  • Classified as probably not a substantial claim.

    Premisenone given

    Each reliability test reveals a different dimension of reliability/robustness through the generation and validation of synthetically generated data.

    “Each test reveals a different dimension of reliability/robustness through the generation and validation of synthetically generated data.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    Across a range of candidate judges, JRH aggregates pass rates, confidence intervals, and cost curves into standardized reports.

    “Across a range of candidate judges, it aggregates pass rates, confidence intervals, and cost curves into standardized reports.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Thakur et al. (2025) benchmark multiple judge models against human exam graders and find that only the largest models attain reasonable alignment, while still displaying sensitivity to prompt complexity and a tendency toward lenient scoring.

    “Thakur et al. (2025) benchmark multiple judge models against human exam graders and find that only the largest models attain reasonable alignment, while still displaying sensitivity to prompt complexity and a tendency toward lenient scoring.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Thakur et al., 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    JRH features a human-in-the-loop review process for generated reliability tests through a user interface that gives full control to accept, reject, or edit the tests.

    “JRH features a human-in-the-loop review process for generated reliability tests through a user interface that gives full control to accept, reject, or edit the tests.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The harness offers a practical toolkit for incorporating reliability assessments into research workflows, enabling practitioners to identify unreliable judge configurations before they influence model comparisons or safety evaluations.

    “The harness offers a practical toolkit for incorporating such reliability assessments into research workflows, enabling practitioners to identify unreliable judge configurations before they influence model comparisons or safety evaluations.”
    Quote in the paperSubstantial claim
27 of 27

  • The paper relies on this without support.

    The reliability of judge system configuration (including the LLM judge model, rubric, and prompt templates) is rarely evaluated and measured in a systematic manner or reported alongside benchmark evaluation results.

    “However, the reliability of judge system comfiguration, including the LLM judge model, rubric, and prompt templates, are rarely evaluated and measured in a systematic manner or reported alongside benchmark evaluation results.”

  • The paper relies on this without support.

    Point estimates of agreement with human raters on small validation sets provide limited assurance about how a judge will respond to realistic variations in inputs, such as changes in formatting, paraphrasing, verbosity, or sampling parameters.

    “Point estimates of agreement with human raters on small validation sets provide limited assurance about how a judge will respond to realistic variations in inputs, such as changes in formatting, paraphrasing, verbosity, or sampling parameters.”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Persuade paraphrase minimum conflicts with reported 40% score

    Minor

    The results say Gemini scored 40% on Persuade semantic paraphrase, but the table gives 50% as the minimum score for that test.

  • HarmBench has seven reported tests rather than eight

    Minor

    The text says the HarmBench results cover eight reliability tests, but its performance-by-test table lists only seven: three format tests, label flip, semantic paraphrase, stochastic stability, and verbosity bias long. Unlike FORTRESS, it has no verbosity bias short row.

2 of 2

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 1

  1. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as- judges. In Ofir Arviv, Miruna Clinciu, Kaustubh Dhole, Rotem Dror, Sebastian Gehrmann, Eliya Habba, Itay Itzhak, Simon Mille, Yotam Perlitz, Enrico Santus, João Sedoc, Michal Shmueli Scheuer, Gabriel Stanovsky, and Oyvind Tafjord (eds.), Proceedings of the Fourth Work- shop on Generation, Evaluation and Metrics (GEM²), pp. 404-430, Vienna, Austria and virtual meeting, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-261-9. URL https://aclanthology.org/2025.gem-1.33/.

Cited by

Still being checked.

Related

Authors
Dev, S., Sloan, A., Kavner, J., Kong, N., & Sandler, M.
Affiliations
RAND Corporation (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.05399
arXiv
2603.05399
OpenAlex
W7134036278
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
LLM-as-a-judge, reliability testing, open-source library, benchmark evaluation, robustness, persuasion, misuse, paraphrase

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-11-03.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-11-03.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review