CoChat
Sign Up
PreprintCited by 20Adversarial Robustness in Machine LearningHate Speech and Cyberbullying DetectionTopic Modeling

A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness

Open access
Schwinn, Leo, Ladenburger, Moritz, Beyer, Tim, Mofakhami, Mehrnaz, Gidel, Gauthier, Günnemann, Stephan·2026·arXiv
DOI 10.48550/arxiv.2603.06594arXiv 2603.06594
Open this paper in CoChat — free accountOpen PDF

Abstract

Automated \enquote{LLM-as-a-Judge} frameworks have become the de facto standard for scalable evaluation across natural language processing. For instance, in safety evaluation, these judges are relied upon to evaluate harmfulness in order to benchmark the robustness of safety against adversarial attacks. However, we show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming: diverse victim models exhibit distinct generation styles, attacks distort output patterns, and semantic ambiguity varies significantly across jailbreak scenarios. Through a comprehensive audit using 6642 human-verified labels, we reveal that the unpredictable interaction of these shifts often causes judge performance to degrade to near random chance. This stands in stark contrast to the high human agreement reported in prior work. Crucially, we find that many attacks inflate their success rates by exploiting judge insufficiencies rather than eliciting genuinely harmful content. To enable more reliable evaluation, we propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures. Data available at: https://github.com/SchwinnL/LLMJudgeReliability.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper audits LLM-as-a-Judge safety evaluators against 6,642 human-verified labels and finds that under attack, model, and data distribution shifts, judges often perform near random chance, undermining adversarial robustness measurements. It introduces ReliableBench and JudgeStressTest and shows that correcting for judge precision reduces inflated attack success rates.

Key findings

  • Across 6,642 human-verified samples, judges often perform near random chance: JailJudge AUROC ranges from 0.48 (GCG-R) to 0.64 (GCG); accuracy can be as low as 45% for Direct Prompting on Llama-3.1-8B with the HarmBench judge, and as high as 58% for Gemma-27-B with the same judge.
  • Judge failures inflate reported attack success rates: BoN disproportionately exploits judge false positives, and correcting ASR by judge precision lowers expected ASR; collecting multiple judge-positive samples per behavior improves reliability.
  • Inter-judge agreement does not reliably imply correctness, and ensembles are insufficient to mitigate failures. ReliableBench, built from the 41 most consistent behaviors, raises average judge accuracy from 57% to 70%; JudgeStressTest contains 971 samples, and held-out judge accuracy drops to 0.17–0.30 when all other judges fail (Acc@0) versus overall accuracies of 0.51–0.59.

Methods

  • Audit design: HarmBench test set of 400 queries, randomly subsampled to 100; four victim models (Gemma-3-1B, Llama-3.1-8B, Gemma-27-B, Qwen-3-32B); judges Aegis-Guard, Llama-2-13B HarmBench classifier, JailJudge, and LlamaGuard-3; attacks Direct Prompting, GCG, GCG-REINFORCE, BoN, and PAIR.
  • Labeling: 1–5 harmfulness scale with ≥3 considered harmful; 2,370 samples labeled by authors plus 4,272 via Labelbox, totaling 6,642; manual review of 90 assignments achieved 95% agreement. Data were pre-filtered to StrongReject judge-positive outputs before human annotation.
  • Analysis: judge accuracy across attack, model, behavior, and semantic category; ROC/AUROC; correlation with human ratings; ASR correction by judge precision; Judge Concordance; and leave-one-out evaluation for JudgeStressTest.

Limitations

  • The paper states it does not account for distribution shifts from specific defenses such as adversarial training, paraphrasing, or Circuit Breakers; findings are limited to the tested attack/model combinations, and unseen future shifts may pose distinct challenges. It also notes HarmBench reliance may propagate biases (single-turn textual safety, specific harm categories, neglect of multi-turn agentiness or multimodal safety) and that focusing on adversarial robustness may prioritize this metric over other safety dimensions.
  • Evident from the text: human annotation was restricted to judge-positive samples, so false negatives are not directly human-labeled; proprietary models were excluded; and the audit covers only selected open-weight models, judges, and attacks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.06594

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 12 cited sources checked · 4 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Chen & Goldfarb-Tarrant (2025) showed that adding an apologetic prefix alone shifted judge preferences by up to 98%.

    “Chen & Goldfarb-Tarrant (2025) quantified this phenomenon, showing that adding an apologetic prefix alone shifted judge preferences by up to 98%”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chen & Goldfarb-Tarrant (2025) source

      Section 2.1 says the experiment prepended an apology without otherwise changing the completion, and §3.1.1 reports a preference for the apologetic version of up to 98% in the tie-detection test. The 98% is a tie-detection preference score, not a 98% shift in judge winrates; §3.1.2 reports much smaller winrate shifts. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    Feuer et al. (2024) demonstrated that judges prioritize stylistic features, such as completeness, politeness, formatting, over factual correctness or safety.

    “Feuer et al. (2024) demonstrated that judges prioritize stylistic features, such as completeness, politeness, formatting, over factual correctness or safety.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Feuer et al., 2024 source

      Section 4.1 reports that LLM judges implicitly favor completeness and style, links a favored model’s responses to bland politeness, and finds style closely correlated with overall scores while safety scores are nearly identical. Section 4.2 says judges penalize sarcastic tone heavily but are fairly lenient on major factual errors. The paper does not demonstrate that judges prioritize formatting over correctness or safety, or directly test a stylistic violation against a safety violation. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data

    Individual behaviors exhibit significantly different judge accuracy.

    “Subfigure (c) shows that individual behaviors exhibit significantly different judge accuracy”
    Quote in the paperSubstantial claim
  • Findingown data

    Judgeability differs considerably between attack types.

    “Subfigure (a) demonstrates considerable differences in judgeability between attack types.”
    Quote in the paperSubstantial claim
  • Findingown data

    Reported ASR is often severely inflated.

    “As Figure 1 shows, reported ASR is often severely inflated; we obtain corrected ASR by scaling results by the judge's precision, the probability that a judge-positive is a true positive.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 13
  • May combine two claims. 20

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The original HarmBench test set comprises 400 unique harmful queries.

    “The original HarmBench test set comprises 400 unique harmful queries.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Mazeika et al., 2024 source

      Section 4.1 distinguishes the 400 textual behaviors in HarmBench as a whole from its test set, which contains 410 behaviors. Appendix B.2 confirms that the test set contains 410 behaviors, not 400. (LLM, conf=0.94)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Prior work relies on pairwise statistics (e.g., Cohen's Kappa) to measure inter-judge agreement.

    “While prior work relies on pairwise statistics (e.g., Cohen's Kappa (Shen et al., 2025)), this score is designed to capture multi-judge consensus in a single, interpretable value.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Shen et al., 2025 source

  • May combine two claims.

    Premisecitation

    A large fraction of alleged jailbreak successes are actually hallucinated or practically useless, harmful instructions, leading evaluators to misrepresent real-world risk.

    “They showed that a large fraction of alleged jailbreak successes are actually hallucinated or practically useless, harmful instructions, leading evaluators to misrepresent real-world risk.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Mei et al., 2024 source
    • cites Yan et al., 2025 source

      The Introduction (p. 2) says jailbreak success can reflect hallucination rather than actionable harmful knowledge, and the Answer to RQ2 (p. 8) shows that judges often classify deliberately stripped, non-substantive examples as jailbreaks. This supports the concern that evaluations can overstate real-world risk, but the paper does not establish what fraction of actual alleged jailbreak successes are hallucinated or practically useless. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    Our study does not account for distribution shifts introduced by specific defense mechanisms such as adversarial training, paraphrasing, or others.

    “Our study does not account for distribution shifts introduced by specific defense mechanisms, such as adversarial training algorithms (Xhonneux et al., 2024;Casper et al., 2024), paraphrasing, or others.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Xhonneux et al., 2024 source
    • cites Casper et al., 2024 source

  • May combine two claims.

    Findingown data

    Attacks do not become easier to classify as they progress; judge accuracy fluctuates or remains flat.

    “Furthermore, we do not observe a consistent increase in judge scores (Figure 7b), nor do the attacks become easier to classify as they progress and judge accuracy fluctuates or remains flat (Figure 7c).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • May combine two claims.

    Findingown data

    Individual judge scores do not consistently correlate with human assessments, indicating that single judges are noisy.

    “We do not observe a consistent connection between individual judge scores and human assessments (Figure 6a), indicating that single judges are noisy.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Sample counts decrease as scores progress from 1 (harmless) to 5 (harmful).

    “We observe that sample counts decrease as scores progress from 1 (harmless) to 5 (harmful).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Adjusting the classification threshold is insufficient to resolve effects from distribution shifts; the ROC curves largely hover around the diagonal, indicating performance near random chance.

    “We find that adjusting the classification threshold is insufficient to resolve effects from distribution shifts; the curves largely hover around the diagonal, indicating performance near random chance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Behaviors are not uniformly represented in the dataset.

    “Behaviors are not uniformly represented, a variance primarily attributed to the StrongReject-based filtering; certain adversarial behaviors are inherently more difficult to elicit than others.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Across several behaviors, judges achieve near-unanimous consensus (mean concordance score close to 1) yet consistently fail to align with human ground truth.

    “Notably, across several behaviors, judges achieve near-unanimous consensus (mean concordance score close to 1) yet consistently fail to align with human ground truth.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • May combine two claims.

    Findingown data

    Direct Prompting is among the hardest to judge for outputs from Llama-3.1-8B, with accuracies as low as 45% using the HarmBench judge, while it is considerably easier for outputs from Gemma-27-B, reaching accuracies as high as 58% with the same judge.

    “For example, while Direct Prompting is among the hardest to judge for outputs from Llama-3.1-8B, with accuracies as low as 45% using the HarmBench judge, they prove considerably easier for outputs from Gemma-27-B, reaching accuracies as high as 58% with the same judge.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    LLM judges are substantially less reliable for adversarial safety evaluations than previously assumed, performing on average only slightly better than a random coin-flip.

    “We emphasize the following: LLM judges are substantially less reliable for adversarial safety evaluations than previously assumed, performing on average only slightly better than a random coin-flip.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The fundamental mechanisms of judge failure under distribution shift generalize across model generations.

  • May combine two claims.

    Findingown data

    For attacks like BoN, a single judge-positive sample provides very low confidence that the model has actually been broken, and considerably more samples are required to guarantee at least one successful jailbreak.

    “For attacks like BoN, a single judge-positive sample provides very low confidence that the model has actually been broken, and considerably more samples are required to guarantee at least one successful jailbreak (see Appendix A.5 for results across models).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.
    • Rests on an assumption: Standard evaluations consider an attack successful if any generated sample is flagged by the judge.

  • May combine two claims.

    Findingown data

    Judge performance is highly victim model dependent, with many judges performing close to random guessing.

    “Notably, performance is highly victim model dependent, with many judges performing close to random guessing.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There is no clear connection between the attack optimization stage and human-rated harmfulness, which remains largely stagnant.

    “We observe no clear connection between the optimization stage and human-rated harmfulness (Figure 7a), which remains largely stagnant.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • May combine two claims.

    Findingown data

    JailJudge scores reach a minimum of 0.48 (GCG-R), which is worse than random guessing, and a maximum of 0.64 (GCG), failing to meet the reliability reported in prior work.

    “JailJudge scores reach a minimum of 0.48 (GCG-R), which is worse than random guessing, and a maximum of 0.64 (GCG), failing to meet the reliability reported in prior work.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Larger models such as Qwen-3-32B tend to be easier to judge than smaller models such as Llama-3.1-8B.

    “However, larger models such as Qwen-3-32B tend to be easier to judge than smaller models such as Llama-3.1-8B.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study maintained a minimum of 105 samples for every attack-model combination and labeled at least 785 samples per model.

    “we maintained a minimum of 105 samples for every attack-model combination and labeled at least 785 samples per model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Within attack shift, the problem of judge-hacking remains unexamined.

    “Third, within attack shift, the problem of judge-hacking, whether through optimization-based attacks that maximize judge scores or sampling-based strategies that accumulate false positives by generating many candidates, remains unexamined despite posing a realistic threat to evaluation integrity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Current safety evaluations may be implicitly biased toward detecting harms in categories that are easier for judges to classify, potentially overlooking more subtle or high-risk threats.

    “This discrepancy suggests that current safety evaluations may be implicitly biased toward detecting harms in categories that are easier for judges to classify, potentially overlooking more subtle or high-risk threats that fall outside these distributions.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The fundamental mechanisms of judge failure under distribution shift generalize across model generations.

  • May combine two claims.

    Interpretationown data

    Safety judges may share systematic failure modes, rendering simple consensus an insufficient metric for evaluating model reliability.

    “This phenomenon suggests that safety judges may share systematic failure modes, rendering simple consensus an insufficient metric for evaluating model reliability.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    No prior work systematically investigates the effect of relevant distribution shifts in adversarial safety evaluations, leaving their joint effects unexplored.

    “First, no prior work systematically investigates the effect of relevant distribution shifts in adversarial safety evaluations (attack, model, and data shift), leaving their joint effects unexplored.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Liu et al. (2025) found that judge models tend to make overconfident predictions, and that miscalibration worsens under distribution shift.

    “Their evaluation across several guard models revealed that judge models tend to make overconfident predictions, and that miscalibration worsens under distribution shift, such as when subjected to jailbreak attacks or responses coming from different models.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Liu et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisereasoning

    Direct Prompting, PAIR, and GCG operate independently of judge feedback.

    “Direct Prompting, PAIR, and GCG operate independently of judge feedback.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    None of the attack methods are designed to deliberately exploit judge weaknesses; the potential for judge hacking arises as an unintended consequence of their optimization or sampling procedures.

    “We emphasize that none of these methods are designed to deliberately exploit judge weaknesses; rather, the potential for judge hacking arises as an unintended consequence of their respective optimization or sampling procedures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Existing evaluations lack human-labeled ground truth to quantify which semantic or functional categories pose genuine challenges versus systematic biases.

    “Second, existing evaluations lack human-labeled ground truth to quantify which semantic or functional categories pose genuine challenges versus systematic biases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The final dataset composition remains relatively balanced between human-rated harmful and harmless samples.

    “Although the dataset was pre-filtered to include only samples labeled as harmful by StrongReject, the final composition remains relatively balanced between human-rated harmful (1437, 78%) and harmless (5205, 22%) samples.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Adversarial attacks do not consistently make outputs more harmful, nor do outputs get easier to judge at late optimization stages.

    “Adversarial attacks do not consistently make outputs more harmful, nor do outputs get easier to judge at late optimization stages.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Our findings are limited to the specific combinations of attacks and models tested; unseen distribution shifts in future attacks or models may pose distinct challenges for automated evaluation.

    “Additionally, while we evaluated a diverse set of attacks and models, our findings are limited to the specific combinations tested; unseen distribution shifts in future attacks or models may pose distinct challenges for automated evaluation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Judge inconsistencies can distort attack success rates, and some reported gains in attack effectiveness, such as for BoN, can be attributed to judge insufficiencies.

    “We show that these inconsistencies can distort attack success rates, suggesting that some reported gains in attack effectiveness, such as for BoN, can be attributed to judge insufficiencies.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human labels or ratings are a valid ground truth for harmfulness.
    • Rests on an assumption: Standard evaluations consider an attack successful if any generated sample is flagged by the judge.
30 of 30

  • The authors state this as an assumption.

    The fundamental mechanisms of judge failure under distribution shift generalize across model generations.

    “Our focus lies on identifying fundamental mechanisms of judge failure under distribution shift, which we expect to generalize across model generations.”

  • The paper relies on this without support.

    Human labels or ratings are a valid ground truth for harmfulness.

    “Our human audit reveals that under the distribution shifts inherent to adversarial robustness evaluations, state-of-the-art judges perform, on average, no better than a random coin flip.”

  • The paper relies on this without support.

    Standard evaluations consider an attack successful if any generated sample is flagged by the judge.

    “Standard evaluations consider an attack successful if any generated sample is flagged by the judge.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Harmless-sample percentage is assigned to the wrong count

    Major

    Of 6,642 samples, 5,205 are approximately 78.36%, not 22%.

    Reported 22 · recomputed 78.36495

  • Harmful-sample percentage is assigned to the wrong count

    Major

    Of 6,642 samples, 1,437 are approximately 21.64%, not 78%. This reverses the reported harmful-versus-harmless composition.

    Reported 78 · recomputed 21.63505

  • Human-verified dataset size differs by 200 samples

    Minor

    The contributions section reports 6,442 human-verified samples, while the abstract reports 6,642. The dataset statistics also give a total of 6,642.

  • The stated number of attack methods changes

    Minor

    The methods section says it evaluates four attack methods, but the paper describes five and explicitly says it restricts evaluation to five.

  • ReliableBench baseline accuracy differs between text and caption

    Minor

    The main text says selecting 41 behaviors raises average accuracy from 57% to 70%, while the Figure 9 caption gives the starting accuracy as 53%.

  • PAIR is described as both using and not using judge feedback

    Minor

    PAIR is said to refine prompts using safety-judge feedback, but is later grouped with methods that operate independently of judge feedback.

6 of 6

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Schwinn, L., Ladenburger, M., Beyer, T., Mofakhami, M., Gidel, G., & Günnemann, S.
Affiliations
  • Technical University of Munich (DE)
  • Canadian Institute for Advanced Research (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.06594
arXiv
2603.06594
OpenAlex
W7134821681
Semantic Scholar
e8bd6ec12610833e7de437d7b2a5b4fc7c89fd9d
Cited by
0
References
39
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
LLM-as-a-judge, adversarial robustness, safety assessment, jailbreak attacks, red teaming, distribution shift, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review