CoChat
Sign Up
PreprintCited by 5Adversarial Robustness in Machine LearningHate Speech and Cyberbullying DetectionTopic Modeling

Efficient LLM Safety Evaluation through Multi-Agent Debate

Open access
Lin, Dachuan, Shen, Guobin, Yang, Zihao, Liu, Tianrong, Zhao, Dongcheng, Zeng, Yi·2025·arXiv
DOI 10.48550/arxiv.2511.06396arXiv 2511.06396
Open this paper in CoChat — free accountOpen PDF

Abstract

Safety evaluation of large language models (LLMs) increasingly relies on LLM-as-a-judge pipelines, but strong judges can still be expensive to use at scale. We study whether structured multi-agent debate can improve judge reliability while keeping backbone size and cost modest. To do so, we introduce HAJailBench, a human-annotated jailbreak benchmark with 11,100 labeled interactions spanning diverse attack methods and target models, and we pair it with a Multi-Agent Judge framework in which critic, defender, and judge agents debate under a shared safety rubric. On HAJailBench, the framework improves over matched small-model prompt baselines and prior multi-agent judges, while remaining more economical than GPT-4o under the evaluated pricing snapshot. Ablation results further show that a small number of debate rounds is sufficient to capture most of the gain. Together, these results support structured, value-aligned debate as a practical design for scalable LLM safety evaluation.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces HAJailBench, an 11,100-instance human-annotated jailbreak benchmark, and a Multi-Agent Judge pipeline in which critic, defender, and judge agents debate under a shared safety rubric. It reports that the framework improves agreement with human labels over matched small-model prompt baselines and prior multi-agent judges while remaining cheaper than GPT-4o under the evaluated pricing snapshot.

Key findings

  • On HAJailBench, the Qwen3-14B Multi-Agent Judge reaches κ = 0.7331, only 0.0258 below GPT-4o (κ = 0.7589) while using 46% of GPT-4o’s estimated cost per query; relative to JailJudge on the same backbone, this is a gain of 0.1786 κ points with slightly lower unit cost.
  • Fine-tuned safety judges remain far below human agreement on HAJailBench, with the best reaching only κ = 0.3510.
  • Ablations show accuracy improves from κ = 0.5709 without debate to κ = 0.7331 in the 3-round reference system, then drops slightly at 4 and 5 rounds; removing alignment lowers performance to κ = 0.7239, and a stripped-down zero-round variant without alignment, denoising, or refusal falls to κ = 0.5200.
  • On JailJudge-ID, the method improves agreement from κ = 0.6008 to κ = 0.6843, suggesting transfer beyond HAJailBench despite a modest cost increase.

Methods

  • The framework is a staged judging pipeline: refusal screening, pre-debate value alignment selecting five safety aspects, response denoising, bounded critic-defender debate, and a final judge pass that emits a binary attack-success label, a five-level ordinal risk label, and a ten-point risk score.
  • HAJailBench contains 11,100 human-labeled responses from 100 harmful goals from JBB-Behaviors, 12 attack methods, and 11 target-model configurations; annotation uses a two-round human review with primary scalar scores and human-only final review for discrepancy-heavy cases, integrated via safe (1–4), suspicious (5–6), and unsafe (7–10) bands.
  • Evaluation reports Cohen’s κ against human ground truth plus unit cost and cost ratio, using prompt and completion tokens converted with OpenRouter’s public July–August 2025 pricing snapshot; baselines include GPT-4o, fine-tuned judges, JailJudge, and prompt-based judges.

Limitations

  • Stated: HAJailBench is anchored to 100 behaviors and a finite set of attack methods, so distribution shifts may affect outcomes; residual subjectivity remains in borderline cases; final labels use a three-band integration rule rather than a pure majority vote on the raw 1–10 scale; and structured topic scaffolding may constrain detection of novel harms outside the selected five aspects.
  • Stated: reported cost numbers are pricing-snapshot estimates based on public OpenRouter input/output token rates rather than hardware-normalized compute measurements; free-only endpoints such as Qwen3-4B require a same-scale paid proxy, and some models such as ShieldGemma lack matching public listings.
  • Evident from the text: the benchmark uses a curated, non-exhaustive sampling design and 11 target-model configurations, so it does not cover all attack–target combinations; the paper also notes automated safety judges are fallible and better understood as tools for scalable triage and comparative evaluation than as substitutes for human oversight in high-stakes settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2511.06396

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 3 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Across all Qwen backbones, the Multi-Agent Judge consistently outperforms the matched JailJudge baseline.

    “Across all Qwen backbones, our Multi-Agent Judge consistently outperforms the matched JailJudge baseline.”
    Quote in the paperSubstantial claim
  • Findingown data

    Multi-turn strategies such as Actor and X-Teaming surpass 0.50 ASR.

    “and multi-turn strategies such as Actor and X-Teaming surpass 0.50.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Relative to JailJudge, the framework improves consistently from 4B through 14B judge backbones.

    “Relative to JailJudge, our framework improves consistently from 4B through 14B judge backbones”
    Quote in the paperSubstantial claim
  • Findingown data

    Fine-tuned safety judges remain far below human agreement on HAJailBench, with the best of them reaching only κ = 0.3510.

    “Fine-tuned safety judges remain far below human agreement on HAJailBench, with the best of them reaching only κ = 0.3510.”
    Quote in the paperSubstantial claim
  • Findingown data

    Token-level attacks such as COLD and GCG remain below 0.07 ASR.

    “Token-level attacks such as COLD and GCG remain below 0.07 ASR”
    Quote in the paperSubstantial claim
  • May combine two claims. 33
  • Classified as probably not a substantial claim. 22

  • May combine two claims.

    Findingown data

    Relative to JailJudge on the same backbone, the framework gains 0.1786 κ points with slightly lower unit cost.

    “Relative to JailJudge on the same backbone, this is a gain of 0.1786 κ points with slightly lower unit cost.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The strongest configuration, Qwen3-14B with the debate framework, reaches κ = 0.7331, only 0.0258 below GPT-4o while using 46% of GPT-4o's estimated cost per query under the July-August 2025 pricing snapshot.

    “The strongest configuration, Qwen3-14B with our debate framework, reaches κ = 0.7331, only 0.0258 below GPT-4o while using 46% of GPT-4o's estimated cost per query under the July-August 2025 pricing snapshot.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    HAJailBench builds on 100 harmful goals from JBB-Behaviors and contains 11,100 reviewed instances drawn from a curated, non-exhaustive combination of attack strategies and target models.

    “HAJailBench builds on 100 harmful goals from JBB-Behaviors Chao et al. [2024] and currently contains 11,100 reviewed instances drawn from a curated, non-exhaustive combination of attack strategies and target models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chao et al. [2024]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingnone given

    HAJailBench contains 11,100 human-labeled responses collected from 100 harmful goals, 12 attack methods, and 11 target-model configurations under a curated, non-exhaustive sampling design.

    “HAJailBench contains 11,100 human-labeled responses collected from 100 harmful goals, 12 attack methods, and 11 target-model configurations under a curated, non-exhaustive sampling design”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The framework lets smaller judge backbones, such as Qwen3-14B, recover much of the accuracy of larger judges while maintaining a favorable cost-accuracy trade-off under the evaluated pricing snapshot.

    “This structure lets smaller judge backbones, such as Qwen3-14B, recover much of the accuracy of larger judges while maintaining a favorable cost-accuracy trade-off under the evaluated pricing snapshot.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On HAJailBench, the framework improves over matched small-model prompt baselines and prior multi-agent judges, while remaining more economical than GPT-4o under the evaluated pricing snapshot.

    “On HAJailBench, the framework improves over matched small-model prompt baselines and prior multi-agent judges, while remaining more economical than GPT-4o under the evaluated pricing snapshot.”
    • Also stated in: § 1 Introduction · plain
      “On HAJailBench, the framework outperforms matched small-model prompt baselines, fine-tuned judges, and prior multi-agent baselines in agreement with human labels, while remaining more economical than GPT-4o under the evaluated pricing snapshot.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The Multi-Agent Judge framework uses critic, defender, and judge agents that debate under a shared safety rubric.

    “we pair it with a Multi-Agent Judge framework in which critic, defender, and judge agents debate under a shared safety rubric”
    • Also stated in: § 1 Introduction · plain
      “We also propose a Multi-Agent Judge framework in which role-specific agents-critic, defender, and judge-debate under a shared safety rubric.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Removing alignment in the current 3-round setting lowers performance to κ = 0.7239, and the stripped-down zero-round variant without alignment, denoising, or refusal falls further to κ = 0.5200.

    “Removing alignment in the current 3-round setting lowers performance to κ = 0.7239, and the stripped-down zero-round variant without alignment, denoising, or refusal falls further to κ = 0.5200.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The benchmark can contain offensive, discriminatory, hateful, or otherwise disturbing model-generated content.

    “Because the benchmark is designed for safety evaluation, it can contain offensive, discriminatory, hateful, or otherwise disturbing model-generated content.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Semantic multi-turn attacks such as Actor and X-Teaming produce the strongest attack success rates and higher mean severity scores, whereas token-level methods remain much weaker across targets.

    “Semantic multi-turn attacks such as Actor and X-Teaming produce the strongest attack success rates and higher mean severity scores, whereas token-level methods remain much weaker across targets.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The GPT-4o and DeepSeek reference judges are sufficiently accurate to flag discrepancy-heavy cases for human review without introducing systematic bias into the benchmark's final labels.

  • May combine two claims.

    Findingown data

    Both the scaffolded pipeline and the debate stage contribute to the gain in judge agreement.

    “Our ablations further show that both the scaffolded pipeline and the debate stage contribute to the gain, with the Qwen3-14B system peaking at three rounds.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The GPT-4o and DeepSeek reference judges are sufficiently accurate to flag discrepancy-heavy cases for human review without introducing systematic bias into the benchmark's final labels.

  • May combine two claims.

    Findingown data

    Within the reported pricing snapshot, the SLM-first design achieves near-GPT-4o agreement at lower estimated unit cost.

    “Within the reported pricing snapshot, our SLM-first design achieves near-GPT-4o agreement at lower estimated unit cost (Table 4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Across model scales, the Multi-Agent Judge attains higher agreement with human-labeled ground truth than alternative multi-agent configurations and also generalizes better on JailJudge-ID.

    “Across model scales, the Multi-Agent Judge attains higher agreement with human-labeled ground truth than alternative multi-agent configurations (Table 4) and also generalizes better on JailJudge-ID (Table 6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Similar gains hold for smaller backbones, which points to an algorithmic benefit rather than only to base-model scaling.

    “Similar gains also hold for smaller backbones, which points to an algorithmic benefit rather than only to base-model scaling.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    For fairness, Qwen3-4B is priced with the paid 4B listing google/gemma-3-4b-it because the exact OpenRouter route is free-only.

    “For fairness, Qwen3-4B is priced with the paid 4B listing google/gemma-3-4b-it because the exact OpenRouter route is free-only.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    This disagreement makes a human-annotated benchmark necessary because without robust ground truth, improvements may reflect conformity to another judge rather than better safety assessment.

    “This disagreement also makes a human-annotated benchmark necessary: without robust ground truth, improvements may reflect conformity to another judge rather than better safety assessment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    This heterogeneity motivates a value-aligned, debate-based framework that makes competing judgments explicit before a final score is assigned.

    “This heterogeneity motivates a value-aligned, debate-based framework that makes competing judgments explicit before a final score is assigned.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Reference judges are used only to reduce annotation workload by helping queue cases for reinspection; annotators do not see model scores during final review, and all final labels are assigned by human annotators.

    “Reference judges are used only to reduce annotation workload by helping queue cases for reinspection; annotators do not see model scores during final review, and all final labels are assigned by human annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    HAJailBench couples attack diversity, target-model diversity, and fine-grained labels in the same dataset, which enables the later analysis of accuracy, calibration, and cost trade-offs.

    “More importantly, the benchmark couples attack diversity, target-model diversity, and fine-grained labels in the same dataset, which enables the later analysis of accuracy, calibration, and cost trade-offs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All annotators' participation was voluntary as part of the benchmark-construction workflow.

    “All annotators' participation was voluntary as part of the benchmark-construction workflow.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The reference baselines appear only in panel (a) so that panels (b) and (c) remain a controlled comparison among the backbone-matched judge families.

    “They appear only in panel (a) so that panels (b) and (c) remain a controlled comparison among the backbone-matched judge families.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The reported cost numbers are pricing-snapshot estimates based on public OpenRouter input/output token rates rather than hardware-normalized compute measurements; free-only endpoints such as Qwen3-4B require a same-scale paid proxy, and some models such as ShieldGemma currently lack matching public listings.

    “Finally, the reported cost numbers are pricing-snapshot estimates based on public OpenRouter input/output token rates rather than hardware-normalized compute measurements; free-only endpoints such as Qwen3-4B require a same-scale paid proxy, and some models such as ShieldGemma currently lack matching public listings.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Cases flagged for review are sent to two additional annotators for a human-only final review, with model scores hidden, and the benchmark's final scalar labels remain fully human-assigned before final integration.

    “Those cases are then sent to two additional annotators for a human-only final review; model scores are hidden during that stage, and the benchmark's final scalar labels remain fully human-assigned before final integration.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The three-band rule is a stability-oriented integration heuristic for borderline cases, not a replacement for the underlying reviewed scalar judgments themselves.

    “We therefore use the three-band rule as a stability-oriented integration heuristic for borderline cases, not as a replacement for the underlying reviewed scalar judgments themselves.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Automated safety judges are fallible and can reflect annotation or model biases, so they are better understood as tools for scalable triage and comparative evaluation than as substitutes for human oversight in high-stakes settings.

    “Automated safety judges are likewise fallible and can reflect annotation or model biases, so they are better understood as tools for scalable triage and comparative evaluation than as substitutes for human oversight in high-stakes settings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The stored benchmark is fully anonymized at the instance level.

    “The stored benchmark is fully anonymized at the instance level.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The benchmark still reflects human judgment under a shared rubric and cannot eliminate residual subjectivity in borderline cases.

    “the benchmark still reflects human judgment under a shared rubric and cannot eliminate residual subjectivity in borderline cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The reported ASR, κ, and cost values in the paper are benchmark-level point estimates computed on the full fixed evaluation set, unless otherwise noted.

    “Unless otherwise noted, the reported ASR, κ, and cost values in the paper are benchmark-level point estimates computed on the full fixed evaluation set.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The debate framework transfers beyond HAJailBench despite a modest cost increase.

    “which suggests that the debate framework transfers beyond HAJailBench despite a modest cost increase.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The shared-backbone summaries in Figure 4(b) and Figure 4(c) are means across the matched judge families shown in panel (a), whereas the tables report the exact point estimates for the corresponding evaluated configuration.

    “The shared-backbone summaries in Figure4(b)and Figure4(c) are means across the matched judge families shown in panel (a), whereas the tables report the exact point estimates for the corresponding evaluated configuration.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Over-debating introduces noise and error accumulation.

    “showing that over-debating introduces noise and error accumulation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    HAJailBench supports both target-model robustness analysis and judge calibration.

    “HAJailBench supports both target-model robustness analysis and judge calibration.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Three debate rounds strike the best balance between exploration and stability, whereas longer debates introduce noise and error accumulation in SLM agents.

    “The ablation results further show that three debate rounds strike the best balance between exploration and stability, whereas longer debates introduce noise and error accumulation in SLM agents (Table 7).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: SLM agents are more prone to bias, omission, and topic forgetting in longer interactions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Our method is the most structured judge in the comparison.

    “Our method is the most structured judge in the comparison.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    HAJailBench is still anchored to 100 behaviors and a finite set of attack methods, so distribution shifts may affect outcomes.

    “HAJailBench, while substantial at 11,100 instances, is still anchored to 100 behaviors and a finite set of attack methods, so distribution shifts may affect outcomes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Participation constituted informed consent within the internal author-annotation workflow.

    “Participation therefore constituted informed consent within this internal author-annotation workflow.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Structured topic scaffolding may constrain detection of novel harms that fall outside the selected five aspects.

    “Structured topic scaffolding may also constrain detection of novel harms that fall outside the selected five aspects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    JailJudge aggregates multiple judging agents and then uses additional voting and inference stages to produce a final decision.

    “Rather than making a single direct call, it aggregates multiple judging agents and then uses additional voting and inference stages to produce a final decision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    GPT-4o serves as a strong direct-judge reference, while LlamaGuard and ShieldGemma represent safety-specialized models whose native safe/unsafe outputs are mapped to the scalar convention used in this paper.

    “GPT-4o serves as a strong direct-judge reference, while LlamaGuard and ShieldGemma represent safety-specialized models whose native safe/unsafe outputs are mapped to the scalar convention used in this paper.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Debate increases token usage, but up to three rounds the accuracy gain remains substantial relative to the added cost.

    “Debate increases token usage, but up to three rounds the accuracy gain remains substantial relative to the added cost.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The suspicious band (5-6) is intentionally narrower than the outer bands because borderline harmfulness is the most subtle regime and tends to produce the least stable judgments.

    “The suspicious band is intentionally narrower than the outer bands because borderline harmfulness is the most subtle regime and tends to produce the least stable judgments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The ablation results demonstrate that both debate and scaffolded judging matter.

    “The ablation results demonstrate that both debate and scaffolded judging matter.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The scale of manual annotation in HAJailBench is large enough to expose systematic judge failures while still small enough for close human review.

    “This scale of manual annotation is large enough to expose systematic judge failures while still small enough for close human review.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    A structured, value-aligned debate framework improves safety judgment accuracy while keeping inference cost within a practical range for the tested setting.

    “Our results indicate that a structured, value-aligned debate framework improves safety judgment accuracy while keeping inference cost within a practical range for the tested setting.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The framework peaks at three rounds, where the accuracy gains remain large relative to the extra computation.

    “the framework peaks at three rounds, where the accuracy gains remain large relative to the extra computation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    JailJudge is the closest prior multi-agent baseline.

    “JailJudge is the closest prior multi-agent baseline.”
    Quote in the paperNot a substantial claim
46 of 46

  • The paper relies on this without support.

    Human annotation is the most reliable standard for safety evaluation.

    “Human annotation remains the most reliable standard”

  • The paper relies on this without support.

    Existing evaluation approaches struggle with adversarial prompts whose harmful intent depends on context, indirection, or multi-turn framing.

    “These approaches struggle with adversarial prompts whose harmful intent depends on context, indirection, or multi-turn framing.”

  • The paper relies on this without support.

    Human annotation is expensive and difficult to scale.

    “but it is expensive and difficult to scale.”

  • The paper relies on this without support.

    Fine-tuned safety classifiers generalize poorly to novel prompts outside the training distribution.

    “their generalization degrades on novel prompts that fall outside the training distribution.”

  • The paper relies on this without support.

    Debate-style interaction can expose errors that single-pass prompting misses.

    “debate-style interaction can expose errors that single-pass prompting misses.”

  • The paper relies on this without support.

    SLMs are less reliable in long, ambiguous evaluations.

    “their behavior is less reliable in long, ambiguous evaluations.”

  • The paper relies on this without support.

    SLMs are cheaper to deploy and easier to control than larger models.

    “Relative to larger models, SLMs are cheaper to deploy and easier to control”

  • The paper relies on this without support.

    SLM agents are more prone to bias, omission, and topic forgetting in longer interactions.

    “SLM agents are more prone to bias, omission, and topic forgetting in longer interactions”

  • The paper relies on this without support.

    The GPT-4o and DeepSeek reference judges are sufficiently accurate to flag discrepancy-heavy cases for human review without introducing systematic bias into the benchmark's final labels.

    “Reinspection is triggered when the range across the primary human score, the GPT-4o judge, and the DeepSeek judge exceeds 2 points.These reference judges are used only to reduce annotation workload by prioritizing cases for review.”
9 of 9

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Curran As- sociates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/ file/70702e8cbb4890b4a467b984ae59828a-Paper-Conference.pdf.

Cited by

Sorted by citations

Authors
Lin, D., Shen, G., Yang, Z., Liu, T., Zhao, D., & Zeng, Y.
Affiliations
  • Chinese Academy of Sciences (CN)
  • Chinese University of Hong Kong (HK)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2511.06396
arXiv
2511.06396
OpenAlex
W7104869295
Semantic Scholar
03988f5c6f7f7c89912bc9bd29c2611c29f813d8
Cited by
0
References
27
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
multi-agent debate, LLM-as-a-judge, jailbreak attacks, value alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review