Brittlebench: Quantifying LLM robustness via prompt sensitivity

Romanou, Angelika; Ibrahim, Mark; Ross, Candace; Oktar, Kerem; Shaib, Chantal; Bell, Samuel J.; Ovalle, Anaelia; Bosselut, Antoine; Dodge, Jesse; Sinha, Koustuv; Williams, Adina

doi:10.48550/arxiv.2603.13285

PreprintCited by 24Topic ModelingAdversarial Robustness in Machine LearningNatural Language Processing Techniques

The DOI matches its DataCite record; 56 of the 56 references it cites were found in a registry or an index; 7 of the 22 sources its claims cite were checked for support.

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Open access

Romanou, Angelika, Ibrahim, Mark, Ross, Candace, Oktar, Kerem, Shaib, Chantal, Bell, Samuel J., Ovalle, Anaelia, Bosselut, Antoine, Dodge, Jesse, Sinha, Koustuv, Williams, Adina·2026·arXiv

DOI 10.48550/arxiv.2603.13285arXiv 2603.13285

Open this paper in CoChat — free account Open PDF

Abstract

Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.

AI snapshot

Written from the paper’s full text.

TL;DR

Brittlebench introduces a variance-decomposition framework and perturbation pipeline to quantify how much LLM benchmark performance varies due to semantics-preserving prompt changes rather than task difficulty. Across six MCQ benchmarks and frontier/open-weight models, prompt perturbations degrade accuracy by up to 12.83%, alter model rankings in 63% of cases, and account for up to half of total performance variance.

Key findings

Semantics-preserving perturbations degrade performance by as much as 12% overall, with prompt padding reaching up to 12.83% degradation in few-shot settings; GPT-5 shows the largest absolute declines among tested models, including a 5.53-point word-level drop and 4.02-point micro-average drop.
Even a single perturbation changes the relative ranking of the six open-weight models in 63% of cases; prompt padding with quotation marks and newlines yields the lowest rank correlations (zero-shot: 0.63–0.67; few-shot: 0.73), while spaces around punctuation or word splits are less disruptive (zero-shot: 0.87–0.91; few-shot: 0.95–0.96).
Perturbation-induced variance accounts for roughly half of total performance variance in most open-weight models and more than ~25% for commercial models; MMLU and GPQA are dominated by perturbation-induced variance, whereas TruthfulQA, LogiQA, MathQA, and ARC are driven more by item difficulty.
Few-shot prompting improves baseline accuracy but often increases brittleness; chain-of-thought boosts accuracy but reduces the perturbation-induced accuracy drop by only 0.41 percentage points (from 2.79% to 2.38%, a 17% relative reduction), and paired perturbations on Qwen3-8B/MMLU can compound to drops of up to ~45%.

Methods

The paper formalizes correctness as \(Y(D,P,R)\) and decomposes total variance into data-difficulty variance and brittleness variance; deterministic inference via lm-evaluation-harness removes the inference-variance term, yielding model-level and benchmark-level brittleness scores.
Brittlebench applies a taxonomy of semantics-preserving perturbations—word manipulation, prompt augmentation, prompt padding, paraphrasing, math perturbations, and code perturbations—across six MCQ benchmarks: MMLU, TruthfulQA, ARC, MathQA, LogiQA, and GPQA.
Models include commercial GPT-5 and Claude 4.5 Opus plus open-weight Llama3.1, Llama3.3, and Qwen3 families; open-weight models are evaluated via log-probabilities over answer options, commercial models via API answer-letter output, with over 1,800 inference runs, GPT-4o-generated paraphrases, human annotation, and an LLM judge.

Limitations

The paper states GPT-5 was run in minimal thinking mode due to computational constraints, so its results reflect a lower bound under perturbations; it also notes that benign paraphrasing effects may stem from LLM-generated text being easier or from self-bias.
The paper acknowledges its behavioral definition of semantic preservation may not align with model-internal tokenization or representations, and that CoT mitigation is small and uneven across benchmarks.
Evident from the text: empirical validation is limited to six multiple-choice benchmarks and a fixed set of models, so generalization to other task formats, open-ended generation, or broader model populations is not demonstrated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.13285

Cites

56 of 56 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 22 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

Premisecitation
Benchmark creators often fail to assess whether their benchmark's construct validity might be compromised by arbitrary format effects or prompt-related biases.
“Moreover, benchmark creators often fail to assess whether their benchmark's construct validity might be compromised by arbitrary format effects or prompt-related biases (Bean et al., 2025).”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Bean et al., 2025 source
  The “Justify construct validity” section says only 53.4% of reviewed benchmarks justify their validity. Section 5.2 recommends assessing format constraints, and section 5.3 notes that minor prompt variations can affect LLM results. Our copy does not establish how often creators specifically fail to assess format effects or prompt-related biases. (LLM, conf=0.82)
  Experimental — accuracy not yet measured
Premisecitation
It is often unclear whether reported performance improvements reflect genuine advances in model capability or optimization toward benchmark-specific artifacts.
“As a result, it is often unclear whether reported performance improvements reflect genuine advances in model capability, or simply optimization toward benchmark-specific artifacts (Gururangan et al., 2018;Geirhos et al., 2020;Gardner et al., 2021).”
Quote in the paperSubstantial claim2 of 3 sources checked
- cites Gururangan et al., 2018 source
- cites Geirhos et al., 2020
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Gardner et al., 2021 source
Findingown data
The magnitude of the ranking effect depends strongly on the type of perturbation.
“The magnitude of this effect depends strongly on the type of perturbation.”
Quote in the paperSubstantial claim
Findingown data
On MMLU, the performance of Claude 4.5 Opus drops ∼2% when subjected to simple, character-level typos.
“For example, on MMLU, the performance of Claude 4.5 Opus drops ∼2% when subjected to simple, character-level typos.”
Quote in the paperNot a substantial claim
Findingown data
Even a single semantics-preserving perturbation alters the relative ranking of models in 63% of cases.
“even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance.”
- Also stated in: § 5.1 Impact of Perturbations on Model Performance · hedged
  “Overall, introducing even a single perturbation changes the relative ranking of the six open-weight models in 63% of cases, indicating that model comparisons are often sensitive to prompt formulation.”
Quote in the paperSubstantial claim

Findingown data
Applying semantics-preserving perturbations to a suite of popular benchmarks, model performance degrades by as much as 12%.
“We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%.”
- Also stated in: § 1 Introduction · plain
  “we find that introducing semantics-preserving input perturbations can decrease performance by up to ∼12%.”
Quote in the paperSubstantial claim
Findingown data
Commercial frontier models are similarly susceptible to input perturbations.
“Commercial frontier models are similarly susceptible to input perturbations.”
Quote in the paperSubstantial claim
Findingown data
Semantics-preserving input perturbations can account for up to half of the performance variance for a given model.
“Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model.”
- Also stated in: § 5.2 Variance and Brittleness Attribution · plain
  “perturbation-induced variance constitutes a substantial fraction of total performance variability, accounting for roughly half of the observed variance in most open-weight models.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
Prompt padding leads to systematic degradation, particularly in few-shot settings, with up to 12.83% performance degradation.
“Prompt padding also leads to systematic degradation, particularly in few-shot settings, with up to 12.83% performance degradation, as shown in Table 4 in Appendix B.”
Quote in the paperSubstantial claim
Findingown data
Surface-form changes have the largest impact on model performance among perturbation types.
“All models and benchmarks exhibit sensitivity to perturbations; Surface-form changes have the largest impact.”
Quote in the paperSubstantial claim
Findingown data
Word-level manipulations, such as typos or spacing changes, consistently degrade performance, with accuracy drops of several points, even for the largest models tested.
“Word-level manipulations, such as typos or spacing changes, consistently degrade performance, with accuracy drops of several points, even for the largest models we test.”
Quote in the paperSubstantial claim
Findingown data
All models and benchmarks exhibit sensitivity to perturbations.
“All models and benchmarks exhibit sensitivity to perturbations; Surface-form changes have the largest impact.”
Quote in the paperSubstantial claim
Findingown data
Even simple surface-form perturbations induce nonlinear degradation as their prevalence increases.
“Even simple surface-form perturbations induce nonlinear degradation as their prevalence increases.”
Quote in the paperSubstantial claim
Findingown data
As the number of perturbations increases, both Llama3.1-8B and Qwen3-8B exhibit non-uniform degradation patterns across perturbation types.
“As the number of perturbations increases, both models exhibit non-uniform degradation patterns across perturbation types (Figure 2a).”
Quote in the paperSubstantial claim
Findingown data
Model rankings are sensitive to prompt perturbations.
“Model rankings are sensitive to prompt perturbations.”
Quote in the paperSubstantial claim
Findingown data
Across all models and evaluation settings, the impact of input perturbations depends strongly on which part of the prompt they target.
“Across all models and evaluation settings, the impact of input perturbations depends strongly on which part of the prompt they target (Table 1).”
Quote in the paperSubstantial claim
Findingown data
Perturbations consistently degrade the quality of reasoning traces, as assessed by an LLM-based judge (GPT-5).
“Beyond reducing accuracy, perturbations also consistently degrade the quality of reasoning traces, as assessed by an LLM-based judge (GPT-5; Appendix C), and higher reasoning quality is associated with higher accuracy.”
Quote in the paperSubstantial claim
Findingown data
Such drops exceed the absolute differences between models' baseline performances (≈2-10% in our sample).
“Such drops also exceed the absolute differences between models' baseline performances (≈ 2-10% in our sample).”
Quote in the paperSubstantial claim
Findingown data
Overall decrease in reasoning quality with perturbations is significant (β = -2.61, p < .001).
“The effects that we observe within most benchmarks also translate to significant overall decrease in reasoning quality with perturbations (β = -2.61,p < .001).”
Quote in the paperSubstantial claim
Findingown data
Compositional input perturbations substantially exacerbate model brittleness.
“Compositional input perturbations substantially exacerbate model brittleness.”
Quote in the paperSubstantial claim
Findingown data
Benchmark-level brittleness scores demonstrate pronounced heterogeneity in how benchmarks distribute performance variance between item difficulty and perturbation sensitivity.
“Benchmark-level brittleness scores (Figure 2b bottom) demonstrate pronounced heterogeneity in how benchmarks distribute performance variance between item difficulty and perturbation sensitivity.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
Such perturbations can account for up to half of total performance variance.
“Our results show that performance on widely used benchmarks can degrade by up to 12% under meaning-preserving input perturbations, and that such perturbations can account for up to half of total performance variance.”
Quote in the paperSubstantial claim
Findingown data
Higher reasoning quality is associated with higher accuracy.
“Beyond reducing accuracy, perturbations also consistently degrade the quality of reasoning traces, as assessed by an LLM-based judge (GPT-5; Appendix C), and higher reasoning quality is associated with higher accuracy.”
Quote in the paperSubstantial claim
Findingown data
Few-shot prompting leads to higher baseline performance than zero-shot prompting.
“Few-shot prompting leads to higher baseline performance, but more brittleness under perturbations.”
Quote in the paperSubstantial claim
Findingown data
The interaction between perturbations is order-dependent; same cumulative perturbations in different orders can yield substantially different outcomes.
“Notably, the interaction between perturbations is order-dependent, as evidenced by the asymmetry around the diagonal of the heatmap, indicating that the same, cumulative perturbations applied in different orders can yield substantially different outcomes.”
Quote in the paperSubstantial claim
Findingown data
This pattern is consistent across model scales: both larger models and smaller models exhibit perturbation-driven variance comparable to, or exceeding, variance attributable to item difficulty.
“Notably, this pattern is consistent across model scales: both larger models (e.g., Llama3.3-70B,Llama3.1-70B) and smaller models (e.g., Qwen3-4B, Llama3.1-8B)exhibit perturbation-driven variance comparable to, or exceeding, variance attributable to item difficulty.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
Across all evaluated models, perturbation-induced variance constitutes a substantial fraction of total performance variability, accounting for roughly half of the observed variance in most open-weight models.
“The modellevel brittleness scores (Figure 2b top) reveal that, across all evaluated models, perturbation-induced variance constitutes a substantial fraction of total performance variability, accounting for roughly half of the observed variance in most open-weight models.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
TruthfulQA, LogiQA, MathQA, and ARC show variance profiles largely driven by item difficulty, indicating stronger measurement stability under input perturbations.
“TruthfulQA, LogiQA, MathQA, and ARC show variance profiles largely driven by item difficulty, indicating stronger measurement stability under input perturbations.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
Few-shot prompting leads to more brittleness under perturbations than zero-shot prompting.
“Few-shot prompting leads to higher baseline performance, but more brittleness under perturbations.”
Quote in the paperSubstantial claim
Findingown data
Correct answers are associated with higher reasoning quality across benchmarks (β = 0.011, p < .001).
“Our analyses show that there is a significant and substantial relationship between reasoning quality scores and accuracy, with the correct answer being associated with higher quality reasoning across benchmarks (β = 0.011, p < .001).”
Quote in the paperSubstantial claim
Findingown data
MMLU and GPQA are dominated by perturbation-induced variance, with over half of their total variance attributable to sensitivity to input perturbations rather than underlying task difficulty.
“MMLU and GPQA are dominated by perturbation-induced variance, with over half of their total variance attributable to sensitivity to input perturbations rather than underlying task difficulty.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
Performance on widely used benchmarks can degrade by up to 12% under meaning-preserving input perturbations.
“Our results show that performance on widely used benchmarks can degrade by up to 12% under meaning-preserving input perturbations, and that such perturbations can account for up to half of total performance variance.”
Quote in the paperSubstantial claim
Findingown data
Combining perturbations rarely offsets performance loss; in most cases further degrades model accuracy rather than stabilizing it.
“Finally, we observe little evidence of systematic mitigation effects: combining perturbations rarely offsets performance loss, and in most cases further degrades model accuracy rather than stabilizing it.”
Quote in the paperSubstantial claim
Findingown data
Zero-shot prompting yields lower absolute accuracy than few-shot prompting.
“Zero-shot prompting yields lower absolute accuracy but is sometimes more stable under noisy conditions due to its simpler prompt structure.”
Quote in the paperSubstantial claim
Findingown data
Benchmarks differ substantially in whether model performance variability is driven by task difficulty or by sensitivity to input perturbations.
“Benchmarks differ substantially in whether model performance variability is driven by task difficulty or by sensitivity to input perturbations.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Findingown data
Under perturbations, CoT reduces the accuracy drop by only 0.41 percentage points (from 2.79% to 2.38%), corresponding to a 17% relative reduction.
“Moreover, higher accuracy does not translate into commensurate robustness: under perturbations, CoT reduces the accuracy drop by only 0.41 percentage points (from 2.79% to 2.38%), corresponding to a 17% relative reduction.”
Quote in the paperSubstantial claim
Findingown data
Perturbations commonly reduce reasoning quality across benchmarks and perturbations.
“Our analyses show that such drops are common across benchmarks and perturbations (see Figure 6).”
Quote in the paperSubstantial claim
Findingown data
Evaluating models on perturbed benchmarks reveals substantially richer performance variability than evaluation on the original benchmarks alone.
“Evaluating models on perturbed benchmarks reveals substantially richer performance variability than evaluation on the original benchmarks alone (Figure 1 bottom).”
Quote in the paperSubstantial claim
Findingown data
CoT substantially outperforms non-CoT prompting, producing correct answers in 2.4× more cases than those in which it introduces new errors (1.7% error rate with CoT versus 4% without).
“As shown in Appendix C, CoT substantially outperforms non-CoT prompting, producing correct answers in 2.4× more cases than those in which it introduces new errors (1.7% error rate with CoT versus 4% without).”
Quote in the paperSubstantial claim
Study design and conductcitation
In the evaluation setup, inference is deterministic because performance is assessed based on output log-prob, so the inference variance term vanishes.
“In our evaluation setup, inference is deterministic, since we are using an evaluation framework (Gao et al., 2024) that assesses performance based on the output log-prob of the model; for any fixed pair (D, P ), the model performance is fully determined and therefore the inference variance term vanishes.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Gao et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Interpretationown data
Surface-form noise remains a persistent challenge that is not fully mitigated by scale alone.
“This indicates that surface-form noise remains a persistent challenge that is not fully mitigated by scale alone.”
Quote in the paperSubstantial claim
Premisereasoning
Arena-style evaluations treat prompt variation as incidental rather than a controlled factor of analysis, leaving prompt robustness as a largely unresolved dimension of model evaluation.
“Although the arena-style approaches broaden the distribution of evaluated inputs, they treat prompt variation as incidental rather than a controlled factor of analysis, leaving prompt robustness as a largely unresolved dimension of model evaluation.”
Quote in the paperSubstantial claim
Premisecitation
No principled framework exists for quantifying how much of the observed performance variability comes from the explicit formulation of prompts.
“Even though format sensitivity is well documented, no principled framework exists for quantifying how much of the observed performance variability comes from the explicit formulation of prompts (Wu et al., 2025).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Wu et al., 2025
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Study design and conductown data
Cosine similarity scores between original benchmark data points and their perturbed versions are consistently high across all 15 perturbation types, indicating minimal semantic difference.
“We observe consistently high cosine similarity scores between baseline examples and their corresponding perturbations, indicating minimal semantic difference.”
Quote in the paperSubstantial claim
Premisecitation
Widely adopted benchmarks may propagate their own inductive biases stemming from factors such as multiple-choice question formats, fixed reasoning templates, or overly narrow or broad task definitions.
“they may propagate their own inductive biases, stemming from factors such as multiple-choice question formats (Gupta et al., 2024), fixed reasoning templates, or overly narrow or broad task definitions (Raji et al., 2021).”
Quote in the paperSubstantial claim0 of 2 sources checked
- cites Gupta et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Raji et al., 2021
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Interpretationown data
Differences in model performance across tasks are not solely governed by intrinsic task difficulty but are strongly influenced by robustness to input variations.
“Consequently, differences in model performance across tasks are not solely governed by intrinsic task difficulty but are strongly influenced by robustness to input variations.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Interpretationreasoning
Zero-shot prompting is sometimes more stable under noisy conditions due to its simpler prompt structure.
“Zero-shot prompting yields lower absolute accuracy but is sometimes more stable under noisy conditions due to its simpler prompt structure.”
Quote in the paperSubstantial claim
Premisecitation
Even minor perturbations to formatting, such as changing casing, spaces, or separators, can have significant impacts on model performance.
“Sclar et al. (2024) shows that even minor perturbations to formatting, such as changing the casing, spaces, or separators in the prompt, can have significant impacts on model performance.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Sclar et al. (2024)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Study design and conductown data
Agreement between the LLM judge and the human reference was substantial, with a quadratically weighted Cohen's κ of 0.678.
“Agreement between the LLM judge and the human reference was substantial, with a quadratically weighted Cohen's κ of 0.678, indicating strong ordinal consistency.”
Quote in the paperSubstantial claim
Interpretationreasoning
Benchmark scores can increasingly capture how robust a model is to input variations unless perturbation-aware evaluations are explicitly incorporated.
“As a consequence, benchmark scores can increasingly capture how robust a model is to input variations unless perturbation-aware evaluations are explicitly incorporated.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Study design and conductreasoning
The variance decomposition used in the main text is equivalent to a random-effects ANOVA with perturbations nested within data items.
“We show that the variance decomposition used in the main text is equivalent to a random-effects ANOVA with perturbations nested within data items.”
Quote in the paperNot a substantial claim
- The authors state this as an assumption: The random effects α_d and β_p(d) have zero mean.
- The authors state this as an assumption: In the evaluation setting, inference is deterministic, so no residual noise term is present.
Premisecitation
Chain-of-thought prompting is known to lead to more deliberative responses.
“To examine whether reasoning can provide similar mitigation in LLMs, we leverage chain-of-thought (CoT) prompting, which is known to lead to more deliberative responses (Wei et al., 2022).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Wei et al., 2022 source
Study design and conductown data
Models exhibit broadly consistent trends across perturbation groups, lending support to the validity of the perturbation groupings used in the evaluation.
“Across these results, models exhibit broadly consistent trends, lending support to the validity of the perturbation groupings used in our evaluation.”
Quote in the paperNot a substantial claim
Study design and conductown data
The judge's scores exhibited a moderate monotonic association with human judgments, as measured by Spearman's rank correlation of 0.447.
“Finally, the judge's scores exhibited a moderate monotonic association with human judgments, as measured by Spearman's rank correlation of 0.447, indicating that the judge largely preserved the relative ordering of paraphrase quality as perceived by humans.”
Quote in the paperSubstantial claim
Interpretationown data
Reasoning can mitigate brittleness only when the reasoning process itself remains intact.
“Taken together, these results suggest that reasoning can mitigate brittleness only when the reasoning process itself remains intact.”
Quote in the paperSubstantial claim
Interpretationown data
There is a consistent trade-off between prompting strategy and robustness.
“We observe a consistent trade-off between prompting strategy and robustness.”
Quote in the paperSubstantial claim
Interpretationown data
Robustness is a distinct axis of model behavior that cannot be inferred from mean performance alone.
“These findings underscore that robustness is a distinct axis of model behavior that cannot be inferred from mean performance alone.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Interpretationown data
Reasoning can mitigate the effects of perturbations primarily when it is not itself compromised by the perturbation.
“Taken together with the previous result that perturbations lower reasoning quality, these analyses support a simple mechanism: reasoning can mitigate the effects of perturbations primarily when it is not itself compromised by the perturbation.”
Quote in the paperSubstantial claim
Interpretationown data
CoT specifically mitigates robustness.
“These results show that, beyond merely boosting overall accuracy, CoT specifically mitigates robustness.”
Quote in the paperSubstantial claim

Probably not a substantial claim. 22
It may state two separate claims as one. 40
The quote only partly states this claim. 5

Probably not a substantial claim. The paper’s conclusions may not depend on it.
Interpretationcitation
The approximately unchanged performance on LLM-generated paraphrases might be due to a generalized form of self-bias, whereby an LLM-as-judge prefers its own generated text outputs.
“However, the fact that performance on LLM-generated paraphrases remains approximately the same might nevertheless be the result of a generalized form of the so-called”
Quote in the paperNot a substantial claim1 of 3 sources checked
- cites Panickssery et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Xu et al., 2024
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Spiliopoulou et al., 2025 source
Probably not a substantial claim. It may state two separate claims as one. The quote only partly states this claim.
Premisereasoning
The paper's definition of semantics-preservation relies to some extent on the intent of a query, but unlike Chatterjee et al. 2024, it is built on the gold answer and model behavior rather than internal processes or representations.
“LikeChatterjee et al. 2024, our definition relies to some extent on the”
Quote in the paperSubstantial claim1 of 1 sources checked
- cites Chatterjee et al. 2024 source
  Section 3.1 defines intent-aligned prompts by their shared underlying goal or meaning, supporting the comparison about query intent. The Introduction says POSIX does not use the ground-truth response, but measures sensitivity through the likelihood of generated responses across prompt variants. That is model behavior, so the contrast portraying Chatterjee et al. as relying on internal processes or representations rather than model behavior is not supported. (LLM, conf=0.83)
  Experimental — accuracy not yet measured
Probably not a substantial claim.
Premisecitation
LLMs are often not robust to semantically-preserving structural perturbations.
“Prior work suggests LLMs are often not robust to semantically-preserving structural perturbations.”
Quote in the paperSubstantial claim2 of 3 sources checked
- cites Gu et al. (2023)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Pezeshkpour and Hruschka (2024) source
- cites Haller et al. (2025) source
It may state two separate claims as one.
Findingown data
Claude 4.5 Opus shows performance drops comparable to open-weight models under word-level and context-augmentation perturbations, while remaining relatively robust to prompt-padding perturbations.
“Claude 4.5 Opus shows performance drops comparable to open-weight models that are observed under word-level and context-augmentation perturbations, while remaining relatively robust to prompt-padding perturbations.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
In both zero-shot and few-shot settings, prompt padding with quotation marks and newlines yields the lowest rank correlations (zero-shot: 0.63-0.67; few-shot: 0.73), frequently altering model rankings compared to the baseline.
“In both zero-shot and few-shot settings, prompt padding with quotation marks and newlines yields the lowest rank correlations (zero-shot: 0.63˘0.67;few-shot: 0.73), frequently altering model rankings compared to the baseline.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
LLM-generated paraphrasing perturbations are comparatively benign and occasionally improve performance in open-weight models.
“In contrast, LLM-generated paraphrasing perturbations are comparatively benign and occasionally improve performance in open-weight models.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Adding spaces around punctuation or splitting words produces much higher rank correlations (zero-shot: 0.87-0.91; few-shot: 0.95-0.96), indicating minimal impact on relative ordering.
“In contrast, adding spaces around punctuation or splitting words produces much higher rank correlations (zero-shot: 0.87˘0.91;few-shot: 0.95˘0.96),indicating minimal impact on relative ordering.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
GPT-5 (minimal thinking mode) exhibits the largest absolute performance declines, despite achieving high overall baseline accuracy.
“In contrast, GPT-5 (minimal thinking mode) exhibits the largest absolute performance declines, despite achieving high overall baseline accuracy.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Despite their superior average performance compared to open-weight models, commercial frontier models remain sensitive to input perturbations, incurring absolute performance drops comparable in magnitude to those observed for open-weight models.
“Overall, despite their superior average performance compared to open-weight models, commercial frontier models remain sensitive to input perturbations, incurring absolute performance drops comparable in magnitude to those observed for open-weight models.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Rankings are generally more stable in the few-shot setting, but sensitivity remains non-trivial, as even the least impactful perturbations still change rankings in roughly half of the evaluated tasks.
“Rankings are generally more stable in the few-shot setting, however, sensitivity remains non-trivial, as even the least impactful perturbations still change rankings in roughly half of the evaluated tasks.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Chain-of-thought (CoT) boosts accuracy but only weakly mitigates brittleness to prompt perturbations.
“Chain-of-thought (CoT) boosts accuracy but only weakly mitigates brittleness to prompt perturbations.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
GPT-5 achieves an average accuracy of 73.97% on the baseline and an average drop of 5.01% under prompt padding perturbations across the six benchmarks (MMLU, ARC, TruthfulQA, MathQA, GPQA, LogiQA).
“| GPT-5 | 73.97 | 69.14 | 4.83 | 68.31 | 5.66 | 69.42 | 4.55 | 68.96 | 5.01 |”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Typos lead to a largely monotonic decline in performance, while prompt-padding perturbations show more irregular behavior.
“Typos lead to a largely monotonic decline in performance, while, in contrast, prompt-padding perturbations show more irregular behavior.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
The paraphrases produced by GPT-4o were generally of high quality, with approximately 98% of ratings in the 4-5 range.
“Because the paraphrases produced by GPT-4o were generally of high quality, the observed score distribution was strongly skewed toward the upper end of the scale, with approximately 98% of ratings in the 4-5 range.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
The drop in performance from baseline to perturbed conditions was 2.38% for CoT (from 93.02% to 90.64%) and 2.79% for Non-CoT (from 92.00% to 89.21%).
“Note that the practical size of this effect is quite small (the drop in performance from baseline to perturbed conditions was 2.38% for CoT (from 93.02% to 90.64%) and 2.79% for Non-CoT (from 92.00% to 89.21%).”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
These trends are consistent across both models, although Qwen3-8B generally exhibits larger fluctuations, suggesting greater brittleness.
“These trends are consistent across both models, although Qwen3-8B generally exhibits larger fluctuations, suggesting greater brittleness.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Larger models are more robust on average to perturbations, particularly paraphrasing and contextual changes, but remain vulnerable to word-level perturbations and prompt augmentation.
“Model scale mitigates, but does not eliminate, sensitivity to perturbations: larger models are more robust on average, particularly to paraphrasing and contextual changes, yet remain vulnerable to word-level perturbations and prompt augmentation.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Across 83,870 questions, CoT prompting significantly outperformed Non-CoT (McNemar's χ2 = 787.71, p < 0.001), with CoT answering correctly on 3,413 questions (4.1%) where Non-CoT failed, compared to only 1,454 questions (1.7%) where Non-CoT succeeded and CoT failed.
“Across 83,870 questions, CoT prompting significantly outperformed Non-CoT (McNemar's χ 2 = 787.71,p < 0.001), with CoT answering correctly on 3,413 questions (4.1%)where Non-CoT failed, compared to only 1,454 questions (1.7%) where Non-CoT succeeded and CoT failed.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
CoT improved overall accuracy, though with variance across benchmarks.
“We found that CoT improved overall accuracy, though with variance across benchmarks (see Figure 5).”
Quote in the paperNot a substantial claim
It may state two separate claims as one.
Findingown data
While baseline benchmarks produce relatively compact performance distributions, the perturbed counterparts expose a wider spread in accuracy, uncovering sensitivity that is otherwise masked by single-format evaluations.
“While baseline benchmarks produce relatively compact performance distributions, the perturbed counterparts expose a wider spread in accuracy, uncovering sensitivity that is otherwise masked by single-format evaluations.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
The interaction effect between CoT and perturbation is significant (β = .007, p = .006), indicating CoT mitigates brittleness.
“If CoT mitigates brittleness, the interaction effect should be significant, and this prediction is borne out in our data (β = .007,p = .006).”
Quote in the paperSubstantial claim
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
Findingown data
For zero-shot, padding with new lines has the lowest mean Spearman rank correlation (0.6327) across tasks.
“Padding with new lines 0.6327”
Quote in the paperNot a substantial claim
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
Findingown data
For few-shot, padding with quotes has the lowest mean Spearman rank correlation (0.6735) across tasks.
“Padding with quotes 0.6735”
Quote in the paperNot a substantial claim
It may state two separate claims as one.
Findingown data
Claude 4.5 Opus achieves an average accuracy of 78.64% on the baseline and an average drop of 0.13% under prompt padding perturbations across the six benchmarks (MMLU, ARC, TruthfulQA, MathQA, GPQA, LogiQA).
“| Claude 4.5 Opus | 78.64 | 78.56 | 0.08 | 78.26 | 0.38 | 78.71 | -0.07 | 78.51 | 0.13 |”
Quote in the paperSubstantial claim
The quote only partly states this claim.
Findingown data
For few-shot, word split has the highest mean Spearman rank correlation (0.9673) across tasks.
“Word split 0.9673”
Quote in the paperNot a substantial claim
It may state two separate claims as one.
Findingown data
Commercial models display lower absolute variance overall, yet prompt sensitivity still accounts for more than ∼25% of their total variance.
“Commercial models display lower absolute variance overall, yet prompt sensitivity still accounts for more than ∼ 25% of their total variance.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
It may state two separate claims as one.
Findingown data
Spaces and new lines are comparatively benign and often result in stable or mildly fluctuating performance, while quotation marks induce sharper drops at higher intensities.
“Spaces and new lines are comparatively benign and often result in stable or mildly fluctuating performance, while quotation marks induce sharper drops at higher intensities.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Model scale mitigates but does not eliminate sensitivity to perturbations.
“Model scale mitigates, but does not eliminate, sensitivity to perturbations: larger models are more robust on average, particularly to paraphrasing and contextual changes, yet remain vulnerable to word-level perturbations and prompt augmentation.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
CoT gains are uneven across benchmarks, with limited improvements on MathQA and TruthfulQA.
“However, these gains are uneven across benchmarks, with limited improvements on tasks such as MathQA and TruthfulQA.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Findingown data
Variance increases across model scales, indicating that robustness to input variations is not uniform, even when average performance appears stable.
“Variance increases across model scales, indicating that robustness to input variations is not uniform, even when average performance appears stable.”
Quote in the paperSubstantial claim
Probably not a substantial claim. The paper’s conclusions may not depend on it. The quote only partly states this claim.
Findingown data
For zero-shot, adding punctuation spaces has the highest mean Spearman rank correlation (0.9102) across tasks.
“Add punctuation spaces 0.9102”
Quote in the paperNot a substantial claim
It may state two separate claims as one.
Findingown data
Combining perturbations generally amplifies performance degradation, with some combinations leading to drops of up to ~45%, far exceeding individual perturbations.
“Combining perturbations generally amplifies performance degradation (Figure 3), with some combinations leading to drops of up to ∼ 45%, far exceeding the effect of individual perturbations.”
Quote in the paperSubstantial claim
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Study design and conductreasoning
Due to computational constraints, many evaluation pipelines deploy GPT-5 in minimal thinking mode; the results therefore reflect a lower bound on its performance under perturbations.
“Notably, due to computational constraints, many evaluation pipelines deploy GPT-5 in minimal thinking mode; our results therefore reflect a lower bound on its performance under perturbations.”
Quote in the paperNot a substantial claim
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Interpretationnone given
The Brittlebench approach can apply to any text-based benchmark in principle.
“The Brittlebench approach can apply to any text-based benchmark in principle.”
Quote in the paperNot a substantial claim
Probably not a substantial claim.
Study design and conductown data
For paraphrasing-based perturbations, the resulting outputs were evaluated using both human annotation and an LLM-based judge with GPT-4o.
“For paraphrasing-based perturbations, we use GPT-4o (Achiam et al., 2023) to generate paraphrases of each input question and evaluate the resulting outputs using both human annotation and an LLM-based judge with GPT-4o.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Achiam et al., 2023
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Interpretationnone given
A variant can be deemed semantically-equivalent to the original according to the behavioral definition, but a particular model may nevertheless tokenize the two examples completely differently.
“For example, a variant can be deemed semantically-equivalent to the original according to the definition, but a particular model may nevertheless tokenize the two examples completely differently.”
Quote in the paperNot a substantial claim
Probably not a substantial claim.
Study design and conductreasoning
When applying typos to benchmarks containing equations or mathematical expressions, perturbations are explicitly restricted from modifying these components.
“For example, when applying typos to benchmarks containing equations or mathematical expressions, perturbations are explicitly restricted from modifying these components.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Interpretationreasoning
Relying solely on original benchmarks can overestimate model reliability, whereas incorporating perturbed evaluations enables a more comprehensive characterization of model behavior under realistic input conditions.
“Consequently, relying solely on original benchmarks can overestimate model reliability, whereas incorporating perturbed evaluations enables a more comprehensive characterization of model behavior under realistic input conditions.”
Quote in the paperSubstantial claim
- Rests on an assumption: The semantics-preserving perturbations used in Brittlebench are representative of the noise and variability in real-world user inputs.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation
As one progresses deeper into model internals, further differences arising from formatting variations can be observed.
“As one progresses deeper into model internals, further differences arising from formatting variations can be observed (Davidson et al., 2025).”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Davidson et al., 2025
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim.
Premisecitation
Paraphrased instructions cause consistent performance degradation.
“Sun et al. (2024) systematically evaluate prompt sensitivity to humanwritten, paraphrased instructions over MMLU (Hendrycks et al., 2021) and BBL (BIG-bench, 2023), and find consistent performance degradation on the paraphrased instructions.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Sun et al. (2024)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
Interpretationreasoning
For the brittleness-dominant benchmarks, many models have already reached similar levels of semantic competence on these tasks, causing surface-form robustness to become the main factor distinguishing them.
“For the brittleness-dominant benchmarks, this suggests that many models have already reached similar levels of semantic competence on these tasks, causing surface-form robustness to become the main factor distinguishing them.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
It may state two separate claims as one.
Study design and conductown data
Raw exact agreement between the judge and the rounded mean human score was 0.833, and mean absolute difference was 0.244.
“This alignment is further reflected by a high raw exact agreement of 0.833 between the judge and the rounded mean human score, and a low mean absolute difference of 0.244 between the judge score and the mean human rating, suggesting that disagreements were generally small in magnitude.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Interpretationown data
Increased scale and few-shot prompting improve performance but do not guarantee robustness.
“Overall, these results highlight that increased scale and few-shot prompting improve performance but do not guarantee robustness.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Study design and conductown data
The judge's multiple dimensions were highly correlated (all r ≥ .9), and the overall quality score had the highest pairwise correlations with the other scores (all r ≥ .94).
“As the judge outputs multiple dimensions, we examined the relationship between these to decide whether to analyze them independently, and found them all to be highly correlated (all r ≥ .9);we therefore decided to use the overall quality score (which had the highest pairwise correlations with the other scores; all r ≥ .94)as the reasoning score.”
Quote in the paperSubstantial claim
Probably not a substantial claim. The paper may give no reason why it holds.
Premisenone given
Applying word-level perturbations such as typos, word merges, and word splits can in some cases alter the surface form of an input question in ways that may inadvertently break semantic equivalence between the original and a perturbed variant.
“This being said, applying word-level perturbations such as typos, word merges, and word splits can, in some cases, alter the surface form of an input question in ways that may inadvertently break semantic equivalence between the original and a perturbed variant.”
Quote in the paperNot a substantial claim
It may state two separate claims as one.
Interpretationreasoning
The brittleness-dominant benchmark result does not indicate a flaw in the benchmark itself; instead, it reflects a saturation effect under modern models.
“Importantly, this does not indicate a flaw in the benchmark itself; instead, it reflects a saturation effect under modern models.”
Quote in the paperSubstantial claim
- Rests on an assumption: The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone given
The approach provides universal model and evaluation support, is compatible with any model architecture, enables both probabilistic and generative evaluation, and is optimized for efficient execution even with vLLM.
“Our approach also provides universal model and evaluation support, making it compatible with any model architecture and enabling both probabilistic and generative evaluation, all optimized for efficient execution, even with vLLM.”
Quote in the paperNot a substantial claim
Probably not a substantial claim. It may state two separate claims as one.
Premisenone given
Brittlebench integrates flexible and scalable perturbation capabilities, allowing easy addition of custom perturbations, uses a large pool of existing perturbations and benchmarks, and has built-in support for robustness analysis and similarity metrics for each perturbation.
“To provide a comprehensive model assessment, Brittlebench integrates flexible and scalable perturbation capabilities, allowing for the easy addition of custom perturbations, while also using a large pool of existing ones and benchmarks, with built-in support for robustness analysis and similarity metrics for each perturbation.”
Quote in the paperSubstantial claim
Probably not a substantial claim.
Premisecitation
Deliberative reasoning and explanation allow people to discover and use generalizable principles as opposed to brittle heuristics.
“Cognitive scientists have found that deliberative reasoning and explanation allow people to discover and use generalizable principles as opposed to brittle heuristics (Williams and Lombrozo, 2010).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Williams and Lombrozo, 2010
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. It may be too vague to check against evidence.
Interpretationown data
Current evaluation practices have limitations.
“These findings underscore the limitations of current evaluation practices and motivate the adoption of robustness-aware evaluation frameworks to more faithfully assess model capability.”
Quote in the paperNot a substantial claim
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone given
The orchestration simplifies the evaluation workflow, maximizes throughput with advanced caching and profiling, and allows for single-call or single-script orchestration.
“The efficient and user-friendly orchestration further simplifies the evaluation workflow, maximizing throughput with advanced caching and profiling and allowing for single-call or single-script orchestration.”
Quote in the paperNot a substantial claim
Probably not a substantial claim.
Study design and conductreasoning
The perturbations implemented in the Brittlebench meta-evaluation framework are designed with careful, test-driven validation to minimize unintended meaning changes.
“To avoid this problem, the perturbations implemented in the Brittlebench meta-evaluation framework are designed with careful, test-driven validation to minimize unintended meaning changes.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Interpretationown data
Robustness is not only perturbation-type dependent but also intensity-dependent.
“Overall, our results demonstrate that robustness is not only perturbation-type dependent but also intensity-dependent, highlighting the importance of evaluating models beyond single-instance perturbations and toward more realistic, compounded noise settings.”
Quote in the paperSubstantial claim
It may state two separate claims as one.
Study design and conductown data
Inter-annotator agreement was quantified using quadratically weighted Cohen's κ (averaged across annotator pairs) of 0.781 and Krippendorff's α of 0.836 with an ordinal distance function.
“Inter-annotator agreement was quantified using metrics appropriate for ordinal data and skewed score distributions: quadratically weighted Cohen's κ (averaged across annotator pairs) of 0.781 and Krippendorff's α of 0.836 with an ordinal distance function.”
Quote in the paperNot a substantial claim
- Rests on an assumption: Range restriction reduces the stability and interpretability of chance-corrected agreement statistics.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Study design and conductown data
Raw exact agreement was 0.902.
“For completeness, we additionally report the raw exact agreement of 0.902, which is generally considered high.”
Quote in the paperNot a substantial claim
Probably not a substantial claim. It may state two separate claims as one.
Premisecitation
Reasoning primarily benefits performance when it is not itself compromised through distraction; semantic distractions lead to sharper drops in task performance than perceptual distractions in cognitive tasks.
“Studies of distraction show that reasoning primarily benefits performance when it is not itself compromised through distraction-for instance, semantic distractions lead to sharper drops in task performance than perceptual distractions in cognitive tasks (Kucwaj et al., 2022).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Kucwaj et al., 2022
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

56 of 56

The paper relies on this without support.
The semantics-preserving perturbations used in Brittlebench are representative of the noise and variability in real-world user inputs.
“Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs.”
The paper relies on this without support.
The selected models (GPT-5, Claude 4.5 Opus, Llama3.1, Llama3.3, Qwen3) are representative of frontier and state-of-the-art open-weight models.
“We benchmark the commercial models of GPT-5foot_0 and Claude 4.5 Opusfoot_1 , and state-of-the-art open-weight model families of Llama3.1,Llama3.3 (Dubey et al., 2024) and Qwen3 (Yang et al., 2025).”
The paper relies on this without support.
The variance decomposition's components (V_data and V_brittleness) correctly attribute observed performance variability to data difficulty and prompt sensitivity, respectively.
“We interpret V data as data difficulty variance, capturing stable differences in correctness across items when averaged over perturbations, and V brittleness as perturbation-attributable variance, capturing within-item instability under surface-form changes.”
The paper relies on this without support.
The 15 perturbation types constitute a sufficiently comprehensive taxonomy of semantics-preserving prompt perturbations.
“We observe consistently high cosine similarity scores between baseline examples and their corresponding perturbations, indicating minimal semantic difference.”
The authors state this as an assumption.
The random effects α_d and β_p(d) have zero mean.
“We treat both effects as random with zero mean.”
The authors state this as an assumption.
In the evaluation setting, inference is deterministic, so no residual noise term is present.
“In our evaluation setting, inference is deterministic, and thus no residual noise term is present.”
The paper relies on this without support.
Paraphrase-based perturbations mimic expected variability in real-world interactions with LLMs.
“Paraphrase-based perturbations mimic expected variability in real-world interactions with LLMs, and performance sensitivity raises concerns about the validity of model performance over popular benchmarks.”
The paper relies on this without support.
To a human user, low-level modeling decisions are irrelevant because the same output is expected.
“Moreover, to the human-in particular, to a user in the real world-such a low-level modeling decision is irrelevant, because the same output is expected.”
The paper relies on this without support.
Range restriction reduces the stability and interpretability of chance-corrected agreement statistics.
“Such range restriction is known to reduce the stability and interpretability of chance-corrected agreement statistics, despite high substantive agreement among annotators.”

9 of 9

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Llama3.3-70B padding accuracy is incompatible with its reported drop
Minor
The few-shot row reports a baseline of 62.84, a quotes accuracy of 659.25, and a drop of 3.59. Subtracting the reported accuracy from the baseline instead gives −596.41 percentage points.
Reported 3.59 · recomputed -596.41
Perturbation taxonomy says four categories but lists six
Minor
The sentence specifies four categories, but its list names six: word manipulation, prompt augmentation, prompt padding, paraphrasing, math perturbations, and code perturbations.
CoT's relative reduction in accuracy drop is not 17%
Minor
Reducing the accuracy drop from 2.79 to 2.38 percentage points is a 0.41-point reduction, or about 14.7% relative to 2.79, not 17%.
Reported 17 · recomputed 14.695341
Claude prompt-padding average differs between tables
Minor
Table 1 reports Claude's prompt-padding accuracy as 78.64, whereas Table 4 reports the average as 78.84 for the same baseline and model. Table 1's stated drop of −0.19 is also incompatible with its displayed accuracy of 78.64.

4 of 4

Qwen3 Technical Report
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.09388 · Cited by 17,283
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., & Hajishirzi, H. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1245 · Cited by 956
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Synthetic and Natural Noise Both Break Neural Machine TranslationPreprint
Belinkov, Y., & Bisk, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02173 · Cited by 614
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
The Llama 3 Herd of Models
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · cited 1×
“3 (Dubey et al., 2024) and Qwen3 (Yang et al., 2025).”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
Self-citation
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman, S. R., & Dahl, G. E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.385 · Cited by 206
The Role of Explanation in Discovery and Generalization: Evidence From Category Learning
Williams, J. J., & Lombrozo, T. (2010) · Cognitive Science · doi:10.1111/j.1551-6709.2010.01113.x · Cited by 190
Rephrase and Respond: Let Large Language Models Ask Better Questions for ThemselvesPreprint
Deng, Y., Zhang, W., Chen, Z., & Gu, Q. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04205 · Cited by 160
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151
Algorithmic monoculture and social welfare
Kleinberg, J., & Raghavan, M. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2018340118 · Cited by 147
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.04132 · Cited by 118
Perturbation Sensitivity Analysis to Detect Unintended Model Biases
Prabhakaran, V., Hutchinson, B., & Mitchell, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1578 · Cited by 115
State of What Art? A Call for Multi-Prompt LLM Evaluation
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., & Stanovsky, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00681 · Cited by 113
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., Mayne, H., Batzner, J., Foroutan Eghlidi, N., Schmitz, C., Korgul, K., Batra, H., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-0590 · Cited by 110
Self-citation
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Pezeshkpour, P., & Hruschka, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.130 · Cited by 85
Competency Problems: On Finding and Removing Artifacts in Language Data
Gardner, M., Merrill, W., Dodge, J., Peters, M., Ross, A., Singh, S., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.135 · Cited by 81
Self-citation
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.108 · Cited by 78
Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements
Voronov, A., Wolf, L., & Ryabinin, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.375 · Cited by 77
Perturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69
Self-citation
More Bang for Your Buck: Natural Perturbation for Robust Question Answering
Khashabi, D., Khot, T., & Sabharwal, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.12 · Cited by 60
Large Language Models Understand and Can be Enhanced by Emotional Stimuli
Li, C., Wang, J., Zhang, Y., Zhu, K., Hou, W., Lian, J., Luo, F., Yang, Q., & Xie, X. (2023) · arXiv · doi:10.48550/arxiv.2307.11760 · Cited by 59
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06709 · Cited by 51
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47
Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks
Thrush, T., Tirumala, K., Gupta, A., Bartolo, M., Rodríguez, P., Kane, T., Rojas, W. G., Mattson, P. R., Williams, A., & Kiela, D. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.17 · Cited by 43
Self-citation
Tailor: Generating and Perturbing Text with Semantic Controls
Ross, A., Wu, T., Peng, H. M., Peters, M., & Gardner, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.228 · Cited by 41
Deciding to be authentic: Intuition is favored over deliberation when authenticity matters
Oktar, K., & Lombrozo, T. (2022) · Cognition · doi:10.1016/j.cognition.2022.105021 · Cited by 40
Self-citation
Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement
Xu, W., Zhu, G., Zhao, X., Pan, L., Li, L., & Wang, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.826 · Cited by 34
POSIX: A Prompt Sensitivity Index For Large Language Models
Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.852 · Cited by 34
Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation ModelsPreprint
Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.05176 · Cited by 31
Robustness of Learning from Task Instructions
Gu, J., Zhao, H., Xu, H., Nie, L., Mei, H., & Yin, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.875 · Cited by 21
On the Worst Prompt Performance of Large Language Models
Cao, B., Cai, D., Zhang, Z., et al. (2024) · neural information processing systems · doi:10.52202/079017-2205 · Cited by 21
VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation
Qian, K., Wan, S., Tang, C., Wang, Y., Zhang, X., Chen, M., & Zhou, Y. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.946 · Cited by 19
reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed InputsPreprint
Wu, Z., Yasunaga, M., Cohen, A., Kim, Y., Celikyilmaz, A., & Ghazvininejad, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.11751 · Cited by 19
Prompt Perturbation Consistency Learning for Robust Language Models
Qiang, Y., Nandi, S., Mehrabi, N., Steeg, G. V., Kumar, A., Rumshisky, A., & Galstyan, A. (2024) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2024.findings-eacl.91 · Cited by 19
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
Habba, E., Arviv, O., Itzhak, I., Perlitz, Y., Bandel, E., Choshen, L., Shmueli-Scheuer, M., & Stanovsky, G. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.611 · Cited by 16
LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance
Haller, P., Ibrahim, M., Kirichenko, P., Sagun, L., & Bell, S. J. (2025) · arXiv · doi:10.48550/arxiv.2510.11905 · Cited by 15
Self-citation
Language model acceptability judgements are not always robust to context
Sinha, K., Gauthier, J., Mueller, A., Misra, K., Fuentes, K., Lévy, R., & Williams, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.333 · Cited by 12
Self-citation
Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., & Griffiths, T. L. (2024) · arXiv · doi:10.48550/arxiv.2410.21333 · Cited by 6
Various sources of distraction during analogical reasoning
Kucwaj, H., Ociepka, M., & Chuderski, A. (2022) · Memory & Cognition · doi:10.3758/s13421-022-01285-3 · Cited by 6
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical ReasoningPreprint
Jian, L., Cui, L., Liu, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.08124 · Cited by 5
Enhancing LLM Robustness to Perturbed Instructions: An Empirical StudyPreprint
Agrawal, A., Alazraki, L., Honarvar, S., & Rei, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.02733 · Cited by 1
Measuring all the noises of LLM Evals
Wang, S. I. (2025) · arXiv · doi:10.48550/arxiv.2512.21326
GPT-4 Technical Report
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Do different prompting methods yield a common task representation in language models?
Davidson, G., Gureckis, T., Lake, B., & Williams, A. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-2380
Self-citation
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., Mayne, H., Batzner, J., Foroutan, N., Schmitz, C., Korgul, K., Batra, H., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.04703
Self-citation
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
Habba, E., Arviv, O., Itzhak, I., Perlitz, Y., Bandel, E., Choshen, L., Shmueli-Scheuer, M., & Stanovsky, G. (2025) · arXiv · doi:10.48550/arxiv.2503.01622
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts
Hashemi, H., Eisner, J., Rosset, C., Durme, B. V., & Kedzie, C. (2025) · arXiv · doi:10.48550/arxiv.2501.00274

Flagged first, then by citations

Research works printed but not linked: 9

BIG-bench. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. https://openreview.net/forum?id=uyTL5Bvosj.
Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. Changing answer order can decrease MMLU accuracy. In Proceedings of the Workshop on Datasets and Evaluators of AI Safety at AAAI 2025, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. Evaluating the instruction-following robustness of large language models to prompt injection. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 557-568, 2024.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 642-669, 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 02dec8877fb7c6aa9a79f81661baca7c-Abstract-Conference.html.
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. Ai and the everything in the whole wide world benchmark. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google-proof Q&A benchmark. In First Conference on Language Modeling, 2024.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=RIu5lyNXjT.
Jiuding Sun, Chantal Shaib, and Byron C Wallace. Evaluating the zero-shot robustness of instruction-tuned language models. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/ forum?id=g9diuvxN6D.

Sorted by citations

The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?Preprint
Hägele, A., Gema, A. P., Sleight, H., Perez, E., & Sohl‐Dickstein, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.23045 · Cited by 12
Efficient Benchmarking Is Just Feature Selection and Multiple RegressionPreprint
Bowyer, S., Locatelli, A., & Cao, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.25773 · Cited by 5
One prompt is not enough: Instruction Sensitivity Undermines Embedding Model Evaluation
Kostiuk, Y., & Enevoldsen, K. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.919 · Cited by 4
Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and PersistencePreprint
Zhao, J., Bhattacharjee, H., Korevaar, H., Radharapu, B., & El-Arini, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.12645 · Cited by 4
Online Shift Detection and Conformal Adaptation for Deployed Safety ClassifiersPreprint
Leong, J. W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.11949 · Cited by 3
Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language ModelsPreprint
Paruchuri, A., Koyejo, S., & Adeli, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.26079 · Cited by 3
Second-Order Response Laws for LLM Judges: Debiased Estimation of Prompt InstabilityPreprint
Feng, P., Meng, C., Qu, D., Zhang, Z., Liu, H., & Wu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.16253 · Cited by 3
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEvalPreprint
Janeiro, J. M., Videau, M., Caciolai, A., Piwowarski, B., Gallinari, P., & Barrault, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.10657 · Cited by 2
SCRuB: Social Concept Reasoning under Rubric-Based EvaluationPreprint
Watson-Daniels, J., Bhattacharjee, H., Wang, S., Handoko, B., Li, A., Ovalle, A., Pasupuleti, M. B., Ross, C., Sarma, V., Subramonian, A., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.06444 · Cited by 1
Self-citation
PRISM: A Benchmark for Programmatic Spatial-Temporal ReasoningPreprint
Zhang, Q., Wang, Y., Yang, R., Wu, L., Fan, J., Shu, Y., Zhang, J. M., Zhou, T., Li, H., Shi, R., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.19382 · Cited by 1
Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking
Messing, S. (2026) · arXiv · doi:10.48550/arxiv.2604.11581
Does Slightly Mean Somewhat? Measuring Vague Intensity Words in LLM Numeric ActionsPreprint
Tabach, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.21827
A Definition of Good Explanations and the Challenges Explaining LLM OutputsPreprint
Mahon, L., Ford, E., & Hackett, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.14838
Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?Preprint
Chlenski, P., Carmichael, Z., Warikoo, A., Shao, C.-T., Ye, Y., Yang, A., Miglani, V., & Bandi, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.32008
Structure-Guided Approaches for Robust Language Model Reasoning
Sclar, M. (2026) · ResearchWorks at the University of Washington (University of Washington)
CHART: A Harness-Rotation Curriculum for Harness-Robust Search AgentsPreprint
Zhang, X., Lin, Y.-C., Zhang, Z., Fetahu, B., & Chen, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.22247
CLARIN Annual Conference Proceedings 2026
Wynne, M., & Kontino, T. (2026) · U-PAD Unimc - Open Digital Publications (University of Macerata) · doi:10.5281/zenodo.22916660
Stochastic Execution under Mathematical Constraints: An Empirical Study of Trajectory Variance and Governance Invariant PreservationPreprint
Chen, W. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7531901
Latent Inference-Time Guidance of Time Series Foundation ModelsPreprint
Hashimoto-Cullen, C., Durand, A., Bozzi, L., Guedj, B., Goude, Y., & Corff, S. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38058
Quantization Effects on Tool-Failure Recovery Vary Across Prompts and Evaluation DesignsPreprint
Hu, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.07781
When the Trigger Fails: Metamorphic Testing of Tool-Calling AI Agents
de Amorim, W. F. (2026) · Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST) · doi:10.5753/sast.2026.32354
perturber: A Stateless Service for Semantics-Preserving Text PerturbationsPreprint
Tran, K.-N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.12667
On the Fragility of Tasks under Composition: Implicit Budgeting in Language ModelsPreprint
Li, C., Brockmoeller, T., Jegelka, S., Gupta, S., & Karalias, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.27190
How Reliable Is LLM Confidence in Hindi and Marathi? A Study of Calibration and Answer Stability
Deshmukh, S. S., & Ansari, D. M. A. (2026) · International Journal of Creative and Open Research in Engineering and Management · doi:10.55041/ijcope.v2i10.135

ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments
Lior, G., Habba, E., Levy, S., Caciularu, A., & Stanovsky, G. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.findings-emnlp.594 · Cited by 18
Evaluating Prompt Brittleness and Robustness in Large Language Models Across Classification, Reasoning, and Retrieval Tasks
Sanghvi, S. (2026) · doi:10.31979/etd.e6zn-8fbd
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
Habba, E., Arviv, O., Itzhak, I., Perlitz, Y., Bandel, E., Choshen, L., Shmueli-Scheuer, M., & Stanovsky, G. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.611 · Cited by 16
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
Hua, A., Tang, K., Gu, C., Gu, J., Wong, E., & Qin, Y. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1006 · Cited by 28
LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances
Kodavali, V. K., & Singh, R. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465013

Authors: Romanou, A., Ibrahim, M., Ross, C., Oktar, K., Shaib, C., Bell, S. J., Ovalle, A., Bosselut, A., Dodge, J., Sinha, K., & Williams, A.
Affiliations: École Polytechnique Fédérale de Lausanne (CH)
Northeastern University (US)
ORCID: Ross, Candace
Ovalle, Anaelia
Bosselut, Antoine
Sinha, Koustuv
Williams, Adina
Venue: arXiv
Publisher: arXiv
Published: 2026
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2603.13285
arXiv: 2603.13285
OpenAlex: W7138511917
Semantic Scholar: e0dd48e1a2f29b42114afbf7457e8d1c7e4ac2f9
Cited by: 24
References: 37
Citations by year: 2026: 24
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: cc-by-sa
creativecommons.org/licenses/by-sa/4.0/legalcode
Locations: arXiv (Cornell University) (submittedVersion, cc-by-sa)
arXiv (Cornell University) (cc-by-sa)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords: LLM robustness, prompt sensitivity, model brittleness, benchmark evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Abstract

Overview

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Verification

Claims the paper makes

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks