The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Changing Answer Order Can Decrease MMLU Accuracy
Abstract
As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, we use test accuracy averaged across multiple subtasks in order to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“they may propagate their own inductive biases, stemming from factors such as multiple-choice question formats (Gupta et al., 2024), fixed reasoning templates, or overly narrow or broad task definitions (Raji et al., 2021).”
From Brittlebench: Quantifying LLM robustness via prompt sensitivity (2026) · cites this paper as Gupta et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3018 · Cited by 131
Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H., Schuetze, H., & Hovy, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.816 · Cited by 89
Perturbation augmentation for fairer NLP
Qian, R., Ross, C., Fernandes, J., Smith, E. M., Kiela, D., & Williams, A. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.646 · Cited by 69
Self-citationPhysics of Language Models: Part 3.2, Knowledge Manipulation
Allen-Zhu, Z., & Li, Y. (2025) · doi:10.2139/ssrn.5250621 · Cited by 8
Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H. R., Schütze, H., & Hovy, D. (2024) · arXiv · doi:10.48550/arxiv.2402.16786 · Cited by 7
Gema, A. P., Leang, J. O. J., Hong, G., Devoto, A., Mancino, A. C. M., Saxena, R., He, X., Zhao, Y., Du, X., Madani, M. R. G., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.04127 · Cited by 1
Cited by
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Romanou, A., Ibrahim, M., Ross, C., Oktar, K., Shaib, C., Bell, S. J., Ovalle, A., Bosselut, A., Dodge, J., Sinha, K., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.13285 · Cited by 24
Self-citationQuestionable practices in machine learning
Leech, G., Vazquez, J. J., Kupper, N., Yagudin, M., & Aitchison, L. (2024) · arXiv · doi:10.48550/arxiv.2407.12220 · Cited by 21
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationZhang, J., & Bottou, L. (2025) · arXiv · doi:10.48550/arxiv.2507.03285 · Cited by 4
Metadata
- Authors
- Gupta, V., Pantoja, D., Ross, C., Williams, A., & Ung, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- arXiv
- 2406.19470
- References
- 6
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Changing Answer Order Can Decrease MMLU Accuracy. https://app.cochat.ai/papers/10.48550/arxiv.2406.19470
- The paper
- Gupta, V., Pantoja, D., Ross, C., Williams, A., & Ung, M. (2024). Changing Answer Order Can Decrease MMLU Accuracy. arXiv. https://doi.org/10.48550/arxiv.2406.19470
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.