The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Open accessAbstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks.However, previous works have shown these models are sensitive towards prompt wording, and few-shot demonstrations and their order, posing challenges to fair assessment of these models.As these models become more powerful, it becomes imperative to understand and address these limitations.In this paper, we focus on LLMs robustness on the task of multiple-choice questionscommonly adopted task to study reasoning and fact-retrieving capability of LLMs.Investigating the sensitivity of LLMs towards the order of options in multiple-choice questions, we demonstrate a considerable performance gap of approximately 13% to 85% in LLMs on different benchmarks, when answer options are reordered, even when using demonstrations in a few-shot setting.Through a detailed analysis, we conjecture that this sensitivity arises when LLMs are uncertain about the prediction between the top-2/3 choices, and specific options placements may favor certain prediction between those top choices depending on the question caused by positional bias.We also identify patterns in top-2 choices that amplify or mitigate the model's bias toward option placement.We found that for amplifying bias, the optimal strategy involves positioning the top two choices as the first and last options.Conversely, to mitigate bias, we recommend placing these choices among the adjacent options.To validate our conjecture, we conduct various experiments and adopt two approaches to calibrate LLMs' predictions, leading to up to 8 percentage points improvement across different models and benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Pezeshkpour & Hruschka (2024) demonstrated performance fluctuations of 13-85% across option orderings.”
From Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging (2026) · cites this paper as Pezeshkpour & Hruschka (2024)
The cited source says what the claim attributes to it.
In the citing paper
“Cyclic shift preserves relative distractor ordering, which likely accounts for the contrast with the large sensitivity effects reported under full permutation in prior work (Pezeshkpour & Hruschka, 2024).”
From Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging (2026) · cites this paper as Pezeshkpour & Hruschka, 2024
The cited source says what the claim attributes to it.
In the citing paper
“Prior work suggests LLMs are often not robust to semantically-preserving structural perturbations.”
From Brittlebench: Quantifying LLM robustness via prompt sensitivity (2026) · cites this paper as Pezeshkpour and Hruschka (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020. From 'f'to 'a'on the ny regents science exams: An overview of the aristo project. AI Magazine, 41(4):39-53.
- Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706.
- PMLR.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
Zheng, S., Pan, K., Liu, J., & Chen, Y. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110382 · Cited by 127
Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design
Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A., & Neubig, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00685 · Cited by 171
Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
Tang, R., Zhang, C., Ma, X., Lin, J., & Türe, F. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.129 · Cited by 97
Metadata
- Authors
- Pezeshkpour, P., & Hruschka, E.
- ORCID
- Pezeshkpour, Pouya
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2024
- Published
- 2024
- Pages
- 2006-2017
- OpenAlex
- W4401043020
- Cited by
- 85
- References
- 0
- Citations by year
- 2026: 36 · 2025: 35 · 2024: 14
- FWCI
- 25.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, multiple-choice questions, position bias, prompt sensitivity, few-shot learning, forecast calibration, model robustness, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
