Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Open accessAbstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks.However, previous works have shown these models are sensitive towards prompt wording, and few-shot demonstrations and their order, posing challenges to fair assessment of these models.As these models become more powerful, it becomes imperative to understand and address these limitations.In this paper, we focus on LLMs robustness on the task of multiple-choice questionscommonly adopted task to study reasoning and fact-retrieving capability of LLMs.Investigating the sensitivity of LLMs towards the order of options in multiple-choice questions, we demonstrate a considerable performance gap of approximately 13% to 85% in LLMs on different benchmarks, when answer options are reordered, even when using demonstrations in a few-shot setting.Through a detailed analysis, we conjecture that this sensitivity arises when LLMs are uncertain about the prediction between the top-2/3 choices, and specific options placements may favor certain prediction between those top choices depending on the question caused by positional bias.We also identify patterns in top-2 choices that amplify or mitigate the model's bias toward option placement.We found that for amplifying bias, the optimal strategy involves positioning the top two choices as the first and last options.Conversely, to mitigate bias, we recommend placing these choices among the adjacent options.To validate our conjecture, we conduct various experiments and adopt two approaches to calibrate LLMs' predictions, leading to up to 8 percentage points improvement across different models and benchmarks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pezeshkpour & Hruschka demonstrated performance fluctuations of 13-85% across option orderings.
“Pezeshkpour & Hruschka (2024) demonstrated performance fluctuations of 13-85% across option orderings.”
From Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging (2026) · cites this paper as Pezeshkpour & Hruschka (2024)
Cyclic shift preserves relative distractor ordering, which likely accounts for the contrast with the large sensitivity effects reported under full permutation in prior work.
“Cyclic shift preserves relative distractor ordering, which likely accounts for the contrast with the large sensitivity effects reported under full permutation in prior work (Pezeshkpour & Hruschka, 2024).”
From Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging (2026) · cites this paper as Pezeshkpour & Hruschka, 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020. From 'f'to 'a'on the ny regents science exams: An overview of the aristo project. AI Magazine, 41(4):39-53.
- Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706.
- PMLR.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · see citing works
Zheng, S., Pan, K., Liu, J., & Chen, Y. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110382 · Cited by 127 · see citing works
Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design
Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A., & Neubig, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00685 · Cited by 100 · see citing works
Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
Tang, R., Zhang, C., Ma, X., Lin, J., & Türe, F. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.129 · Cited by 59 · see citing works
Lee, S., Peng, T., Goldberg, M. H., Rosenthal, S. A., Kotcher, J., Maibach, E. W., & Leiserowitz, A. (2024) · PLOS Climate · doi:10.1371/journal.pclm.0000429 · Cited by 50 · see citing works
On Context Utilization in Summarization with Large Language Models
Ravaut, M., Sun, A., Chen, N. F., & Joty, S. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.153 · Cited by 45 · see citing works
Prompt architecture induces methodological artifacts in large language models
Brucks, M. S., & Toubia, O. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319159 · Cited by 42 · see citing works
Fostering effective hybrid human-LLM reasoning and decision making
Passerini, A., Gema, A. P., Minervini, P., Sayin, B., & Tentori, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1464690 · Cited by 35 · see citing works
Are Large Language Models Good at Utility Judgments?
Zhang, H., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657784 · Cited by 26 · see citing works
Hong, J., Seto, C., Fan, A., & Maciejewski, R. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3536358 · Cited by 25 · see citing works
LEXam: Benchmarking Legal Reasoning on 340 Law ExamsPreprint
Fan, Y., Ni, J., Merane, J., Salimbeni, E., Tian, Y., Hermstrüwer, Y., Huang, Y., Akhtar, M., Geering, F., Dreyer, O., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5265144 · Cited by 25 · see citing works
MovieChat+: Question-Aware Sparse Memory for Long Video Question Answering
Song, E., Chai, W., Tian, Y., Hwang, J., Li, X., & Wang, G. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3604614 · Cited by 24 · see citing works
DracoGPT: Extracting Visualization Design Preferences from Large Language Models
Wang, H. W., Gordon, M., Battle, L., & Heer, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456350 · Cited by 23 · see citing works
Integrating LLMs into Database Systems Education
Prakash, K., Rao, S. V. N., Hamza, R., Lukich, J., Chaudhari, V., & Nandi, A. (2024) · International Workshop on Data Systems Education (DataEd) · doi:10.1145/3663649.3664371 · Cited by 22 · see citing works
Virtual Personas for Language Models via an Anthology of Backstories
Moon, S., Abdulhai, M., Kang, M., Suh, J., Soedarmadji, W., Behar, E. K., & Chan, D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1110 · Cited by 21 · see citing works
Self-Assessment Tests are Unreliable Measures of LLM Personality
Gupta, A., Song, X., & Anumanchipalli, G. (2024) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2024.blackboxnlp-1.20 · Cited by 20 · see citing works
Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
Deng, A., Cao, T., Chen, Z., & Hooi, B. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00366 · Cited by 19 · see citing works
Hijacking Large Language Models via Adversarial In-Context Learning
Zhou, X., Yao, Q., Zade, S. Z., Khanduri, P., & Zhu, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37673-2_13 · Cited by 19 · see citing works
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
Reif, Y., & Schwartz, R. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.378 · Cited by 18 · see citing works
Overloaded minds and machines: a cognitive load framework for human-AI symbiosis
Wang, P., Liu, H., Zou, L., & PAAS, F. G. W. C. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11510-z · Cited by 17 · see citing works
Dual-process theory and decision-making in large language models
Brady, O., Nulty, P., Zhang, L., Ward, T. E., & McGovern, D. P. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00506-1 · Cited by 16 · see citing works
Robustness of large language models in moral judgements
Oh, S., & Demberg, V. (2025) · Royal Society Open Science · doi:10.1098/rsos.241229 · Cited by 15 · see citing works
Pecher, B., Srba, I., & Bieliková, M. (2024) · ACM Computing Surveys · doi:10.1145/3691339 · Cited by 15 · see citing works
Position: LLM Unlearning Benchmarks are Weak Measures of Progress
Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., & Smith, V. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00035 · Cited by 15 · see citing works
The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs
Gourabathina, A., Gerych, W., Pan, E., & Ghassemi, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732121 · Cited by 14 · see citing works
Cognitive Biases in Large Language Models: A Survey and Mitigation Experiments
Sumita, Y., Takeuchi, K., & Kashima, H. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707812 · Cited by 14 · see citing works
Human-guided collective LLM intelligence for strategic planning via two-stage information retrieval
Kim, S., Ha, J., Lee, H., Park, S., & Cho, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104288 · Cited by 13 · see citing works
Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences
Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00843-4 · Cited by 13 · see citing works
On Positional Bias of Faithfulness for Long-form Summarization
Wan, D., Vig, J., Bansal, M., & Joty, S. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.442 · Cited by 13 · see citing works
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12 · see citing works
Trummer, I. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3682017 · Cited by 11 · see citing works
Open (Clinical) LLMs are Sensitive to Instruction Phrasings
Ceballos-Arroyo, A. M., Munnangi, M., Sun, J., Zhang, K., McInerney, J., Wallace, B., & Amir, S. (2024) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2024.bionlp-1.5 · Cited by 11 · see citing works
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Bigverdi, M., Luo, Z., Hsieh, C.-Y., Shen, E., Chen, D., Shapiro, L. G., & Krishna, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00363 · Cited by 11 · see citing works
Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation
Park, J.-W., Park, S., Won, H.-S., & Kim, K.-M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.477 · Cited by 10 · see citing works
CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges
Li, H., Chen, J., Ai, Q., Chu, Z., Zhou, Y., Dong, Q., & Liu, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.808 · Cited by 10 · see citing works
Neubauer, J. C., Kaiser, A., Lettermann, L., Volkert, T., & Häge, A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0332917 · Cited by 10 · see citing works
Lange, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01377-4_9 · Cited by 9 · see citing works
Exploiting Language Models as a Source of Knowledge for Cognitive Agents
Kirk, J. R., Wray, R. E., & Laird, J. E. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27690 · Cited by 9 · see citing works
Exploring Spatial Schema Intuitions in Large Language and Vision Models
Wicke, P., & Wachowiak, L. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.365 · Cited by 8 · see citing works
Messner, W. (2025) · Journal of World Business · doi:10.1016/j.jwb.2025.101622 · Cited by 8 · see citing works
LMLPA: Language Model Linguistic Personality Assessment
Zheng, J., Wang, X., Hosio, S. J., Xu, X., & Lee, L. (2025) · Computational Linguistics · doi:10.1162/coli_a_00550 · Cited by 7 · see citing works
Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
Kabir, M., Abrar, A., & Ananiadou, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.2 · Cited by 6 · see citing works
Attia, M., Muhamed, A., Alkhamissi, M., Solorio, T., & Diab, M. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.341 · Cited by 6 · see citing works
How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments
Ide, Y., Nishida, Y., Vasselli, J., Oba, M., Sakai, Y., Kamigaito, H., & Watanabe, T. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.380 · Cited by 6 · see citing works
Avnat, E., Levy, M., Herstain, D., Yanko, E., Joya, D. B., Katz, M. T., Eshel, D., Laros, S., Dagan, Y., Barami, S., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/64452 · Cited by 6 · see citing works
LLM-empowered literature mining for material substitution studies in sustainable concrete
Duan, Y., Tian, Y., Ghosh, S., Venugopal, V., Chen, J., & Olivetti, E. (2025) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2025.108379 · Cited by 5 · see citing works
Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models
Liusie, A., Fathullah, Y., & Gales, M. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.81 · Cited by 5 · see citing works
When2Call: When (not) to Call Tools
Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.174 · Cited by 4
HyperG: Hypergraph-Enhanced LLMs for Structured Knowledge
Huang, S., Li, H., Gu, Y., Hu, X., Li, Q., & Xu, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730002 · Cited by 4 · see citing works
Cents: A Flexible and Cost-Effective Framework for LLM-Based Table Understanding
Xiao, G., Dong, H., Wang, J., & Bałazińska, M. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749714 · Cited by 4 · see citing works
QA-prompting: Improving Summarization with Large Language Models using Question-Answering
Sinha, N. (2025) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2025.newsum-main.14 · Cited by 4 · see citing works
Evaluating Large Language Model Robustness using Combinatorial Testing
Chandrasekaran, J., Patel, A. R., Lanus, E., & Freeman, L. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962520 · Cited by 4 · see citing works
Wait, that’s not an option: LLMs Robustness with Incorrect Multiple-Choice Options
Góral, G., Wiśnios, E., Sankowski, P., & Budzianowski, P. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.75 · Cited by 4 · see citing works
GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
Xiao, Y., Liu, X., Cheng, Q., Yin, Z., Liang, S., Li, J., Shao, J., Liu, A., & Tao, D. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02556-7 · Cited by 4 · see citing works
Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
Zhou, H., Wan, X., Liu, Y., Collier, N., Vulić, I., & Korhonen, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.72 · Cited by 4 · see citing works
Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness
Li, C., Tian, Y., Zerong, Z., Song, Y., & Xia, F. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.485 · Cited by 3 · see citing works
Zhang, Y., & He, Z. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.691 · Cited by 3 · see citing works
Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors
Xue, M., Hu, Z., Liu, L., Liao, K., Li, S., Han, H., Zhao, M. K., & Yin, C. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.237 · Cited by 3 · see citing works
ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding
Azime, I. A., Tonja, A. L., Belay, T. D., Chanie, Y., Balcha, B. F., Abadi, N. H., Ademtew, H. B., Nerea, M. A., Yadeta, D. D., Geremew, D. D., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.350 · Cited by 3 · see citing works
DARE : Diverse Visual Question Answering with Robustness Evaluation
Sterz, H., Pfeiffer, J., & Vulić, I. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.29 · Cited by 3 · see citing works
Diallo, N., Alsagheer, D., Kamal, M., Xu, L., Lu, Y., Karanjai, R., & Shi, L. (2024) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg61611.2024.10702065 · Cited by 2 · see citing works
Hoch, C. C., Funk, P. F., Guntinas‐Lichius, O., Volk, G. F., Lüers, J.-C., Hussain, T., Wirth, M., Schmidl, B., Wollenberg, B., & Alfertshofer, M. G. (2025) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-025-09404-x · Cited by 2 · see citing works
From Heuristics to Intelligence: Large Language Model-Driven Test Case Generation
Mastropaolo, A., Kuhn, D. R., & Voas, J. (2025) · Computer · doi:10.1109/mc.2025.3609478 · Cited by 2 · see citing works
Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
Balepur, N., & Rudinger, R. (2024) · doi:10.18653/v1/2024.knowllm-1.2 · Cited by 2 · see citing works
Rationales for Answers to Simple Math Word Problems Confuse Large Language Models
Zhang, Y., Xue, M., Liu, D., & He, Z. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.524 · Cited by 2 · see citing works
Structured Document Generation for Industrial Equipment
Lynch, K., Lorenzi, F., Sheehan, J. D., Kabakci-Zorlu, D., & Eck, B. J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i28.35150 · Cited by 2 · see citing works
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. (2025) · Preprints.org · doi:10.20944/preprints202306.0363.v8 · Cited by 2 · see citing works
What Large Language Models Know About Plant Molecular BiologyPreprint
Burda, M. F., Ferrero, L., Gaggion, N., Fonouni‐Farde, C., Crespi, M. D., Ariel, F., & Ferrante, E. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.31.672925 · Cited by 2 · see citing works
Dai, H., Li, Y., Villalobos-Quesada, M. J., Ng, Y. M., Aardoom, J. J., Versluis, A., Shen, H., Spruit, M. R., & Chavannes, N. H. (2026) · medRxiv · doi:10.64898/2026.09.07.26362410 · Cited by 2 · see citing works
The use of artificial intelligence in the assessment of live subtitling quality: the NER Buddy
Romero-Fresco, P., Amigo, Ó. A., & Bacigalupe, L. A. (2024) · Tradumàtica tecnologies de la traducció · doi:10.5565/rev/tradumatica.408 · Cited by 1 · see citing works
Uncovering Large Language Model Weaknesses in Character and Word Understanding and Manipulating
Zhang, Y., He, Z., & Yen, G. G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3575818 · Cited by 1 · see citing works
Zeno, G., Jedidi, N., & Gomez, S. R. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00058 · Cited by 1 · see citing works
Guo, Z., Lv, H., Zhang, C., Zhao, Y., Zhang, Y., & Cui, L. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.458 · Cited by 1 · see citing works
Evaluating Variance in Visual Question Answering Benchmarks
Nikitha, S. R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00442 · Cited by 1 · see citing works
GCAgent: Enhancing Group Chat Communication through Dialogue Agents System
Meng, Z., Xie, Z., Ye, Z., Lu, C., Liu, Z., Niu, Z., Hu, Y., & Cao, S. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3793118 · Cited by 1 · see citing works
Benchmarking Conversation Routing in Chatbot Systems Based on Large Language Models
Maksymenko, D., Kryvoshein, D., Turuta, O., Kazakov, D., & Turuta, O. (2024) · International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI) · doi:10.55056/ceur-ws.org/vol-3777/paper6.pdf · Cited by 1 · see citing works
SATA-Bench : Select All That Apply Benchmark For Multiple Choice Questions
Xu, W., cui, S., Fang, X., Xue, C., Eckman, S., & Reddy, C. K. A. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817472 · Cited by 1 · see citing works
Fukuda, H., Morishita, M., Takahashi, O., Sasaguri, M., & Habu, M. (2026) · International Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.ijom.2026.08.026 · Cited by 1 · see citing works
Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints
Gema, A. P., Lee, C., Minervini, P., Daines, L., Simpson, T., & Alex, B. (2024) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2024.clinicalnlp-1.49 · Cited by 1 · see citing works
Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs
Raimondi, B., Giallorenzo, S., & Gabbrielli, M. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707732 · Cited by 1 · see citing works
QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
Guda, B., Ashungafac, G. Z., Francis, L., & Joe‐Wong, C. (2024) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshp64532.2024.11100592 · Cited by 1 · see citing works
DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs
Meng, Z., (8098), Y. Z., Feng, Z., & Liu, Z. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-16823-8_8 · Cited by 1 · see citing works
Alkhairy, M., Homer, V., & O'Connor, B. (2026) · Journal of Logic Language and Information · doi:10.1007/s10849-026-09481-z · Cited by 1 · see citing works
M, L. M. (2026) · Preprints.org · doi:10.20944/preprints202603.1116.v1 · Cited by 1 · see citing works
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
Cacioli, J.-P. (2026) · arXiv · doi:10.48550/arxiv.2604.26206
Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
Wadi, D., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.28372
Yang, W., Yang, J., Li, W., & Guo, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0348-0_5
CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models
Wagner, E., Slavutsky, Y., & Abend, O. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.866
Expectation management in humans and LLMs
Menashe, B. A., Drake, A., & Ben‐Shachar, M. (2026) · Scientific Reports · doi:10.1038/s41598-026-62526-y
Liu, Y., Wang, X., & Lu, K. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34648
Mir, T. H., Wei, J.-C., Simanjuntak, M. S., Cheng, W.-S., Lee, Y.-H., Zhang, Y.-C., Chen, J., & Dai, H.-J. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20940-w
Li, K., Shi, Z., Hu, M.-J., Jin, Y., Yan, X., He, F., & Wu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128726
Kinder, L., Edman, L., Fraser, A., & Käfer, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02813-6_25
Korn, A. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00075
REINFORCEMENT LEARNING FROM AI FEEDBACK:ACROSS-MODELANALYSIS OF PERFORMANCE, SCALABILITY AND BIAS
Nguyễn, L. V., & Sie, R. L. L. (2025) · doi:10.5121/csit.2025.151916
Ghanim, M. A., Xue, J., Hastuti, R. P., Zheng, M., Solihin, Y., & Lou, Q. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.390
Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D
Panagopoulou, A., Xue, L., Zhou, H., Savarese, S., Xu, R., Xiong, C., Callison-Burch, C., Yatskar, M., & Niebles, J. C. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1288
Hybrid Multi-Agent Systems for Auditable AI SurveyingPreprint
Naky, A., Saravi, S., Batmaz, F., Yang, Y., Derakhshan, P., Nevisi, H., & Storey, G. (2025) · Research Square · doi:10.21203/rs.3.rs-8105108/v1
QMOS: Enhancing LLMs for telecommunications with question-masked loss and option shuffling
Guda, B., Ashungafac, G. Z., Francis, L., & Joe‐Wong, C. (2025) · ITU Journal on Future and Evolving Technologies · doi:10.52953/nmom3434
- Authors
- Pezeshkpour, P., & Hruschka, E.
- ORCID
- Pezeshkpour, Pouya
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2024
- Published
- 2024
- Pages
- 2006-2017
- OpenAlex
- W4401043020
- Cited by
- 85
- References
- 0
- Citations by year
- 2026: 36 · 2025: 35 · 2024: 14
- FWCI
- 25.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, multiple-choice questions, position bias, prompt sensitivity, few-shot learning, forecast calibration, model robustness, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
