The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Open accessAbstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks.However, previous works have shown these models are sensitive towards prompt wording, and few-shot demonstrations and their order, posing challenges to fair assessment of these models.As these models become more powerful, it becomes imperative to understand and address these limitations.In this paper, we focus on LLMs robustness on the task of multiple-choice questionscommonly adopted task to study reasoning and fact-retrieving capability of LLMs.Investigating the sensitivity of LLMs towards the order of options in multiple-choice questions, we demonstrate a considerable performance gap of approximately 13% to 85% in LLMs on different benchmarks, when answer options are reordered, even when using demonstrations in a few-shot setting.Through a detailed analysis, we conjecture that this sensitivity arises when LLMs are uncertain about the prediction between the top-2/3 choices, and specific options placements may favor certain prediction between those top choices depending on the question caused by positional bias.We also identify patterns in top-2 choices that amplify or mitigate the model's bias toward option placement.We found that for amplifying bias, the optimal strategy involves positioning the top two choices as the first and last options.Conversely, to mitigate bias, we recommend placing these choices among the adjacent options.To validate our conjecture, we conduct various experiments and adopt two approaches to calibrate LLMs' predictions, leading to up to 8 percentage points improvement across different models and benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Pezeshkpour & Hruschka (2024) demonstrated performance fluctuations of 13-85% across option orderings.”
From Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging (2026) · cites this paper as Pezeshkpour & Hruschka (2024)
The cited source says what the claim attributes to it.
In the citing paper
“Cyclic shift preserves relative distractor ordering, which likely accounts for the contrast with the large sensitivity effects reported under full permutation in prior work (Pezeshkpour & Hruschka, 2024).”
From Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging (2026) · cites this paper as Pezeshkpour & Hruschka, 2024
The cited source says what the claim attributes to it.
In the citing paper
“Prior work suggests LLMs are often not robust to semantically-preserving structural perturbations.”
From Brittlebench: Quantifying LLM robustness via prompt sensitivity (2026) · cites this paper as Pezeshkpour and Hruschka (2024)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020. From 'f'to 'a'on the ny regents science exams: An overview of the aristo project. AI Magazine, 41(4):39-53.
- Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706.
- PMLR.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
LLM Evaluators Recognize and Favor Their Own Generations
Panickssery, A., Bowman, S. R., & Feng, S. (2024) · arXiv · doi:10.48550/arxiv.2404.13076 · Cited by 559
Zheng, S., Pan, K., Liu, J., & Chen, Y. (2024) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2024.110382 · Cited by 127
Do LLMs Exhibit Human-like Response Biases? A Case Study in Survey Design
Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A., & Neubig, G. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00685 · Cited by 171
Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models
Tang, R., Zhang, C., Ma, X., Lin, J., & Türe, F. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.129 · Cited by 97
Lee, S., Peng, T., Goldberg, M. H., Rosenthal, S. A., Kotcher, J., Maibach, E. W., & Leiserowitz, A. (2024) · PLOS Climate · doi:10.1371/journal.pclm.0000429 · Cited by 50
On Context Utilization in Summarization with Large Language Models
Ravaut, M., Sun, A., Chen, N. F., & Joty, S. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.153 · Cited by 65
Prompt architecture induces methodological artifacts in large language models
Brucks, M. S., & Toubia, O. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319159 · Cited by 58
Fostering effective hybrid human-LLM reasoning and decision making
Passerini, A., Gema, A. P., Minervini, P., Sayin, B., & Tentori, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1464690 · Cited by 39
Are Large Language Models Good at Utility Judgments?
Zhang, H., Zhang, R., Guo, J., de Rijke, M., Fan, Y., & Cheng, X. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657784 · Cited by 34
Hong, J., Seto, C., Fan, A., & Maciejewski, R. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3536358 · Cited by 25
LEXam: Benchmarking Legal Reasoning on 340 Law ExamsPreprint
Fan, Y., Ni, J., Merane, J., Salimbeni, E., Tian, Y., Hermstrüwer, Y., Huang, Y., Akhtar, M., Geering, F., Dreyer, O., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5265144 · Cited by 25
MovieChat+: Question-Aware Sparse Memory for Long Video Question Answering
Song, E., Chai, W., Tian, Y., Hwang, J., Li, X., & Wang, G. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3604614 · Cited by 80
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Romanou, A., Ibrahim, M., Ross, C., Oktar, K., Shaib, C., Bell, S. J., Ovalle, A., Bosselut, A., Dodge, J., Sinha, K., et al. (2026) · arXiv · doi:10.48550/arxiv.2603.13285 · Cited by 24
DracoGPT: Extracting Visualization Design Preferences from Large Language Models
Wang, H. W., Gordon, M., Battle, L., & Heer, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456350 · Cited by 32
Integrating LLMs into Database Systems Education
Prakash, K., Rao, S. V. N., Hamza, R., Lukich, J., Chaudhari, V., & Nandi, A. (2024) · International Workshop on Data Systems Education (DataEd) · doi:10.1145/3663649.3664371 · Cited by 26
Virtual Personas for Language Models via an Anthology of Backstories
Moon, S., Abdulhai, M., Kang, M., Suh, J., Soedarmadji, W., Behar, E. K., & Chan, D. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1110 · Cited by 53
Self-Assessment Tests are Unreliable Measures of LLM Personality
Gupta, A., Song, X., & Anumanchipalli, G. (2024) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2024.blackboxnlp-1.20 · Cited by 51
Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
Deng, A., Cao, T., Chen, Z., & Hooi, B. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00366 · Cited by 19
Hijacking Large Language Models via Adversarial In-Context Learning
Zhou, X., Yao, Q., Zade, S. Z., Khanduri, P., & Zhu, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37673-2_13 · Cited by 49
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
Reif, Y., & Schwartz, R. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.378 · Cited by 29
Overloaded minds and machines: a cognitive load framework for human-AI symbiosis
Wang, P., Liu, H., Zou, L., & PAAS, F. G. W. C. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11510-z · Cited by 18
Dual-process theory and decision-making in large language models
Brady, O., Nulty, P., Zhang, L., Ward, T. E., & McGovern, D. P. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-025-00506-1 · Cited by 19
Robustness of large language models in moral judgements
Oh, S., & Demberg, V. (2025) · Royal Society Open Science · doi:10.1098/rsos.241229 · Cited by 19
Pecher, B., Srba, I., & Bieliková, M. (2024) · ACM Computing Surveys · doi:10.1145/3691339 · Cited by 19
Position: LLM Unlearning Benchmarks are Weak Measures of Progress
Thaker, P., Hu, S., Kale, N., Maurya, Y., Wu, Z. S., & Smith, V. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00035 · Cited by 47
The Medium is the Message: How Non-Clinical Information Shapes Clinical Decisions in LLMs
Gourabathina, A., Gerych, W., Pan, E., & Ghassemi, M. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732121 · Cited by 21
Cognitive Biases in Large Language Models: A Survey and Mitigation Experiments
Sumita, Y., Takeuchi, K., & Kashima, H. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707812 · Cited by 29
Human-guided collective LLM intelligence for strategic planning via two-stage information retrieval
Kim, S., Ha, J., Lee, H., Park, S., & Cho, S. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104288 · Cited by 14
Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences
Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00843-4 · Cited by 14
On Positional Bias of Faithfulness for Long-form Summarization
Wan, D., Vig, J., Bansal, M., & Joty, S. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.442 · Cited by 13
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12
Trummer, I. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3682017 · Cited by 14
Open (Clinical) LLMs are Sensitive to Instruction Phrasings
Ceballos-Arroyo, A. M., Munnangi, M., Sun, J., Zhang, K., McInerney, J., Wallace, B., & Amir, S. (2024) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2024.bionlp-1.5 · Cited by 21
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Bigverdi, M., Luo, Z., Hsieh, C.-Y., Shen, E., Chen, D., Shapiro, L. G., & Krishna, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00363 · Cited by 32
Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation
Park, J.-W., Park, S., Won, H.-S., & Kim, K.-M. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.477 · Cited by 14
CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges
Li, H., Chen, J., Ai, Q., Chu, Z., Zhou, Y., Dong, Q., & Liu, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.808 · Cited by 35
Neubauer, J. C., Kaiser, A., Lettermann, L., Volkert, T., & Häge, A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0332917 · Cited by 10
Lange, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01377-4_9 · Cited by 9
Exploiting Language Models as a Source of Knowledge for Cognitive Agents
Kirk, J. R., Wray, R. E., & Laird, J. E. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27690 · Cited by 15
Exploring Spatial Schema Intuitions in Large Language and Vision Models
Wicke, P., & Wachowiak, L. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.365 · Cited by 8
Messner, W. (2025) · Journal of World Business · doi:10.1016/j.jwb.2025.101622 · Cited by 8
LMLPA: Language Model Linguistic Personality Assessment
Zheng, J., Wang, X., Hosio, S. J., Xu, X., & Lee, L. (2025) · Computational Linguistics · doi:10.1162/coli_a_00550 · Cited by 10
Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs
Kabir, M., Abrar, A., & Ananiadou, S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.2 · Cited by 6
Attia, M., Muhamed, A., Alkhamissi, M., Solorio, T., & Diab, M. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.341 · Cited by 6
How to Make the Most of LLMs’ Grammatical Knowledge for Acceptability Judgments
Ide, Y., Nishida, Y., Vasselli, J., Oba, M., Sakai, Y., Kamigaito, H., & Watanabe, T. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.380 · Cited by 6
Avnat, E., Levy, M., Herstain, D., Yanko, E., Joya, D. B., Katz, M. T., Eshel, D., Laros, S., Dagan, Y., Barami, S., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/64452 · Cited by 9
LLM-empowered literature mining for material substitution studies in sustainable concrete
Duan, Y., Tian, Y., Ghosh, S., Venugopal, V., Chen, J., & Olivetti, E. (2025) · Resources Conservation and Recycling · doi:10.1016/j.resconrec.2025.108379 · Cited by 6
Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models
Liusie, A., Fathullah, Y., & Gales, M. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.81 · Cited by 12
When2Call: When (not) to Call Tools
Ross, H., Mahabaleshwarkar, A. S., & Suhara, Y. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.174 · Cited by 4
HyperG: Hypergraph-Enhanced LLMs for Structured Knowledge
Huang, S., Li, H., Gu, Y., Hu, X., Li, Q., & Xu, G. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730002 · Cited by 5
Cents: A Flexible and Cost-Effective Framework for LLM-Based Table Understanding
Xiao, G., Dong, H., Wang, J., & Bałazińska, M. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749714 · Cited by 4
QA-prompting: Improving Summarization with Large Language Models using Question-Answering
Sinha, N. (2025) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2025.newsum-main.14 · Cited by 6
Evaluating Large Language Model Robustness using Combinatorial Testing
Chandrasekaran, J., Patel, A. R., Lanus, E., & Freeman, L. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962520 · Cited by 4
Wait, that’s not an option: LLMs Robustness with Incorrect Multiple-Choice Options
Góral, G., Wiśnios, E., Sankowski, P., & Budzianowski, P. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.75 · Cited by 12
GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
Xiao, Y., Liu, X., Cheng, Q., Yin, Z., Liang, S., Li, J., Shao, J., Liu, A., & Tao, D. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02556-7 · Cited by 24
Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
Zhou, H., Wan, X., Liu, Y., Collier, N., Vulić, I., & Korhonen, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.72 · Cited by 20
Challenging Large Language Models with New Tasks: A Study on their Adaptability and Robustness
Li, C., Tian, Y., Zerong, Z., Song, Y., & Xia, F. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.485 · Cited by 3
Zhang, Y., & He, Z. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.691 · Cited by 3
Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors
Xue, M., Hu, Z., Liu, L., Liao, K., Li, S., Han, H., Zhao, M. K., & Yin, C. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.237 · Cited by 3
ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding
Azime, I. A., Tonja, A. L., Belay, T. D., Chanie, Y., Balcha, B. F., Abadi, N. H., Ademtew, H. B., Nerea, M. A., Yadeta, D. D., Geremew, D. D., et al. (2025) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2025.findings-naacl.350 · Cited by 3
DARE : Diverse Visual Question Answering with Robustness Evaluation
Sterz, H., Pfeiffer, J., & Vulić, I. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.29 · Cited by 3
Diallo, N., Alsagheer, D., Kamal, M., Xu, L., Lu, Y., Karanjai, R., & Shi, L. (2024) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg61611.2024.10702065 · Cited by 3
Hoch, C. C., Funk, P. F., Guntinas‐Lichius, O., Volk, G. F., Lüers, J.-C., Hussain, T., Wirth, M., Schmidl, B., Wollenberg, B., & Alfertshofer, M. G. (2025) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-025-09404-x · Cited by 2
From Heuristics to Intelligence: Large Language Model-Driven Test Case Generation
Mastropaolo, A., Kuhn, D. R., & Voas, J. (2025) · Computer · doi:10.1109/mc.2025.3609478 · Cited by 2
Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?
Balepur, N., & Rudinger, R. (2024) · doi:10.18653/v1/2024.knowllm-1.2 · Cited by 6
Rationales for Answers to Simple Math Word Problems Confuse Large Language Models
Zhang, Y., Xue, M., Liu, D., & He, Z. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.524 · Cited by 4
Structured Document Generation for Industrial Equipment
Lynch, K., Lorenzi, F., Sheehan, J. D., Kabakci-Zorlu, D., & Eck, B. J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i28.35150 · Cited by 2
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. (2025) · Preprints.org · doi:10.20944/preprints202306.0363.v8 · Cited by 2
What Large Language Models Know About Plant Molecular BiologyPreprint
Burda, M. F., Ferrero, L., Gaggion, N., Fonouni‐Farde, C., Crespi, M. D., Ariel, F., & Ferrante, E. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.08.31.672925 · Cited by 3
Dai, H., Li, Y., Villalobos-Quesada, M. J., Ng, Y. M., Aardoom, J. J., Versluis, A., Shen, H., Spruit, M. R., & Chavannes, N. H. (2026) · medRxiv · doi:10.64898/2026.09.07.26362410 · Cited by 2
The use of artificial intelligence in the assessment of live subtitling quality: the NER Buddy
Romero-Fresco, P., Amigo, Ó. A., & Bacigalupe, L. A. (2024) · Tradumàtica tecnologies de la traducció · doi:10.5565/rev/tradumatica.408 · Cited by 1
Uncovering Large Language Model Weaknesses in Character and Word Understanding and Manipulating
Zhang, Y., He, Z., & Yen, G. G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3575818 · Cited by 1
Zeno, G., Jedidi, N., & Gomez, S. R. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00058 · Cited by 1
Guo, Z., Lv, H., Zhang, C., Zhao, Y., Zhang, Y., & Cui, L. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.458 · Cited by 1
Evaluating Variance in Visual Question Answering Benchmarks
Nikitha, S. R. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00442 · Cited by 2
GCAgent: Enhancing Group Chat Communication through Dialogue Agents System
Meng, Z., Xie, Z., Ye, Z., Lu, C., Liu, Z., Niu, Z., Hu, Y., & Cao, S. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3793118 · Cited by 1
Benchmarking Conversation Routing in Chatbot Systems Based on Large Language Models
Maksymenko, D., Kryvoshein, D., Turuta, O., Kazakov, D., & Turuta, O. (2024) · International Workshop of IT-professionals on Artificial Intelligence (ProfIT AI) · doi:10.55056/ceur-ws.org/vol-3777/paper6.pdf · Cited by 1
SATA-Bench : Select All That Apply Benchmark For Multiple Choice Questions
Xu, W., cui, S., Fang, X., Xue, C., Eckman, S., & Reddy, C. K. A. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817472 · Cited by 1
Fukuda, H., Morishita, M., Takahashi, O., Sasaguri, M., & Habu, M. (2026) · International Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.ijom.2026.08.026 · Cited by 1
Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints
Gema, A. P., Lee, C., Minervini, P., Daines, L., Simpson, T., & Alex, B. (2024) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2024.clinicalnlp-1.49 · Cited by 2
Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs
Raimondi, B., Giallorenzo, S., & Gabbrielli, M. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707732 · Cited by 1
QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
Guda, B., Ashungafac, G. Z., Francis, L., & Joe‐Wong, C. (2024) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshp64532.2024.11100592 · Cited by 2
DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs
Meng, Z., (8098), Y. Z., Feng, Z., & Liu, Z. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-16823-8_8 · Cited by 5
Alkhairy, M., Homer, V., & O'Connor, B. (2026) · Journal of Logic Language and Information · doi:10.1007/s10849-026-09481-z · Cited by 1
M, L. M. (2026) · Preprints.org · doi:10.20944/preprints202603.1116.v1 · Cited by 1
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
Cacioli, J.-P. (2026) · arXiv · doi:10.48550/arxiv.2604.26206
Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
Wadi, D., & Ma, Y. (2026) · arXiv · doi:10.48550/arxiv.2609.28372
Yang, W., Yang, J., Li, W., & Guo, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0348-0_5
CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models
Wagner, E., Slavutsky, Y., & Abend, O. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.866 · Cited by 1
Expectation management in humans and LLMs
Menashe, B. A., Drake, A., & Ben‐Shachar, M. (2026) · Scientific Reports · doi:10.1038/s41598-026-62526-y
Liu, Y., Wang, X., & Lu, K. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34648
Mir, T. H., Wei, J.-C., Simanjuntak, M. S., Cheng, W.-S., Lee, Y.-H., Zhang, Y.-C., Chen, J., & Dai, H.-J. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20940-w
Li, K., Shi, Z., Hu, M.-J., Jin, Y., Yan, X., He, F., & Wu, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128726
Kinder, L., Edman, L., Fraser, A., & Käfer, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02813-6_25
Korn, A. (2025) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re63999.2025.00075
REINFORCEMENT LEARNING FROM AI FEEDBACK:ACROSS-MODELANALYSIS OF PERFORMANCE, SCALABILITY AND BIAS
Nguyễn, L. V., & Sie, R. L. L. (2025) · doi:10.5121/csit.2025.151916
Ghanim, M. A., Xue, J., Hastuti, R. P., Zheng, M., Solihin, Y., & Lou, Q. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.390 · Cited by 6
Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D
Panagopoulou, A., Xue, L., Zhou, H., Savarese, S., Xu, R., Xiong, C., Callison-Burch, C., Yatskar, M., & Niebles, J. C. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1288
Metadata
- Authors
- Pezeshkpour, P., & Hruschka, E.
- ORCID
- Pezeshkpour, Pouya
- Venue
- Findings of the Association for Computational Linguistics: NAACL 2024
- Published
- 2024
- Pages
- 2006-2017
- OpenAlex
- W4401043020
- Cited by
- 85
- References
- 0
- Citations by year
- 2026: 36 · 2025: 35 · 2024: 14
- FWCI
- 25.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, multiple-choice questions, position bias, prompt sensitivity, few-shot learning, forecast calibration, model robustness, question answering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
