CoChat
Sign Up
Cited by 150Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.

Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions

Open access
Pezeshkpour, Pouya, Hruschka, Estevam·2024·Findings of the Association for Computational Linguistics: NAACL 2024
DOI 10.18653/v1/2024.findings-naacl.130
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks.However, previous works have shown these models are sensitive towards prompt wording, and few-shot demonstrations and their order, posing challenges to fair assessment of these models.As these models become more powerful, it becomes imperative to understand and address these limitations.In this paper, we focus on LLMs robustness on the task of multiple-choice questionscommonly adopted task to study reasoning and fact-retrieving capability of LLMs.Investigating the sensitivity of LLMs towards the order of options in multiple-choice questions, we demonstrate a considerable performance gap of approximately 13% to 85% in LLMs on different benchmarks, when answer options are reordered, even when using demonstrations in a few-shot setting.Through a detailed analysis, we conjecture that this sensitivity arises when LLMs are uncertain about the prediction between the top-2/3 choices, and specific options placements may favor certain prediction between those top choices depending on the question caused by positional bias.We also identify patterns in top-2 choices that amplify or mitigate the model's bias toward option placement.We found that for amplifying bias, the optimal strategy involves positioning the top two choices as the first and last options.Conversely, to mitigate bias, we recommend placing these choices among the adjacent options.To validate our conjecture, we conduct various experiments and adopt two approaches to calibrate LLMs' predictions, leading to up to 8 percentage points improvement across different models and benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.findings-naacl.130

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 3

  1. Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020. From 'f'to 'a'on the ny regents science exams: An overview of the aristo project. AI Magazine, 41(4):39-53.
  2. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706.
  3. PMLR.

Cited by

Sorted by citations

Items 101–150 of 150

Related

See all

Metadata

Authors
Pezeshkpour, P., & Hruschka, E.
ORCID
Pezeshkpour, Pouya
Venue
Findings of the Association for Computational Linguistics: NAACL 2024
Publisher
Association for Computational Linguistics
Published
2024
Pages
2006-2017
DOI
10.18653/v1/2024.findings-naacl.130
OpenAlex
W4401043020
Cited by
85
References
0
Citations by year
2026: 36 · 2025: 35 · 2024: 14
FWCI
25.08
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: NAACL (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, multiple-choice questions, position bias, prompt sensitivity, few-shot learning, forecast calibration, model robustness, question answering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review