The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Measuring and Narrowing the Compositionality Gap in Language Models
Open accessAbstract
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap.We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining.In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease.This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning.We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly.We present a new method, self-ask, that further improves on chain of thought.In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question.We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Systematicity continues to challenge models11–18 and motivates new frameworks34–41.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“Notably, modern neural networks still struggle on tests of systematicity11–18—tests that even a minimally algebraic mind should pass2.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [18]
In the citing paper
“recent studies have demonstrated the limited generalization capabilities of LLMs, with deficiencies in planning15, abstraction16, and compositionality17 across various tasks.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Self-citationA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,146
Cited by
Sorted by citations
Items 301–321 of 321
Camlet, A., Kusiak, A., & Świetlik, D. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15197387
AORS: Atomic Operator Driven Structured Reasoning Framework with Dual-Level Retrieval
Zhang, J., Li, W., Zhu, Z., Xie, L., & Zhang, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-5693-8_23
Jin, X., Du, Y., & Han, H. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08909-4
From Natural Language to Formal Specifications: A Metamodel-Driven LLM Pipeline
Ismail, S., & Tessier, P. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3837062.3838893
Phungjununt, W., Srisomboon, K., Lee, W., Prayote, A., & Pipanmekaporn, L. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100317
Graph-Augmented Knowledge Infusion for Dialogue-Based Medication Recommendation
Zhong, J., Li, Y., Hu, S., Zhang, Y., Xu, T., & Zou, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-5696-9_8
Improving Tool-Call Robustness in Companion Robot Agents with ASR N-Best and Clarification
Shin, J., Jun, K., & Lee, D.-W. (2026) · IEEE RO-MAN · doi:10.1109/ro-man72286.2026.11714621
Let’s Chat about Design: A Transformation-Oriented Viewpoint for LLM-Native Software
Braberman, V. A., & Bonomo, F. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834545
Semantic-Driven Context Pruning for Arabic RAG: Quality-Preserving Compression Under Token Inflation
Taresh, A. A. R., & Zghair, N. A. K. (2026) · Information · doi:10.3390/info17100990
Cominelli, L., Bossi, F., Galatolo, F. A., Cimino, M. G. C. A., Manduzio, G. A., Greco, A., & Scilingo, E. P. (2026) · International Journal of Social Robotics · doi:10.1007/s12369-026-01445-4
Adaptive Fault-Tolerant Retrieval-Augmented Generation for Multi-Hop Open-Domain Question Answering
Liu, P., Li, B., Li, S., Ge, X., Ran, Z., & Li, K. (2026) · International Conference of Data and Information Systems (DIS) · doi:10.1109/dis72268.2026.11725720
Li, Z., Gao, C., Wang, Z., Wang, X., Lu, Y., & You, D. (2026) · Expert Systems · doi:10.1111/exsy.70455
Liu, J., Lee, Z., Cao, S., Zhang, J., Liu, W., Che, X., Hou, L., & Li, J. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728497
Ma, Z., 金效嘉, Zhao, D., Wang, B., & Wang, Y. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728968
Information Geometric Uncertainty Modeling for Agentic Reinforcement Learning
Zhao, W., Hua, J., Wang, X., Wang, H., & Kang, M. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728583
AMRAG: Adaptive Multi-step Retrieval-Augmented Generation for Complex Question Answering
Zhang, N., Wang, X., Luo, X., & Gao, J. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728424
Epistemic State Modeling for Adaptive Retrieval-Augmented Generation
Liang, N., Yang, C., Yang, K., Gao, J., Wu, Q., & Wang, M. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728644
Yu, X., Huo, Y., Qin, Y., Feng, F., Liu, G., & Xia, D. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729751
Capability-Aware Dynamic Skill Selection for Memory-Augmented Generation
Qiao, Y., Wang, Y., Huang, C., Zhang, X., & Shang, S. (2026) · International Conference on Information Science and Technology · doi:10.1109/icist71072.2026.11727740
Diagnosing the limits of large language models for structured scientific evidence extraction
Tan, Z., D’Souza, J., & Auer, S. (2026) · International Journal on Digital Libraries · doi:10.1007/s00799-026-00449-9
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
Bogin, B., Yang, K., Gupta, S., Richardson, K., Bransom, E., Clark, P., Sabharwal, A., & Khot, T. (2024) · arXiv · doi:10.48550/arxiv.2409.07440
Metadata
- Authors
- Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Corresponding author
- Press, Ofir
- ORCID
- Smith, Noah
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 5687-5711
- OpenAlex
- W4389520103
- Cited by
- 303
- References
- 61
- Citations by year
- 2026: 97 · 2025: 90 · 2024: 46 · 2023: 69
- FWCI
- 53.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2113530)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- compositional reasoning, multi-hop question answering, chain-of-thought prompting, GPT-3, model size scaling, structured prompting, search engine integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
