Cited by 321Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

Measuring and Narrowing the Compositionality Gap in Language Models

Open access

Press, Ofir, Zhang, Muru, Min, Sewon, Schmidt, Ludwig, Smith, Noah, Lewis, Mike·2023·Findings of the Association for Computational Linguistics: EMNLP 2023

DOI 10.18653/v1/2023.findings-emnlp.378

Open this paper in CoChat — free account Open PDF

Abstract

We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap.We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining.In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease.This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning.We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly.We present a new method, self-ask, that further improves on chain of thought.In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question.We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.378

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Self-citation
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
A Neural Conversational Model
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,146

Show all 61 references

Sorted by citations

Items 101–200 of 321

Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 71
FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline
Singh, K., Kaur, S., & Smiley, C. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698682 · Cited by 15
PestGPT: Leveraging Large Language Models and IoT for Timely and Customized Recommendation Generation in Sustainable Pest Management
Yuan, Z., Liu, K., Peng, R., Li, S., Leybourne, D. J., Musa, N., Huang, H., & Yang, P. (2024) · IEEE Internet of Things Magazine · doi:10.1109/iotm.001.2400036 · Cited by 11
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Nair, I., Somasundaram, S., Saxena, A., & Goswami, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.972 · Cited by 18
Quantum Hamiltonian embedding of images for data reuploading classifiers
Wang, P., Myers, C. R., Hollenberg, L. C. L., & Parampalli, U. (2025) · Quantum Machine Intelligence · doi:10.1007/s42484-025-00247-7 · Cited by 15
Causal Reasoning of Entities and Events in Procedural Texts
Zhang, L., Xu, H., Yang, Y., Zhou, S., You, W., Arora, M., & Callison-Burch, C. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.31 · Cited by 14
Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning
Wang, J., Li, J., & Zhao, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.179 · Cited by 9
Getting MoRE out of Mixture of Language Model Reasoning Experts
Si, C., Shi, W., Zhao, C., Zettlemoyer, L., & Boyd-Graber, J. L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.552 · Cited by 32
A Communication Theory Perspective on Prompting Engineering Methods for Large Language Models
Song, Y., He, Y., Zhao, X.-F., Gu, H., Jiang, D., Yang, H., & Fan, L. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4058-8 · Cited by 11
LLMs as models for analogical reasoning
Musker, S., Duchnowski, A., Millière, R., & Pavlick, E. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104676 · Cited by 34
PEZEGO: A Precision Agriculture System Based on Large Language Models and Internet of Things for Pest Management
Yuan, Z., Liu, K., Li, S., Peng, R., Leybourne, D. J., Musa, N., & Yang, P. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3586374 · Cited by 10
Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models
Zhan, H., Ong, D. C., & Li, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.962 · Cited by 20
Different paths to the same destination: Diversifying LLMs generation for multi-hop open-domain question answering
Li, R., Wang, Y., Wen, Z., Cui, M., & Miao, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112789 · Cited by 10
Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank
Kumar, N. A., Fernandez, N., Wang, Z., & Lan, A. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.22 · Cited by 13
Ontology-Constrained Generation of Domain-Specific Clinical Summaries
Mehenni, G., & Zouaq, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_23 · Cited by 8
CoReaAgents: A Collaboration and Reasoning Framework Based on LLM-Powered Agents for Complex Reasoning Tasks
Han, Z., Wang, J., Yan, X., Jiang, Z., Zhang, Y., Liu, S., Gong, Q., & Song, C. (2025) · Applied Sciences · doi:10.3390/app15105663 · Cited by 7
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier
Macdonald, C., Fang, J., Parry, A., & Meng, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730150 · Cited by 9
Semantic-aware fake news detection with heterogeneous graph attention
Chen, J., Zhou, G., Lan, M., Wang, S., Li, S., & Lu, J. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00966-0 · Cited by 6
Navigating the Landscape of Hint Generation Research: From the Past to the Future
Jangra, A., Mozafari, J., Jatowt, A., & Muresan, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00751 · Cited by 15
Psychopathia Machinalis: A Nosological Framework for Understanding Pathologies in Advanced Artificial Intelligence
Watson, E. ‘., & Hessami, A. G. (2025) · Electronics · doi:10.3390/electronics14163162 · Cited by 7
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
Sun, Z., Wang, Q., Yu, W., Zang, X., Zheng, K., Xu, J., Zhang, X., Song, Y., & Li, H. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730102 · Cited by 5
Enhancing long-form question answering via reflection with question decomposition
Xiao, J., Wu, W., Zhao, J., Fang, M., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104274 · Cited by 6
AMHR COLIEE 2024 Entry: Legal Entailment and Retrieval
Nighojkar, A., Jiang, K. Y., Fields, L., Bilgin, O., Steinle, S., Sadybekov, Y., Marji, Z., & Licato, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-3076-6_14 · Cited by 5
CIRAG: Retrieval-Augmented Language Model with Collective Intelligence
Cui, C., Fan, H., Zhang, J., Shen, L., Li, B., & Wang, W. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729921 · Cited by 5
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 13
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Goyal, N., Briakou, E., Liu, A., Baumler, C., Bonial, C., Micher, J. C., Voss, C. R., Carpuat, M. J., & Daumé, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.201 · Cited by 12
Exploring Prompting Approaches in Legal Textual Entailment
Bilgin, O., Fields, L., Laverghetta, A., Marji, Z., Nighojkar, A., Steinle, S., & Licato, J. (2024) · The Review of Socionetwork Strategies · doi:10.1007/s12626-023-00154-y · Cited by 5
Hint-Enhanced In-Context Learning Wakes Large Language Models Up For Knowledge-Intensive Tasks
Wang, Y., Guo, Q., Ni, X., Shi, C., Liu, L., Jiang, H., & Yang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447527 · Cited by 9
A Personalised Learning Tool for Physics Undergraduate Students Built On a Large Language Model for Symbolic Regression
Zhu, Y., Khoo, Z.-Y., Low, J. S. C., & Bressan, S. (2024) · Conference on Algebraic Informatics · doi:10.1109/cai59869.2024.00017 · Cited by 5
Leveraging Large Language Models for Biomedical Knowledge Graph Construction and Querying: An Advanced NLP Approach
Jadhav, S., Perumal, S., Tadavi, Y., Dash, B., & Parthiban, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717817 · Cited by 4
Fine-Grained Entity Recognition via Large Language Models
Qiao, X., Gu, S., Cheng, J., Chen, P., Xiong, Z., Shen, H., & Jiang, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3574197 · Cited by 4
Leveraging Metadata and LLM Context to Overcome RAG Challenges and Limitations
Mukherjee, S. (2025) · doi:10.1109/icbdse65491.2025.11220064 · Cited by 4
Enhancing large language models: alleviating knowledge deficiency with external knowledge and semantically aware reasoning (SAR)
Sornlertlamvanich, V. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02485-4 · Cited by 4
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Singh, C., Morris, J. X., Rush, A. M., Gao, J., & Deng, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.384 · Cited by 18
CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
Kim, S., Joo, S. J., Jang, Y., Chae, H., & Yeo, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.23 · Cited by 5
HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution
Ye, L., Yavuz, S., Meng, R., Radev, D., Xiong, C., Joty, S., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.293 · Cited by 8
Automated Theorem Provers Help Improve Large Language Model Reasoning
McGinness, L. P., & Baumgartner, P. (2024) · EPiC series in computing · doi:10.29007/2n9m · Cited by 9
Empowering low-resource languages: TraSe architecture for enhanced retrieval-augmented generation in Bangla
Ipa, A. S., Rony, M. A. T., & Islam, M. S. (2025) · Proceedings of the 1st Workshop on Language Models for Underserved Communities (LM4UC 2025) · doi:10.18653/v1/2025.lm4uc-1.2 · Cited by 3
Look Before You Leap: Problem Elaboration Prompting Improves Mathematical Reasoning in Large Language Models
Liao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for Deep Research
Rosset, C., Chung, H.-L., Qin, G., Chau, E. C., Feng, Z., Awadallah, A. H., Neville, J., & Rao, N. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730275 · Cited by 49
LLM-based Crowdsourced Test Report Clustering
Ling, Y., Yu, S., Fang, C., Zhou, Q., & Chen, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3765756 · Cited by 4
CP-RAG: Mitigating Distracting Content in Retrieval-Augmented Generation for Industrial Knowledge Question Answering
Wang, C., Chai, S., Xu, T., Adil, M., & Qiu, T. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3652422 · Cited by 4
q2d: Turning Questions into Dialogs to Teach Models How to Search
Bitton, Y., Cohen-Ganor, S., Hakimi, I., Lewenberg, Y., Aharoni, R., & Weinreb, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.843 · Cited by 7
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 7
RECoT: Relation-enhanced Chains-of-Thoughts for knowledge-intensive multi-hop questions answering
Li, R., Li, D., Yang, H., Liu, X., Jin, H., Pu, R., & Miao, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129903 · Cited by 5
The “Curious Case of Contexts” in Retrieval‐Augmented Generation With a Combination of Labeled and Unlabeled Data
Santra, P., Ghosh, M., Ganguly, D., Basuchowdhuri, P., & Naskar, S. K. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70021 · Cited by 5
E-InMeMo: Enhanced Prompting for Visual In-Context Learning
Zhang, J., Wang, B., Liu, H., Li, L., Nakashima, Y., & Nagahara, H. (2025) · Journal of Imaging · doi:10.3390/jimaging11070232 · Cited by 4
Analyzing Modular Approaches for Visual Question Decomposition
Khandelwal, A., Pavlick, E., & Sun, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.157 · Cited by 6
Emulating fundamental analysts: Analytical stage-based multi-agent framework enhanced with expert guidance and Preference-Anchored Likelihood Adjustment
Xu, T., Piao, Z. H., Mukai, T., Murayama, Y., & Izumi, K. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200496 · Cited by 4
One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models
Zhu, Y., Huang, Z., Dou, Z., & Wen, J.-R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34813 · Cited by 13
Context, Reasoning, and Hierarchy: A Cost-Performance Study of Compound LLM Agent Design in an Adversarial POMDP
Bogdanov, I., Lung, C.-H., Kunz, T., Gao, J., Taylor, A., & Zaman, M. (2026) · Proceedings of the ACM Conference on AI and Agentic Systems · doi:10.1145/3786335.3813149 · Cited by 3
Discovering technology-aided possibilities for automatic analysis of science teacher questions
Lehesvuori, S., Urrutia, F., Heilala, V., Araya, R., & Hämäläinen, R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10545-3 · Cited by 6
Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence
Queloz, M. (2025) · Minds and Machines · doi:10.1007/s11023-025-09738-9 · Cited by 3
A Systematic Literature Review on SWOT Analysis of Prompt Engineering Techniques
Singh, A., Chatta, N. K., Ehtesham, A., Kumar, S., Gupta, G. K., & Khoei, T. T. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3626467 · Cited by 2
Towards philosophical reasoning with agentic LLMs: Socratic method for scientific assistance
Harb, H., Sun, Y., Unal, M., Aggarwal, A., Bissolotti, C., Buyuker, I. S., Hong, S., Johnson, L. R., Jolaoso, L. A., Sengupta, B., et al. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae277f · Cited by 3
Do Large Language Models Know When They Lack Knowledge?
Qin, S., Zhou, L., Sun, L., & wang, N. (2026) · Electronics · doi:10.3390/electronics15020253 · Cited by 2
Exploring Knowledge Filtering for Retrieval-Augmented Question Answering
Qiang, M., Wang, Z., Li, S., & Zhou, G. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3658957 · Cited by 2
Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models
Pang, R. Y., Feng, K. J. K., Feng, S., Li, C., Shi, W., Tsvetkov, Y., Heer, J., & Reinecke, K. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789091 · Cited by 31
Dual-stage and Lightweight Patient Chart Summarization for Emergency Physicians
Wu, J., Zaidi, S., Teitge, B. D., Leung, H., Zhou, J., Holodinsky, J. K., & Drew, S. (2025) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot66900.2025.00018 · Cited by 3
PPR-SQL: a robust text-to-SQL system via probabilistic graph pruning and multi-view reasoning
Xu, X., Jiang, X., Wang, A., Gan, Z., & Yu, B. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01049-4 · Cited by 2
DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models
Han, C., Du, X., Che, Z., Lian, Y., Li, X., Gao, M. Y., & Wang, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.501 · Cited by 6
Case Study of a Class on Classical Narratives Using ChatGPT
Lee, M.-H., & Kim, J.-W. (2023) · JAHR · doi:10.21860/j.14.2.7 · Cited by 5
Evaluating large language models’ ability to generate interpretive arguments
Marji, Z., & Licato, J. (2024) · Argument & Computation · doi:10.3233/aac-230014 · Cited by 2
REA: Refine-Estimate-Answer Prompting for Zero-Shot Relation Extraction
Layegh, A., Payberah, A. H., & Matskin, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_21 · Cited by 2
KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing
Li, R. L., Dai, Q., Zhang, Z., Chen, X., Dong, Z., & Wen, J.-R. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737015 · Cited by 8
AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning
Xin, A., Liu, J., Yao, Z., Lee, Z., Cao, S., Hou, L., & Li, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736849 · Cited by 12
Triggering Multi-Hop Reasoning for Question Answering in Language Models using Soft Prompts and Random Walks
Misra, K., dos Santos, C. N., & Shakeri, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.62 · Cited by 4
PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering
Han, W., Park, J., & Lee, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.517 · Cited by 2
HiddenTables and PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
Watson, W., Cho, N., Balch, T. R., & Veloso, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.442 · Cited by 5
Improving OpenDevin: Boosting code generation LLM through advanced memory management
He, R., Ying, A., & Hu, X. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/68/20241506 · Cited by 1
A tripartite structure of pragmatic language abilities: comprehension of social conventions, intonation processing and causal reasoningPreprint
Floyd, S., Jouravlev, O., Mineroff, Z., Poliak, M., Gibson, E. A. F., & Fedorenko, E. G. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/e2xta · Cited by 3
TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review
Chang, Y., Li, Z., Zhang, H., Kong, Y., Wu, Y., So, H. K.-H., Guo, Z., Zhu, L., & Wong, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.790 · Cited by 2
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models
Yan, J., Wang, C., Zhang, T., He, X., Huang, J., & Zhang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.828 · Cited by 5
Answering Spatial Commonsense Questions Based on Chain-of-Thought Reasoning with Adaptive Complexity
Yin, H., Yu, J., Lin, M., & Wang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7232-2_13 · Cited by 3
WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
Liu, J., Ning, K., Su, Y., Han, W., Xu, J., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889153 · Cited by 21
A Retrieval-Augmented Framework For Meeting Insight Extraction
Bhuvaji, S., Chouhan, P., Irukulla, M., Singhvi, J., Bae, W. D., & Alkobaisi, S. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707915 · Cited by 2
Phone-to-EDU: A Smart Contract-Based Framework for Comprehensive Management of GAI-Assisted Programming Courses
Liu, T., Song, H., Guo, L., Li, B., Lin, Z., Gao, Y., & Ge, W. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3580160 · Cited by 1
AgentTOD: A Task-Oriented Dialogue Agent with a Flexible and Adaptive API Calling Paradigm
Xu, H.-D., Mao, X.-L., Sun, F., Che, T.-Y., Xu, C., & Huang, H. (2025) · ACM Transactions on Information Systems · doi:10.1145/3745021 · Cited by 1
Open-World Knowledge Augmentation for Zero-Shot Information Extraction in LLMs
Li, H., Zhou, C., Zhang, J., & Hu, D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_26 · Cited by 1
Enhancing Retrieval-Augmented Generation for Text Completion Through Query Selection
Pogrebinsky, I., Carmel, D., & Kurland, O. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744610 · Cited by 2
Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning
Wang, J., Luo, T., Zha, Y., Feng, Y., Luo, R., Chen, B., Dai, T., Chen, L., Wang, Y., & Xia, S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02342 · Cited by 1
RAP-RAG: A Retrieval-Augmented Generation Framework with Adaptive Retrieval Task Planning
Ji, X., Luo, X., Gu, L., Ma, J., Zhang, Z., & Jiang, W. T. (2025) · Electronics · doi:10.3390/electronics14214269 · Cited by 3
Learnable Game-Theoretic Policy Optimization for Data-Centric Self-Explanation Rationalization
Zhao, Y., Wang, Z., Yu, X., Li, X., Liang, J., & Li, R. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3638864 · Cited by 5
An LLM-based QA system for Chinese Painting and Calligraphy with Knowledge Graphs and external documents
Wan, J., Li, X., Zhang, H., Zou, A., & Wang, R. (2025) · npj Heritage Science · doi:10.1038/s40494-025-02219-3 · Cited by 2
Early Language and Evaluation Glossary – Historical Record Archive
Jacobs, A., & Lab, S. F. (2025) · Figshare · doi:10.6084/m9.figshare.30500330 · Cited by 1
LoopRAG: A Closed-Loop Multi-Agent RAG Framework for Interactive Semantic Question Answering in Smart Buildings
Junqi, B. A. I., Ning, D., You, Y., & Chen, J. (2026) · Buildings · doi:10.3390/buildings16010196 · Cited by 1
RewriteGen: Autonomous Query Optimization for Retrieval-Augmented Large Language Models via Reinforcement Learning
Zhao, Y., Fan, Z., Cao, Y., Lyu, Z., & Li, J. (2026) · Electronics · doi:10.3390/electronics15051058 · Cited by 1
ExpandFuse: A Hybrid Retrieval Framework with Query Expansion and Topic-Aware Reranking for Multi-Hop Question Answering
Murugaraj, K., Lamsiyah, S., & Theobald, M. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402490 · Cited by 1
Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answering
Shi, Y., Sun, M., Liu, Z., Yang, M., Fang, Y., Sun, T., & Gu, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3794658 · Cited by 19
AdaKG-RAG: Adaptive KG-guided retrieval-augmented generation with hypothesis-and-verification retrieval for multi-hop question answering
Yuan, M., Wang, T., & Qian, Y. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01028-3 · Cited by 3
Uncertainty Quantification for Retrieval-Augmented Reasoning
Soudani, H., Zamani, H., & Hasibi, F. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809717 · Cited by 9
FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4
Kim, J.-H., Hong, G., Myaeng, S.-H., & Whang, J. J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.245 · Cited by 1
Whether Large Language Models Learn at the Inference Stage? Effects of Active Learning and Labelling with LLMs on their Reasoning
Kulikov, V., Neychev, R., & Makarov, I. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-67008-4_4 · Cited by 1
ToFC: Tree-of-Fact with Continued Best-First Search for Commonsense Reasoning
Dai, L., & Qin, K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649921 · Cited by 1
Retrieval Augmented Tree of Thoughts
Liu, X., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_36 · Cited by 1
Natural Language Querying on Domain-Specific NoSQL Database with Large Language Models
Zhang, W., He, C., Yang, G., Bandyopadhyay, D., Shi, T., & Wang, P. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10822485 · Cited by 1
Investigating ChatGPT Translation Hallucination from an Embodied-Cognitive Translatology Perspective
Jiao, H., Li, X., Ding, J., & Xiao-jun, Z. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-2885-8_12 · Cited by 1
Efficient and Deployable Knowledge Infusion for Open-World Recommendations via Large Language Models
Xi, Y., Liu, W., Lin, J., Weng, M., Cai, X., Zhu, H. C., Zhu, J., Chen, B., Tang, R., Yu, Y. H., et al. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3725894 · Cited by 12
Towards a Probabilistic Framework for Analyzing and Improving LLM-Enabled Software
Baldonado, J. M., Bonomo, F., & Braberman, V. A. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962470 · Cited by 2
Moral Judgment with a Large Language Model-Based Agent
Xiong, S., Gu, H., Liang, W., & Yin, L. (2025) · Electronics · doi:10.3390/electronics14132580 · Cited by 1

Authors: Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M.
Affiliations: University of Washington (US)
Allen Institute for Artificial Intelligence (US)
Corresponding author: Press, Ofir
ORCID: Smith, Noah
Venue: Findings of the Association for Computational Linguistics: EMNLP 2023
Publisher: Association for Computational Linguistics
Published: 2023
Pages: 5687-5711
DOI: 10.18653/v1/2023.findings-emnlp.378
OpenAlex: W4389520103
Cited by: 303
References: 61
Citations by year: 2026: 97 · 2025: 90 · 2024: 46 · 2023: 69
FWCI: 53.42
Type: conference paper
Language: en
Open access: hybrid
doi.org
Licence: cc-by
Locations: Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Funders: National Science Foundation (2113530)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords: compositional reasoning, multi-hop question answering, chain-of-thought prompting, GPT-3, model size scaling, structured prompting, search engine integration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Measuring and Narrowing the Compositionality Gap in Language Models

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks