The DOI matches its DataCite record; 59 of the 61 references it cites were found in a registry or an index.
Measuring and Narrowing the Compositionality Gap in Language Models
Open accessAbstract
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease. This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning. We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly. We present a new method, self-ask, that further improves on chain of thought. In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question. We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing benchmarks, such as HotpotQA [59], Bamboogle [38], and 2wiki [21], are often cited as multi-hop benchmarks, yet they typically involve only two information-seeking steps. Moreover, these datasets are not explicitly structured to support long-horizon interactions that necessitate the agent to manage the memory state.”
From MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents (2025) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023), as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Press et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Simpler decomposition methods like Self-Ask [21] often broaden context too aggressively, triggering the dilution trap.”
From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023)”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Press et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“SelfAsk (Press et al., 2022) prompts LLMs to decompose a question into subquestions and answers these subquestions by issuing separate calls to the Google Search API.”
From Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023) · cites this paper as Press et al., 2022
Experimental — accuracy not yet measured
The cited source says what the claim attributes to it.
In the citing paper
“More sophisticated RAG methods, which perform multiple steps of retrieval and generation from an LLM, are even able to integrate information across new or updated knowledge elements[38, 64, 72, 78, 88, 90, 92], another crucial aspect of long-term memory systems.”
From HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) · cites this paper as [64]
The cited source says what the claim attributes to it.
In the citing paper
“Our experiments show that FRESH- PROMPT outperforms both competing search engine-augmented prompting methods such as SELF-ASK (Press et al., 2022) as well as commercial systems such as PERPLEXITY.AI.”
From FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (2024) · cites this paper as Press et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Lastly, we note that our QA scores exceed those reported by recent works on few-shot prompting for open-domain QA (ODQA) (Khot et al., 2023;Press et al., 2022;Yao et al., 2022), although a fair apples-to-apples comparison with them isn't possible (cf.Appendix C).”
From Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023) · cites this paper as Press et al., 2022
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“IRCoT is SOTA for few-shot multistep ODQA.14”
From Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023) · cites this paper as Press et al., 2022
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Self-citationA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,146
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109
Flagged first, then by citations
2 of the 61 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Sewon Min, Victor Zhong, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2019. Multi-hop reading compre- hension through question decomposition and rescor- ing. ArXiv, abs/1906.02916. · arXiv
- Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot in-context learning. ArXiv, abs/2205.03401. · arXiv
Cited by
Sorted by citations
Items 101–200 of 326
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10
“There has been significant interest recently in understanding and improving the reasoning abilities of Transformer-based models (Cobbe et al., 2021; Magister et al., 2023; Wei et al., 2022; Dziri et al., 2023; Suzgun et al., 2022; Lewkowycz et al., 2022a; Press et al., 2023).”
BackgroundZhao, B., Zhang, Y., Xu, Z., Ren, Y., Zhang, X., Luo, R., Feng, Z., & Xia, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761103 · Cited by 14
Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 71
FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline
Singh, K., Kaur, S., & Smiley, C. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698682 · Cited by 15
Yuan, Z., Liu, K., Peng, R., Li, S., Leybourne, D. J., Musa, N., Huang, H., & Yang, P. (2024) · IEEE Internet of Things Magazine · doi:10.1109/iotm.001.2400036 · Cited by 11
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Nair, I., Somasundaram, S., Saxena, A., & Goswami, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.972 · Cited by 18
Quantum Hamiltonian embedding of images for data reuploading classifiers
Wang, P., Myers, C. R., Hollenberg, L. C. L., & Parampalli, U. (2025) · Quantum Machine Intelligence · doi:10.1007/s42484-025-00247-7 · Cited by 15
Causal Reasoning of Entities and Events in Procedural Texts
Zhang, L., Xu, H., Yang, Y., Zhou, S., You, W., Arora, M., & Callison-Burch, C. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.31 · Cited by 14
Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning
Wang, J., Li, J., & Zhao, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.179 · Cited by 9
Getting MoRE out of Mixture of Language Model Reasoning Experts
Si, C., Shi, W., Zhao, C., Zettlemoyer, L., & Boyd-Graber, J. L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.552 · Cited by 32
A Communication Theory Perspective on Prompting Engineering Methods for Large Language Models
Song, Y., He, Y., Zhao, X.-F., Gu, H., Jiang, D., Yang, H., & Fan, L. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-4058-8 · Cited by 11
LLMs as models for analogical reasoning
Musker, S., Duchnowski, A., Millière, R., & Pavlick, E. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104676 · Cited by 34
Yuan, Z., Liu, K., Li, S., Peng, R., Leybourne, D. J., Musa, N., & Yang, P. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3586374 · Cited by 10
Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models
Zhan, H., Ong, D. C., & Li, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.962 · Cited by 20
A Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8
Li, R., Wang, Y., Wen, Z., Cui, M., & Miao, Q. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112789 · Cited by 10
Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank
Kumar, N. A., Fernandez, N., Wang, Z., & Lan, A. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.22 · Cited by 13
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2023) · arXiv · Cited by 7
Ontology-Constrained Generation of Domain-Specific Clinical Summaries
Mehenni, G., & Zouaq, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_23 · Cited by 8
Han, Z., Wang, J., Yan, X., Jiang, Z., Zhang, Y., Liu, S., Gong, Q., & Song, C. (2025) · Applied Sciences · doi:10.3390/app15105663 · Cited by 7
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier
Macdonald, C., Fang, J., Parry, A., & Meng, Z. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730150 · Cited by 9
Semantic-aware fake news detection with heterogeneous graph attention
Chen, J., Zhou, G., Lan, M., Wang, S., Li, S., & Lu, J. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00966-0 · Cited by 6
Navigating the Landscape of Hint Generation Research: From the Past to the Future
Jangra, A., Mozafari, J., Jatowt, A., & Muresan, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00751 · Cited by 15
Watson, E. ‘., & Hessami, A. G. (2025) · Electronics · doi:10.3390/electronics14163162 · Cited by 7
AutoMix: Automatically Mixing Language Models
Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.12963 · Cited by 19
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
Sun, Z., Wang, Q., Yu, W., Zang, X., Zheng, K., Xu, J., Zhang, X., Song, Y., & Li, H. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730102 · Cited by 5
Enhancing long-form question answering via reflection with question decomposition
Xiao, J., Wu, W., Zhao, J., Fang, M., & Wang, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104274 · Cited by 6
AMHR COLIEE 2024 Entry: Legal Entailment and Retrieval
Nighojkar, A., Jiang, K. Y., Fields, L., Bilgin, O., Steinle, S., Sadybekov, Y., Marji, Z., & Licato, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-3076-6_14 · Cited by 5
CIRAG: Retrieval-Augmented Language Model with Collective Intelligence
Cui, C., Fan, H., Zhang, J., Shen, L., Li, B., & Wang, W. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729921 · Cited by 5
Allies: Prompting Large Language Model with Beam Search
Sun, H., Liu, X., Gong, Y., Zhang, Y., Jiang, D., Yang, L., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.247 · Cited by 13
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Goyal, N., Briakou, E., Liu, A., Baumler, C., Bonial, C., Micher, J. C., Voss, C. R., Carpuat, M. J., & Daumé, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.201 · Cited by 12
Exploring Prompting Approaches in Legal Textual Entailment
Bilgin, O., Fields, L., Laverghetta, A., Marji, Z., Nighojkar, A., Steinle, S., & Licato, J. (2024) · The Review of Socionetwork Strategies · doi:10.1007/s12626-023-00154-y · Cited by 5
Hint-Enhanced In-Context Learning Wakes Large Language Models Up For Knowledge-Intensive Tasks
Wang, Y., Guo, Q., Ni, X., Shi, C., Liu, L., Jiang, H., & Yang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447527 · Cited by 9
Zhu, Y., Khoo, Z.-Y., Low, J. S. C., & Bressan, S. (2024) · Conference on Algebraic Informatics · doi:10.1109/cai59869.2024.00017 · Cited by 5
Jadhav, S., Perumal, S., Tadavi, Y., Dash, B., & Parthiban, S. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717817 · Cited by 4
Fine-Grained Entity Recognition via Large Language Models
Qiao, X., Gu, S., Cheng, J., Chen, P., Xiong, Z., Shen, H., & Jiang, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3574197 · Cited by 4
Leveraging Metadata and LLM Context to Overcome RAG Challenges and Limitations
Mukherjee, S. (2025) · doi:10.1109/icbdse65491.2025.11220064 · Cited by 4
Sornlertlamvanich, V. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02485-4 · Cited by 4
Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Singh, C., Morris, J. X., Rush, A. M., Gao, J., & Deng, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.384 · Cited by 18
CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
Kim, S., Joo, S. J., Jang, Y., Chae, H., & Yeo, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.23 · Cited by 5
HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution
Ye, L., Yavuz, S., Meng, R., Radev, D., Xiong, C., Joty, S., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.293 · Cited by 8
Automated Theorem Provers Help Improve Large Language Model Reasoning
McGinness, L. P., & Baumgartner, P. (2024) · EPiC series in computing · doi:10.29007/2n9m · Cited by 9
CiteME: Can Language Models Accurately Cite Scientific Claims?
Press, O., Hochlehnert, A., Prabhu, A., Udandarao, V., Press, O., & Bethge, M. (2024) · arXiv · doi:10.48550/arxiv.2407.12861 · Cited by 4
Self-citationLiao, H., Tian, J., Hu, S., Zhu, Z., He, H., & Jin, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887828 · Cited by 6
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for Deep Research
Rosset, C., Chung, H.-L., Qin, G., Chau, E. C., Feng, Z., Awadallah, A. H., Neville, J., & Rao, N. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730275 · Cited by 49
LLM-based Crowdsourced Test Report Clustering
Ling, Y., Yu, S., Fang, C., Zhou, Q., & Chen, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3765756 · Cited by 4
Wang, C., Chai, S., Xu, T., Adil, M., & Qiu, T. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3652422 · Cited by 4
q2d: Turning Questions into Dialogs to Teach Models How to Search
Bitton, Y., Cohen-Ganor, S., Hakimi, I., Lewenberg, Y., Aharoni, R., & Weinreb, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.843 · Cited by 7
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 7
RECoT: Relation-enhanced Chains-of-Thoughts for knowledge-intensive multi-hop questions answering
Li, R., Li, D., Yang, H., Liu, X., Jin, H., Pu, R., & Miao, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129903 · Cited by 5
Santra, P., Ghosh, M., Ganguly, D., Basuchowdhuri, P., & Naskar, S. K. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70021 · Cited by 5
E-InMeMo: Enhanced Prompting for Visual In-Context Learning
Zhang, J., Wang, B., Liu, H., Li, L., Nakashima, Y., & Nagahara, H. (2025) · Journal of Imaging · doi:10.3390/jimaging11070232 · Cited by 4
Analyzing Modular Approaches for Visual Question Decomposition
Khandelwal, A., Pavlick, E., & Sun, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.157 · Cited by 6
Xu, T., Piao, Z. H., Mukai, T., Murayama, Y., & Izumi, K. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200496 · Cited by 4
Zhu, Y., Huang, Z., Dou, Z., & Wen, J.-R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i24.34813 · Cited by 13
Discovering technology-aided possibilities for automatic analysis of science teacher questions
Lehesvuori, S., Urrutia, F., Heilala, V., Araya, R., & Hämäläinen, R. (2025) · Educational Technology Research and Development · doi:10.1007/s11423-025-10545-3 · Cited by 6
Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence
Queloz, M. (2025) · Minds and Machines · doi:10.1007/s11023-025-09738-9 · Cited by 3
A Systematic Literature Review on SWOT Analysis of Prompt Engineering Techniques
Singh, A., Chatta, N. K., Ehtesham, A., Kumar, S., Gupta, G. K., & Khoei, T. T. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3626467 · Cited by 2
Towards philosophical reasoning with agentic LLMs: Socratic method for scientific assistance
Harb, H., Sun, Y., Unal, M., Aggarwal, A., Bissolotti, C., Buyuker, I. S., Hong, S., Johnson, L. R., Jolaoso, L. A., Sengupta, B., et al. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae277f · Cited by 3
Do Large Language Models Know When They Lack Knowledge?
Qin, S., Zhou, L., Sun, L., & wang, N. (2026) · Electronics · doi:10.3390/electronics15020253 · Cited by 2
Exploring Knowledge Filtering for Retrieval-Augmented Question Answering
Qiang, M., Wang, Z., Li, S., & Zhou, G. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3658957 · Cited by 2
Pang, R. Y., Feng, K. J. K., Feng, S., Li, C., Shi, W., Tsvetkov, Y., Heer, J., & Reinecke, K. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789091 · Cited by 31
Dual-stage and Lightweight Patient Chart Summarization for Emergency Physicians
Wu, J., Zaidi, S., Teitge, B. D., Leung, H., Zhou, J., Holodinsky, J. K., & Drew, S. (2025) · IEEE Annual Congress on Artificial Intelligence of Things (AIoT) · doi:10.1109/aiot66900.2025.00018 · Cited by 3
PPR-SQL: a robust text-to-SQL system via probabilistic graph pruning and multi-view reasoning
Xu, X., Jiang, X., Wang, A., Gan, Z., & Yu, B. (2026) · Journal of Intelligent Information Systems · doi:10.1007/s10844-026-01049-4 · Cited by 2
DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models
Han, C., Du, X., Che, Z., Lian, Y., Li, X., Gao, M. Y., & Wang, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.501 · Cited by 6
Case Study of a Class on Classical Narratives Using ChatGPT
Lee, M.-H., & Kim, J.-W. (2023) · JAHR · doi:10.21860/j.14.2.7 · Cited by 5
Evaluating large language models’ ability to generate interpretive arguments
Marji, Z., & Licato, J. (2024) · Argument & Computation · doi:10.3233/aac-230014 · Cited by 2
REA: Refine-Estimate-Answer Prompting for Zero-Shot Relation Extraction
Layegh, A., Payberah, A. H., & Matskin, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70239-6_21 · Cited by 2
KnowTrace: Bootstrapping Iterative Retrieval-Augmented Generation with Structured Knowledge Tracing
Li, R. L., Dai, Q., Zhang, Z., Chen, X., Dong, Z., & Wen, J.-R. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737015 · Cited by 8
AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning
Xin, A., Liu, J., Yao, Z., Lee, Z., Cao, S., Hou, L., & Li, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736849 · Cited by 12
Misra, K., dos Santos, C. N., & Shakeri, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.62 · Cited by 4
PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering
Han, W., Park, J., & Lee, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.517 · Cited by 2
Watson, W., Cho, N., Balch, T. R., & Veloso, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.442 · Cited by 5
Improving OpenDevin: Boosting code generation LLM through advanced memory management
He, R., Ying, A., & Hu, X. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/68/20241506 · Cited by 1
Floyd, S., Jouravlev, O., Mineroff, Z., Poliak, M., Gibson, E. A. F., & Fedorenko, E. G. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/e2xta · Cited by 3
Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
Zhou, H., Chen, Y., Guo, S., Yan, X., Lee, K. H., Wang, Z., Lee, K. Y., Zhang, G., Shao, K., Yang, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2508.16153 · Cited by 2
“Specifically, MusiQue (Trivedi et al., 2022), Bamboogle (Press et al., 2022), and PopQA (Mallen et al., 2022) are selected as OOD datasets due to their distinct question styles and information distributions, while NQ (Kwiatkowski et al., 2019), TQ (Joshi et al., 2017), HotpotQA (Yang et al., 2018),…”
InfluentialFeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
Dai, D., Liu, M., Li, A., Cao, J., Wang, Y., Wang, C., Peng, X., & Zheng, Z. (2025) · arXiv · doi:10.48550/arxiv.2504.06939 · Cited by 1
“• Self-Ask Prompting (SA) [33] : Instructs the model to generate clarifying questions before repair, promoting problem decomposition and identification of ambiguities.”
From Complex to Simple: Unraveling the Cognitive Tree for Reasoning with Small Language Models
Yan, J., Wang, C., Zhang, T., He, X., Huang, J., & Zhang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.828 · Cited by 5
Answering Spatial Commonsense Questions Based on Chain-of-Thought Reasoning with Adaptive Complexity
Yin, H., Yu, J., Lin, M., & Wang, S. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7232-2_13 · Cited by 3
WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
Liu, J., Ning, K., Su, Y., Han, W., Xu, J., & Zhang, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889153 · Cited by 21
A Retrieval-Augmented Framework For Meeting Insight Extraction
Bhuvaji, S., Chouhan, P., Irukulla, M., Singhvi, J., Bae, W. D., & Alkobaisi, S. (2025) · ACM Symposium on Applied Computing · doi:10.1145/3672608.3707915 · Cited by 2
Liu, T., Song, H., Guo, L., Li, B., Lin, Z., Gao, Y., & Ge, W. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3580160 · Cited by 1
AgentTOD: A Task-Oriented Dialogue Agent with a Flexible and Adaptive API Calling Paradigm
Xu, H.-D., Mao, X.-L., Sun, F., Che, T.-Y., Xu, C., & Huang, H. (2025) · ACM Transactions on Information Systems · doi:10.1145/3745021 · Cited by 1
Open-World Knowledge Augmentation for Zero-Shot Information Extraction in LLMs
Li, H., Zhou, C., Zhang, J., & Hu, D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_26 · Cited by 1
Enhancing Retrieval-Augmented Generation for Text Completion Through Query Selection
Pogrebinsky, I., Carmel, D., & Kurland, O. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744610 · Cited by 2
Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning
Wang, J., Luo, T., Zha, Y., Feng, Y., Luo, R., Chen, B., Dai, T., Chen, L., Wang, Y., & Xia, S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02342 · Cited by 1
RAP-RAG: A Retrieval-Augmented Generation Framework with Adaptive Retrieval Task Planning
Ji, X., Luo, X., Gu, L., Ma, J., Zhang, Z., & Jiang, W. T. (2025) · Electronics · doi:10.3390/electronics14214269 · Cited by 3
Learnable Game-Theoretic Policy Optimization for Data-Centric Self-Explanation Rationalization
Zhao, Y., Wang, Z., Yu, X., Li, X., Liang, J., & Li, R. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3638864 · Cited by 5
Wan, J., Li, X., Zhang, H., Zou, A., & Wang, R. (2025) · npj Heritage Science · doi:10.1038/s40494-025-02219-3 · Cited by 2
Early Language and Evaluation Glossary – Historical Record Archive
Jacobs, A., & Lab, S. F. (2025) · Figshare · doi:10.6084/m9.figshare.30500330 · Cited by 1
Junqi, B. A. I., Ning, D., You, Y., & Chen, J. (2026) · Buildings · doi:10.3390/buildings16010196 · Cited by 1
Zhao, Y., Fan, Z., Cao, Y., Lyu, Z., & Li, J. (2026) · Electronics · doi:10.3390/electronics15051058 · Cited by 1
Murugaraj, K., Lamsiyah, S., & Theobald, M. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402490 · Cited by 1
Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answering
Shi, Y., Sun, M., Liu, Z., Yang, M., Fang, Y., Sun, T., & Gu, X. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774905.3794658 · Cited by 19
Yuan, M., Wang, T., & Qian, Y. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01028-3 · Cited by 3
Uncertainty Quantification for Retrieval-Augmented Reasoning
Soudani, H., Zamani, H., & Hasibi, F. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809717 · Cited by 9
FinePrompt: Unveiling the Role of Finetuned Inductive Bias on Compositional Reasoning in GPT-4
Kim, J.-H., Hong, G., Myaeng, S.-H., & Whang, J. J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.245 · Cited by 1
Kulikov, V., Neychev, R., & Makarov, I. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-67008-4_4 · Cited by 1
ToFC: Tree-of-Fact with Continued Best-First Search for Commonsense Reasoning
Dai, L., & Qin, K. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10649921 · Cited by 1
Retrieval Augmented Tree of Thoughts
Liu, X., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_36 · Cited by 1
Metadata
- Authors
- Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Corresponding author
- Press, Ofir
- ORCID
- Smith, Noah A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 5687-5711
- arXiv
- 2210.03350
- OpenAlex
- W4389520103
- Cited by
- 294
- References
- 61
- Citations by year
- 2026: 90 · 2025: 90 · 2024: 44 · 2023: 69 · 2022: 1
- FWCI
- 52.09
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2113530)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- compositional reasoning, multi-hop question answering, chain-of-thought prompting, GPT-3, model size scaling, structured prompting, search engine integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
