Measuring and Narrowing the Compositionality Gap in Language Models
Open accessAbstract
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap.We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining.In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease.This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning.We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly.We present a new method, self-ask, that further improves on chain of thought.In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question.We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies have demonstrated limited generalization capabilities of LLMs, with deficiencies in planning, abstraction, and compositionality.
“recent studies have demonstrated the limited generalization capabilities of LLMs, with deficiencies in planning15, abstraction16, and compositionality17 across various tasks.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Self-citationLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 1,981 · see citing works
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,009 · see citing works
Decomposed Prompting: A Modular Approach for Solving Complex TasksPreprint
Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., & Sabharwal, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.02406 · Cited by 852 · see citing works
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · see citing works
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J., Gao, J., & Dolan, B. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1020 · Cited by 768 · see citing works
SkipNet: Learning Dynamic Routing in Convolutional Networks
Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., & Gonzalez, J. E. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_25 · Cited by 643 · see citing works
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638 · see citing works
Multi-Scale Dense Networks for Resource Efficient Image ClassificationPreprint
Huang, G. M., Chen, D., Li, T., Wu, F., van der Maaten, L., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09844 · Cited by 638 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · see citing works
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
The Web as a Knowledge-Base for Answering Complex Questions
Talmor, A., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1059 · Cited by 520 · see citing works
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382 · see citing works
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380 · see citing works
The Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A. D., Serban, I. V., Lowe, R., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-29135-8_7 · Cited by 362 · see citing works
The Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.00098 · Cited by 355 · see citing works
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335 · see citing works
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · see citing works
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
Self-citation♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · see citing works
oLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 267 · see citing works
Adaptive Neural Networks for Efficient Inference
Bolukbasi, T., Wang, J., Dekel, O., & Saligrama, V. (2017) · arXiv · doi:10.48550/arxiv.1702.07811 · Cited by 251 · see citing works
Search-based Neural Structured Learning for Sequential Question Answering
Iyyer, M., Yih, W.-T., & Chang, M. (2017) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p17-1167 · Cited by 219 · see citing works
Measuring Compositional Generalization: A Comprehensive Method on Realistic DataPreprint
Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, Ł., Tihon, T., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09713 · Cited by 213 · see citing works
Neural Responding Machine for Short-Text ConversationPreprint
Shang, L., Lu, Z., & Li, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.02364 · Cited by 212 · see citing works
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 206 · see citing works
Multi-hop Reading Comprehension through Question Decomposition and Rescoring
Min, S., Zhong, V., Zettlemoyer, L., & Hajishirzi, H. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1613 · Cited by 197 · see citing works
Self-citationCompositionality Decomposed: How do Neural Networks Generalise?
Hupkes, D., Dankers, V., Mul, M., & Bruni, E. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11674 · Cited by 194 · see citing works
Unsupervised Commonsense Question Answering with Self-Talk
Shwartz, V., West, P., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.373 · Cited by 186 · see citing works
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · see citing works
Reframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140 · see citing works
The Right Tool for the Job: Matching Model and Instance Complexities
Schwartz, R., Stanovsky, G., Swayamdipta, S., Dodge, J., & Smith, N. A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.593 · Cited by 136 · see citing works
Self-citationAsk the Right Questions: Active Question Reformulation with Reinforcement LearningPreprint
Buck, C., Bulian, J., Ciaramita, M., Gajewski, W., Gesmundo, A., Houlsby, N., & Wang, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07830 · Cited by 127 · see citing works
Unsupervised Question Decomposition for Question Answering
Perez, E., Lewis, P., Yih, W.-T., Cho, K., & Kiela, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.713 · Cited by 123 · see citing works
Neural Logic MachinesPreprint
Dong, H., Mao, J., Lin, T., Wang, C., Li, L. C., & Zhou, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11694 · Cited by 123 · see citing works
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image RecognitionPreprint
Wang, Y., Huang, R., Song, S., Huang, Z., & Huang, G. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15075 · Cited by 101 · see citing works
Answering Complex Open-domain Questions Through Iterative Query Generation
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1261 · Cited by 96 · see citing works
Lake, B., & Baroni, M. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00350 · Cited by 85 · see citing works
Generative Deep Neural Networks for Dialogue: A Short ReviewPreprint
Serban, I. V., Lowe, R., Charlin, L., & Pineau, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06216 · Cited by 82 · see citing works
Neural Speed Reading via Skim-RNNPreprint
Seo, M., Min, S., Farhadi, A., & Hajishirzi, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02085 · Cited by 69 · see citing works
Self-citationIs a Question Decomposition Unit All We Need?Preprint
Patel, P. H., Mishra, S., Parmar, M., & Baral, C. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.12538 · Cited by 67 · see citing works
Knowledge-Aware Language Model PretrainingPreprint
Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P. N., & Tiwary, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00655 · Cited by 60 · see citing works
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · see citing works
RealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N. A., Choi, Y., & Inui, K. (2022) · arXiv · doi:10.48550/arxiv.2207.13332 · Cited by 52 · see citing works
Self-citationRealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N., Choi, Y., & Inui, K. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2130 · Cited by 52 · see citing works
Self-citationText Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P. E., & Sabharwal, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.99 · Cited by 48 · see citing works
SkipNet: Learning Dynamic Routing in Convolutional NetworksPreprint
Wang, X., Yu, F., Dou, Z.-Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09485 · Cited by 46 · see citing works
Break It Down: A Question Understanding BenchmarkPreprint
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11770 · Cited by 40 · see citing works
The Unreliability of Explanations in Few-Shot In-Context LearningPreprint
Ye, X., & Durrett, G. (2022) · arXiv.org · doi:10.48550/arxiv.2205.03401 · Cited by 37 · see citing works
Answer-based Adversarial Training for Generating Clarification Questions
Rao, S., & Daumé, H. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1013 · Cited by 34 · see citing works
Is a Question Decomposition Unit All We Need?
Patel, P., Mishra, S. R., Parmar, M., & Baral, C. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.302 · Cited by 25 · see citing works
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2Preprint
Betz, G., Richardson, K., & Voigt, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13033 · Cited by 23 · see citing works
Multi-hop Reading Comprehension through Question Decomposition and RescoringPreprint
Min, S., Zhong, V., Zettlemoyer, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02916 · Cited by 14 · see citing works
Self-citationCan You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent NetworksPreprint
Schwarzschild, A., Borgnia, E., Gupta, A. K., Huang, F., Vishkin, U., Goldblum, M., & Goldstein, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04537 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794 · see citing works
Unleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 764 · see citing works
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497 · see citing works
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428 · see citing works
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · arXiv · doi:10.48550/arxiv.2212.10509 · Cited by 349 · see citing works
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.557 · Cited by 349 · see citing works
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344 · see citing works
Query Rewriting in Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.322 · Cited by 391 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264 · see citing works
Self-citationLarge Language Models for Information Retrieval: A Survey
Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Liu, Z., Dou, Z., & Wen, J.-R. (2026) · ACM Transactions on Information Systems · doi:10.1145/3748304 · Cited by 225 · see citing works
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
Query2doc: Query Expansion with Large Language Models
Wang, L., Yang, N., & Wei, F. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.585 · Cited by 214 · see citing works
Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., & Chen, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.620 · Cited by 200 · see citing works
Human-like systematic generalization through a meta-learning neural network
Lake, B. M., & Baroni, M. (2023) · Nature · doi:10.1038/s41586-023-06668-3 · Cited by 194 · see citing works
(2024) · doi:10.1145/3704435 · Cited by 184 · see citing works
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184 · see citing works
Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models
Xi, Y., Liu, W., Lin, J., Cai, X., Zhu, H. C., Zhu, J., Chen, B., Tang, R., Zhang, W., & Yu, Y. H. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688104 · Cited by 246 · see citing works
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145 · see citing works
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Hu, Y., Liu, B., Kasai, J., Wang, Y., Mari Ostendorf, E., Krishna, R., & Smith, N. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01866 · Cited by 151 · see citing works
Self-citationBridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 169 · see citing works
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.153 · Cited by 120 · see citing works
Self-citationLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Barman, D., Guo, Z., & Conlan, O. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100545 · Cited by 103 · see citing works
DB-GPT: Large Language Model Meets Database
Zhou, X.-H., Sun, Z., & Li, G.-L. (2024) · Data Science and Engineering · doi:10.1007/s41019-023-00235-6 · Cited by 135 · see citing works
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.779 · Cited by 103 · see citing works
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Large models for intelligent transportation systems and autonomous vehicles: A survey
Gan, L., Chu, W., Li, G., Tang, X., & Li, K. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102786 · Cited by 79 · see citing works
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 124 · see citing works
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Nadas, M. D., Dioşan, L. S., & Tomescu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3589503 · Cited by 99 · see citing works
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Wang, Y., Li, P., Sun, M., & Liu, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.691 · Cited by 88 · see citing works
Natural Language Reasoning, A Survey
Yu, F., Zhang, H., Tiwari, P., & Wang, B. (2024) · ACM Computing Surveys · doi:10.1145/3664194 · Cited by 78 · see citing works
Fact-Checking Complex Claims with Program-Guided Reasoning
Pan, L., Wu, X., Lu, X., Luu, A. T., Wang, W. Y., Kan, M., & Nakov, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.386 · Cited by 78 · see citing works
LM vs LM: Detecting Factual Errors via Cross Examination
Cohen, R., Hamri, M., Geva, M., & Globerson, A. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.778 · Cited by 60 · see citing works
Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework
Zhao, R., Li, X., Joty, S., Qin, C., & Bing, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.320 · Cited by 59 · see citing works
Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method
Zhang, X., & Gao, W. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.64 · Cited by 90 · see citing works
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2023) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 74 · see citing works
(2023) · doi:10.18653/v1/2023.emnlp-main.971 · Cited by 55 · see citing works
From language to action: a review of large language models as autonomous agents and tool users
Chowa, S. S., Alvi, R., Rahman, S. S., Rahman, M. A., Raiaan, M. A. K., Islam, M. R., Hussain, M., & Azam, S. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-025-11471-9 · Cited by 50 · see citing works
AutoWebGLM: A Large Language Model-based Web Navigating Agent
Lai, H., Liu, X., Iong, I. L., Yao, S., Chen, Y., Shen, P., Yu, H., Zhang, H., Zhang, X., Dong, Y., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671620 · Cited by 69 · see citing works
Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies
Nan, L., Zhao, Y., Zou, W., Ri, N., Tae, J., Zhang, E., Cohan, A., & Radev, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.996 · Cited by 69 · see citing works
ThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02433-3 · Cited by 52 · see citing works
TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
Santu, S. K. K., & Feng, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.946 · Cited by 47 · see citing works
Complementary Explanations for Effective In-Context Learning
Ye, X., Iyer, S. V., Çelikyılmaz, A., Stoyanov, V., Durrett, G., & Pasunuru, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.273 · Cited by 55 · see citing works
Successive Prompting for Decomposing Complex Questions
Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.81 · Cited by 46 · see citing works
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
You, H., Sun, R., Wang, Z., Chen, L., Wang, G., Ayyubi, H., Chang, K.-W., & Chang, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.755 · Cited by 55 · see citing works
Design and evaluation of an Autonomous Cyber Defence agent using DRL and an augmented LLM
Loevenich, J. F., Adler, E., Hürten, T., & Lopes, R. R. F. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111162 · Cited by 52 · see citing works
Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models
Wang, H., & Shu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.416 · Cited by 49 · see citing works
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K.-F. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.806 · Cited by 38 · see citing works
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38 · see citing works
(2023) · doi:10.18653/v1/2023.emnlp-main.364 · Cited by 38 · see citing works
Xu, S., Pang, L., Shen, H., Cheng, X., & Chua, T. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645363 · Cited by 59 · see citing works
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., & Evans, O. (2023) · arXiv · doi:10.48550/arxiv.2309.12288 · Cited by 36 · see citing works
Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chains
Cheng, Y., Chen, J., Huang, Q. Y., Xing, Z., Xu, X., & Lu, Q. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3638247 · Cited by 35 · see citing works
Thanasi-Boçe, M., & Hoxha, J. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12775-z · Cited by 38 · see citing works
Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities EmergencePreprint
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476325 · Cited by 32 · see citing works
Subramonyam, H., Thakkar, D., Ku, A., Dieber, J., & Sinha, A. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713166 · Cited by 38 · see citing works
The argument for near-term human disempowerment through AI
Dung, L. (2024) · AI & Society · doi:10.1007/s00146-024-01930-2 · Cited by 29 · see citing works
Performance analysis of large language models in the domain of legal argument mining
Zubaer, A. A., Granitzer, M., & Mitrović, J. (2023) · Frontiers Artif. Intell. · doi:10.3389/frai.2023.1278796 · Cited by 44 · see citing works
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 37 · see citing works
Visconde: Multi-document QA with GPT-3 and Neural Reranking
Pereira, J., Fidalgo, R., Lotufo, R., & Nogueira, R. F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_44 · Cited by 27 · see citing works
Leveraging Large Language Models for Automatic Smart Contract Generation
Napoli, E. A., Barbàra, F., Gatteschi, V., & Schifanella, C. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00100 · Cited by 24 · see citing works
Exploring Effective Factors for Improving Visual In-Context Learning
Sun, Y.-P., Chen, Q., Wang, J., Wang, J.-D., & Li, Z.-C. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3554410 · Cited by 73 · see citing works
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29 · see citing works
Modular Visual Question Answering via Code Generation
Subramanian, S., Narasimhan, M., Khangaonkar, K., Yang, K., Nagrani, A., Schmid, C., Zeng, A., Darrell, T., & Klein, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.65 · Cited by 28 · see citing works
Shusterman, R., Waters, A. C., O’Neill, S. K., Bangs, M., Luu, P., & Tucker, D. M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01516-2 · Cited by 22 · see citing works
LoCal: Logical and Causal Fact-Checking with LLM-Based Multi-Agents
Ma, J., Hu, L., Li, R., & Fu, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714748 · Cited by 24 · see citing works
MoT: Memory-of-Thought Enables ChatGPT to Self-Improve
Li, X., & Qiu, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.392 · Cited by 36 · see citing works
FIT-RAG: Black-Box RAG with Factual Information and Token Reduction
Mao, Y., Dong, X., Xu, W., Gao, Y., Wei, B., & Zhang, Y. (2024) · ACM Transactions on Information Systems · doi:10.1145/3676957 · Cited by 29 · see citing works
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.14831 · Cited by 152 · see citing works
Boosting Your Context by Dual Similarity Checkup for In-Context Learning Medical Image Segmentation
Gao, J. Y., Lao, Q., Kang, Q., Liu, P., Du, C., Li, K., & Zhang, L. (2024) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2024.3440311 · Cited by 21 · see citing works
Demystifying Chains, Trees, and Graphs of Thoughts
Besta, M., Memedi, F., Zhang, Z., Gerstenberger, R., Piao, G., Blach, N., Nyczyk, P., Copik, M., Kwaśniewski, G., Müller, J., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3598182 · Cited by 34 · see citing works
Prompting Language Models for Linguistic Structure
Blevins, T., Gonen, H., & Zettlemoyer, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.367 · Cited by 25 · see citing works
Russin, J., Pavlick, E., & Frank, M. J. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2510270122 · Cited by 23 · see citing works
SciSciGPT: advancing human–AI collaboration in the science of science
Shao, E., Wang, Y., Qian, Y., Pan, Z., Liu, H., & Wang, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00906-6 · Cited by 26 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
Self-citationProbabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions
Cao, S., Zhang, J., Shi, J., Lv, X., Yao, Z., Tian, Q., Hou, L., & Li, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.835 · Cited by 17 · see citing works
Instruct Me More! Random Prompting for Visual In-Context Learning
Zhang, J., Wang, B.-W., Li, L.-Z., Nakashima, Y., & Nagahara, H. (2023) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00258 · Cited by 42 · see citing works
Promises and Pitfalls: Using Large Language Models to Generate Visualization Items
Cui, Y., Ge, L. W., Ding, Y., Harrison, L., Yang, F., & Kay, M. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456309 · Cited by 17 · see citing works
Huang, J., Wang, M., Cui, Y., Liu, J., Chen, L., Wang, T., Li, H., & Wu, J. (2024) · Applied Sciences · doi:10.3390/app142311014 · Cited by 17 · see citing works
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., & Lewis, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18796 · Cited by 16 · see citing works
Salikutluk, V., Koert, D., & Jäkel, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230081 · Cited by 15 · see citing works
ContextMate: a context-aware smart agent for efficient data analysis
Jadoon, A. K., Yu, C., & Shi, Y.-C. (2024) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-023-00144-7 · Cited by 15 · see citing works
Learning by thinking in natural and artificial minds
Lombrozo, T. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.07.007 · Cited by 14 · see citing works
Structured knowledge-based causal discovery: Agentic streams of thought
Meier, S., Raut, P. N., Mahr, F., Thielen, N., Franke, J. E., & Risch, F. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104202 · Cited by 13 · see citing works
FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge
Feng, S., Balachandran, V., Bai, Y., & Tsvetkov, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.59 · Cited by 21 · see citing works
Lu, X., Pan, L., Liu, Q., Nakov, P., & Kan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.483 · Cited by 13 · see citing works
ChatLLM network: More brains, more intelligence
Hao, R., Hu, L., Qi, W., Wu, Q., Zhang, Y., & Nie, L. (2025) · AI Open · doi:10.1016/j.aiopen.2025.01.001 · Cited by 29 · see citing works
OpenICL: An Open-Source Framework for In-context Learning
Wu, Z., Wang, Y., Ye, J., Wu, Z., Feng, J., Xu, J., & Qiao, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.47 · Cited by 13 · see citing works
Kim, J., Shin, B., Chung, J., & Rhu, M. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408569 · Cited by 11 · see citing works
Toubal, I. E., Avinash, A., Alldrin, N., Dlabal, J., Zhou, W., Luo, E., Stretcu, O., Xiong, H., Lu, C.-T., Zhou, H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01662 · Cited by 12 · see citing works
SiReRAG: Indexing Similar and Related Information for Multihop ReasoningPreprint
Zhang, N., Choubey, P. K., Fabbri, A. R., Bernadett-Shapiro, G., Zhang, R., Mitra, P., Xiong, C., & Wu, C.-S. (2024) · Qeios · doi:10.32388/93o2s3 · Cited by 17 · see citing works
Beamaggr: Beam aggregation reasoning over multi-source knowledge for multi-hop question answering
Chu, Z., Chen, J., Chen, Q., Wang, H., Zhu, K., Du, X., Yu, W., Liu, M., & Qin, B. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.67 · Cited by 11 · see citing works
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10 · see citing works
Zhao, B., Zhang, Y., Xu, Z., Ren, Y., Zhang, X., Luo, R., Feng, Z., & Xia, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761103 · Cited by 12 · see citing works
Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Guan, S., Wang, J., 渡辺, 将., Zhu, B., Lou, J., & Xiong, H. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3793671 · Cited by 28 · see citing works
FinQAPT: Empowering Financial Decisions with End-to-End LLM-driven Question Answering Pipeline
Singh, K., Kaur, S., & Smiley, C. (2024) · ACM International Conference on AI in Finance (ICAIF) · doi:10.1145/3677052.3698682 · Cited by 11 · see citing works
Yuan, Z., Liu, K., Peng, R., Li, S., Leybourne, D. J., Musa, N., Huang, H., & Yang, P. (2024) · IEEE Internet of Things Magazine · doi:10.1109/iotm.001.2400036 · Cited by 11 · see citing works
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Nair, I., Somasundaram, S., Saxena, A., & Goswami, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.972 · Cited by 11 · see citing works
- Authors
- Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Corresponding author
- Press, Ofir
- ORCID
- Smith, Noah
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 5687-5711
- OpenAlex
- W4389520103
- Cited by
- 303
- References
- 61
- Citations by year
- 2026: 97 · 2025: 90 · 2024: 46 · 2023: 69 · 2022: 1
- FWCI
- 53.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2113530)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- compositional reasoning, multi-hop question answering, chain-of-thought prompting, GPT-3, model size scaling, structured prompting, search engine integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review