The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Open accessAbstract
AbstractThere is a failure mode in large language models that we do not have a good name for, and thatwe therefore tend not to treat seriously enough. It is not hallucination — the model is not assertingsomething false. It is not refusal — the model answers at length. It is the production of responses thatcarry the complete outward form of careful reasoning while the cognitive work that reasoning issupposed to represent has not, in any meaningful sense, occurred. We call this theatrical compliance,and we argue that it is, in practical terms, more dangerous than either of the failure modes thatcurrently dominate alignment research. This paper identifies the phenomenon, characterizes its fiveprincipal forms, explains the asymmetry that makes it particularly costly in high-stakes settings, andoutlines the design requirements for systems intended to resist it. We do not describe such a systemin detail here. Our goal is to establish theatrical compliance as a research problem in its own rightand to argue that addressing it requires instruments operating at a fundamentally different level ofabstraction than task-level prompting frameworks.Keywords: theatrical compliance, large language models, AI reasoning quality, cognitiveprocess evaluation, prompt engineering, metacognitive systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, while recent agent frameworks adopt reasoning heuristics like Chain-of-Thought (CoT) [163,193] and ReAct [136] to simulate deliberative processes, these approaches remain shallow in semantic comprehension.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [163]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, they also report limitations such as sensitivity to surface form variation, degraded performance under increased compositionality, and poor generalization”
From From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs (2026) · cites this paper as Wei et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In most cases, this enhancement can improve model performance39,40.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interestingly, a comparatively simple chain-of-thought (CoT) baseline demonstrates competitive accuracy and efficiency.”
From Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs (2025) · cites this paper as [61]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This agentic loop acts as a powerful test-time scaling mechanism [25, 17, 32], utilizing the reasoning capabilities [31] of LLMs at each turn while grounding them through environment responses.”
From The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management (2025) · cites this paper as [31]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The other approach to provide LLMs with userdefined runtime rails is to use chain-of-thought (CoT) (Wei et al., 2022).”
From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Wei et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The scaling up of network and training data lead to the leap-forward of large language models' capability.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This not only helps to improve the model's transparency, persuasiveness, and reliability, but also facilitates the identification and rectification of potential errors through insightful explanations.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, the Chain of Thought proposed by Jason [38] could trigger LLM to reason and engage in step-by-step thinking, which benefits the tool-using capability.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [38]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These components are fundamental for task decomposition, logical inference, action execution, and iterative improvement, following established patterns in agentic systems literature.”
From Understanding and Optimizing Agentic Workflows via Shapley value (2025) · cites this paper as Wei et al. 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“which has been shown to improve the quality of reasoning and future actions71,72”
From Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024) · cites this paper as [71]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike raw finetuning, this approach does scale with model size [4].”
From Exploring Length Generalization in Large Language Models (2022) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, CoT prompting (Wei et al., 2022;Wang et al., 2022) has been proposed to bolster LLM reasoning capabilities, demonstrating improved performance across various reasoning and natural language processing tasks.”
From Large language models as tool makers (2023) · cites this paper as Wei et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Perhaps the most well-known work of using LLMs for reasoning is Chain-of-Thought (CoT) (Wei et al., 2022), which reveals the ability of LLMs to formulate their own 'thinking procedure' for problem solving.”
From ReAct: Synergizing Reasoning and Acting in Language Models (2022) · cites this paper as Wei et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Agentic systems embed iterative reasoning capabilities using frameworks such as ReAct [136], Chain-of-Thought (CoT) prompting [163], and Tree of Thoughts [164].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [163]
The cited source says what the claim attributes to it.
In the citing paper
“For many of our tasks, it is likely that performance could be improved by more advanced prompting techniques; e.g., Wei et al.’s chain-of-thought approach (26) substantially improved performance on last-letter concatenation, which is similar to our acronym task.”
From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“In previous work (Wei et al., 2022), chain-of-thought (CoT) has been proposed to improve the ability of LLMs to perform reasoning and derive the final answer by introducing a series of intermediate reasoning steps.”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Wei et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“CoT prompting is a powerful method by breaking down complex tasks into a series of intermediate steps [17].”
From Enhancing hepatopathy clinical trial efficiency: a secure, large language model-powered pre-screening pipeline (2025) · cites this paper as [17]
The cited source says what the claim attributes to it.
In the citing paper
“Chain-of-Thought (CoT) prompting elicits explicit reasoning steps from LLMs, improving performance on multi-step tasks (Wei et al., 2022).”
From VeReaFine: Iterative Verification Reasoning Refinement RAG for Hallucination-Resistant on Open-Ended Clinical QA (2025) · cites this paper as Wei et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Conversely, techniques such as Chain-of-Thought (CoT) enable models to generate more extensive outputs detailing their reasoning processes.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [39]
The cited source says what the claim attributes to it.
In the citing paper
“Prompt engineering demonstrated that model behavior could be substantially altered without touching the weights: few-shot examples, role descriptions, chain-of-thought decomposition, and self-consistency traces all changed how the same model performed on the same underlying task [Wei et al., 2022, Wang et al., 2023b, Kojima et al., 2022].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Wei et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“By providing the models with the 'chain of thoughts' [38], such as prompting with 'let us think about it step by step' , the large language models exhibit emergent abilities for reasoning and can arrive at conclusions or judgments according to the evidence or logics.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“Some recent developments [38, 156] involve utilizing chains of thought to prompt reasoning for improved interpretability; however, the opacity of the reasoning process of each step remains a concern, and [157] has questioned the possible unfaithful explanations of chain-of-thought prompting.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“It helps guide the model to output the final answer step by step (Wei et al., 2022;Peng et al., 2023;Kojima et al., 2022).”
From CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023) · cites this paper as Wei et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Wei et al. [4], Nye et al. [15] and Lewkowycz et al. [5] showed that combining prompting (i.e. in-context learning) with scratchpad strategies present a powerful combination.”
From Exploring Length Generalization in Large Language Models (2022) · cites this paper as [4]
The cited source says what the claim attributes to it.
In the citing paper
“Language models can generate reasoning chains to answer complex questions (Wei et al., 2022).”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Wei et al., 2022
In the citing paper
“GPT is known to have difficulty with answering questions that require multiple hops of reasoning [48].”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [48]
In the citing paper
“This generalization gap can be partially bridged by “chain of thought” approaches, whereby a model is prompted to generate intermediate computation steps before arriving at an answer [147].”
From Dissociating language and thought in large language models (2024) · cites this paper as [147]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
Decentralizing Privacy: Using Blockchain to Protect Personal Data
Zyskind, G., Nathan, O., & Pentland, A. '. (2015) · 2015 IEEE Security and Privacy Workshops · doi:10.1109/spw.2015.27 · Cited by 2,617
On Microtargeting Socially Divisive Ads
Ribeiro, F. N., Saha, K., Babaei, M., Henrique, L., Messias, J., Benevenuto, F., Goga, O., Gummadi, K. P., & Redmiles, E. M. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287580 · Cited by 125
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Research works printed but not linked: 85
- Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 . · arXiv
- Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operationbased formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , Minneapolis, Minnesota. Association for Computational Linguistics.
- Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a calculator: Finding operations and arguments with reading comprehension. EMNLP .
- Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language. NAACL .
- Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 . · arXiv
- Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, and Greg Durrett. 2021. Flexible generation of natural language deductions. EMNLP .
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. NeurIPS .
- Jonathon Cai, Richard Shin, and Dawn Song. 2017. Making neural programming architectures generalize via recursion. ICLR .
- Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-SNLI: Natural language inference with natural language explanations. NeurIPS .
- Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. 2022. Can rationalization improve robustness? NAACL .
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 . · arXiv
- Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2019. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. ICLR .
- Ting-Rui Chiang and Yun-Nung Chen. 2019. Semantically-aligned equation generation for solving and reasoning math word problems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2656-2668, Minneapolis, Minnesota. Association for Computational Linguistics.
- Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. IJCAI .
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 . · arXiv
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL .
- Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. 2019. Neural logic machines. ICLR .
- Dheeru Dua, Sameer Singh, and Matt Gardner. 2020. Benefits of intermediate annotations in reading comprehension. ACL .
- Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. TACL .
- Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark. 2022. DREAM: Uncovering mental models behind language models. NAACL .
- Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. ACL .
- Peter Hase and Mohit Bansal. 2022. When can models learn from explanations? a formal framework for understanding the roles of explanation data. ACL . https://arxiv.org/abs/2102.02201 · arXiv
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 . · arXiv
- Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. EMNLP .
- Zhanming Jie, Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving as complex relation extraction. arXiv preprint arXiv:2203.10316 . · arXiv
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 . · arXiv
- Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. NAACL .
- Andrew K. Lampinen, Ishita Dasgupta, Stephanie C.Y. Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane X. Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329 . · arXiv
- Yihuai Lan, Lei Wang, Qiyuan Zhang, Yunshi Lan, Bing Tian Dai, Yan Wang, Dongxiang Zhang, and Ee-Peng Lim. 2021. MWPToolkit: An open-source framework for deep learning-based math word problem solvers. arXiv preprint arXiv:2109.00799 . · arXiv
- Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? NAACL .
- Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. EMNLP .
- Iddo Lev, Bill MacCartney, Christopher Manning, and Roger Levy. 2004. Solving logic puzzles: From robust processing to precise semantics. Proceedings of the 2nd Workshop on Text Meaning and Interpretation .
- Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. ACL .
- Zhengzhong Liang, Steven Bethard, and Mihai Surdeanu. 2021. Explainable multi-hop verbal reasoning through internal monologue. NAACL .
- Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. ACL . https://doi.org/10.18653/v1/P17-1015 · DOI · Paper page
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 . · arXiv
- Bodhisattwa Prasad Majumder, Oana-Maria Camburu, Thomas Lukasiewicz, and Julian McAuley. 2021. Rationale-inspired natural language explanations with commonsense. arXiv preprint arXiv:2106.13876 . · arXiv
- Ana Marasovi´ c, Iz Beltagy, Doug Downey, and Matthew E Peters. 2022. Few-shot self-rationalization with natural language prompts. NAACL Findings .
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In ACL .
- Shen Yun Miao, Chao Chun Liang, and Keh Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. ACL .
- Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 . · arXiv
- Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. WT5?! Training text-to-text models to explain their predictions. arXiv preprint arXiv:2004.14546 . · arXiv
- Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114 . · arXiv
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 . · arXiv
- Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? NAACL .
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. NAACL .
- Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, and Weizhu Chen. 2022. Reasoning like program executors. arXiv preprint arXiv:2201.11473 . · arXiv
- Piotr Pi˛ ekos, Mateusz Malinowski, and Henryk Michalewski. 2021. Measuring and improving BERT's mathematical abilities by predicting the order of reasoning. ACL .
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446 . · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:1-67.
- Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H. Hovy, and Yulia Tsvetkov. 2021. SelfExplain: A self-explaining architecture for neural text classifiers. EMNLP .
- Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! Leveraging language models for commonsense reasoning. ACL .
- Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. NumNet: Machine reading comprehension with numerical reasoning. EMNLP .
- Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021. Measuring attribution in natural language generation models. arXiv preprint arXiv:2112.12870 . · arXiv
- Gabriel Recchia. 2021. Teaching autoregressive language models complex tasks by demonstration. arXiv preprint arXiv:2109.02102 . · arXiv
- Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. ACL .
- Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems .
- Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. EMNLP .
- Subhro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about Quantities in Natural Language. TACL .
- Mohammed Saeed, Naser Ahmadi, Preslav Nakov, and Paolo Papotti. 2021. RuleBERT: Teaching soft rules to pre-trained language models. EMNLP .
- Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. ICLR .
- Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021 .
- Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. NAACL .
- Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020. Leap-ofthought: Teaching pre-trained models to systematically reason over implicit knowledge. NeurIPS .
- Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. CommonsenseQA 2.0: Exposing the limits of ai through gamification. NeurIPS Track on Datasets and Benchmarks .
- Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131 . · arXiv
- Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239 . · arXiv
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022a. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 . · arXiv
- Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022b. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705 . · arXiv
- Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. ICLR .
- Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022b. Emergent abilities of large language models. Transactions on Machine Learning Research .
- Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. NAACL .
- Sarah Wiegreffe and Ana Marasovi´ c. 2021. Teach me to explain: A review of datasets for explainable NLP. NeurIPS .
- Sarah Wiegreffe, Ana Marasovi´ c, and Noah A. Smith. 2021. Measuring association between labels and free-text rationales. EMNLP .
- Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022a. PromptChainer: Chaining large language model prompts through visual programming. CHI Extended Abstracts .
- Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022b. AI chains: Transparent and controllable human-AI interaction by chaining large language model prompts. CHI . https://dl.acm.org/doi/abs/10.1145/3491102.3517582 · DOI
- Yujun Yan, Kevin Swersky, Danai Koutra, Parthasarathy Ranganathan, and Milad Hashemi. 2020. Neural execution engines: Learning to execute subroutines. NeurIPS .
- Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xiang Ren. 2021. Refining language models with compositional explanations. NeurIPS .
- Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot in-context learning. arXiv preprint arXiv:2205.03401 . · arXiv
- Yordan Yordanov, Vid Kocijan, Thomas Lukasiewicz, and Oana-Maria Camburu. 2021. Few-shot out-of-domain transfer learning of natural language explanations. arXiv preprint arXiv:2112.06204 . · arXiv
- Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using 'annotator rationales' to improve machine learning for text categorization. NAACL .
- Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615 . · arXiv
- Eric Zelikman, Yuhuai Wu, and Noah D. Goodman. 2022. STaR: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465 . · arXiv
- Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. ICML .
- Wangchunshu Zhou, Jinyi Hu, Hanlin Zhang, Xiaodan Liang, Maosong Sun, Chenyan Xiong, and Jian Tang. 2020. Towards interpretable natural language understanding with explanations as latent variables. NeurIPS .
Cited by
Sorted by citations
Items 201–300 of 341
CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors
Yang, B., Tian, H., Pian, W., Yu, H., Wang, H., Klein, J., Bissyandé, T. F., & Jin, S. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680328 · Cited by 41
Rodriguez, M. Y., Goldkind, L., Victor, B. G., Hiltz, B., & Perron, B. E. (2024) · Journal of Social Work Education · doi:10.1080/10437797.2024.2340931 · Cited by 42
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
The Crowdless Future? Generative AI and Creative Problem Solving
Boussioux, L., N. Lane, J., Zhang, M., Jacimovic, V., & Lakhani, K. R. (2023) · doi:10.2139/ssrn.4533642 · Cited by 343
Winners and losers of generative AI: Early Evidence of Shifts in Freelancer Demand
Teutloff, O., Einsiedler, J., Kässi, O., Braesemann, F., Mishkin, P., & del Rio-Chanona, R. M. (2025) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2024.106845 · Cited by 39
Zhang, L., Fu, X., Li, Y., & Chen, J. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106244 · Cited by 40
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38
Studying and improving reasoning in humans and machines
Yax, N., Anlló, H., & Palminteri, S. (2024) · Communications Psychology · doi:10.1038/s44271-024-00091-8 · Cited by 40
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37
RareBench: Can LLMs Serve as Rare Diseases Specialists?
Chen, X., Mao, X., Guo, Q., Wang, L., Zhang, S., & Chen, T. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671576 · Cited by 41
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36
PANDALens: Towards AI-Assisted In-Context Writing on OHMD During Travels
Cai, R., Janaka, N., Chen, Y., Wang, L., Zhao, S., & Liu, C. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642320 · Cited by 36
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models
Chia, Y. K., Hong, P., Bing, L., & Poria, S. (2023) · arXiv · doi:10.48550/arxiv.2306.04757 · Cited by 36
“In this subsection, we detail the benchmarks used for evaluating various problem-solving aspects.”
MethodJacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36
The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey
Masterman, T., Besen, S., Sawtell, M., & Chao, A. (2024) · arXiv · doi:10.48550/arxiv.2404.11584 · Cited by 35
Qian, X., Zhang, Y., Zhao, Y., Zhou, B., Sui, X., Zhang, L., & Song, K. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.394 · Cited by 35
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K.-F. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.806 · Cited by 38
POSIX: A Prompt Sensitivity Index For Large Language Models
Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., & Chakraborty, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.852 · Cited by 34
Prompted LLMs as Chatbot Modules for Long Open-domain Conversation
Lee, G., Hartmann, V., Park, J., Papailiopoulos, D., & Lee, K. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.277 · Cited by 34
Beattie, J., Neufeld, S., Yang, D., Chukwuma, C., Gul, A., Desai, N., Jiang, S., & Dohopolski, M. (2024) · Cureus · doi:10.7759/cureus.60044 · Cited by 30
“The screening process, inherently reliant on interpreting extensive unstructured text, finds a promising solution in NLP and LLMs due to their advanced reasoning and semantic understanding capabilities [9].”
Loya, M., Sinha, D., & Futrell, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.241 · Cited by 33
Automatic Construction Accident Report Analysis Using Large Language Models (LLMs)
Ahmadi, E., Muley, S., & Wang, C. (2025) · Journal of Intelligent Construction · doi:10.26599/jic.2024.9180039 · Cited by 35
Use of artificial intelligence for liver diseases: A survey from the EASL congress 2024
Žigutytė, L., Sorz-Nechay, T., Clusmann, J., & Kather, J. N. (2024) · JHEP Reports · doi:10.1016/j.jhepr.2024.101209 · Cited by 33
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32
Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning
Tang, R., Kong, D., Huang, L., & Xue, H. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.284 · Cited by 31
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
Yan, W., Tian, Y., Li, Y., Chen, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.337 · Cited by 31
SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I., & Suenderhauf, N. (2023) · arXiv · doi:10.48550/arxiv.2307.06135 · Cited by 29
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022) · arXiv · doi:10.48550/arxiv.2208.07852 · Cited by 204
RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models
Van Veen, D., Van Uden, C., Attias, M., Pareek, A., Bluethgen, C., Polacin, M., Chiu, W., Delbrouck, J.-B., Zambrano Chaves, J., Langlotz, C., et al. (2023) · The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks · doi:10.18653/v1/2023.bionlp-1.42 · Cited by 25
Large language models as tool makers
Cai, T., Wang, X., Ma, T., Chen, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2305.17126 · Cited by 28
Self-citationEvaluating Cognitive Maps and Planning in Large Language Models with CogEval
Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3056 · Cited by 27
Repository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27
“Examples of this behavior include GPT-3 (Brown et al., 2020) doing better in tasks it has never seen during training and improved reasoning capabilities with few-shot (Wei et al., 2022) and zero-shot (Kojima et al., 2022) prompts that encourage a chain of thoughts.”
BackgroundChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models
Chen, Z., Zhou, K., Zhang, B., Gong, Z., Zhao, X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.985 · Cited by 30
Reliable Natural Language Understanding with Large Language Models and Answer Set Programming
Rajasekharan, A., Zeng, Y., Padalkar, P., & Gupta, G. (2023) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.385.27 · Cited by 26
Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection
Fatouros, G., Metaxas, K., Soldatos, J., & Kyriazis, D. (2024) · doi:10.2139/ssrn.4693849 · Cited by 25
scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis
Liu, T., Chen, T., Zheng, W., Luo, X., Chen, Y., & Zhao, H. (2023) · doi:10.1101/2023.12.07.569910 · Cited by 30
Workum, J. D., van de Sande, D., Gommers, D., & van Genderen, M. E. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1504805 · Cited by 25
Leveraging Large Language Models to Generate Answer Set Programs
Ishay, A., Yang, Z., & Lee, J. (2023) · Proceedings of the Twentieth International Conference on Principles of Knowledge Representation and Reasoning · doi:10.24963/kr.2023/37 · Cited by 25
Generating Executable Action Plans with Environmentally-Aware Language Models
Gramopadhye, M., & Szafir, D. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341989 · Cited by 26
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Sarch, G., Wu, Y., Tarr, M., & Fragkiadaki, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.226 · Cited by 25
Testolin, A. (2024) · Applied Sciences · doi:10.3390/app14020744 · Cited by 24
Lampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32
Inverse Scaling Can Become U-Shaped
Wei, J., Kim, N., Tay, Y., & Le, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.963 · Cited by 25
Self-citationDoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2023) · arXiv · doi:10.48550/arxiv.2309.03883 · Cited by 29
GraphWiz: An Instruction-Following Language Model for Graph Computational Problems
Chen, N., Li, Y., Tang, J., & Li, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3672010 · Cited by 22
ZeroSCROLLS: A zero-shot benchmark for long text understanding
Shaham, U., Ivgi, M., Efrat, A., Berant, J., & Levy, O. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.536 · Cited by 23
Quality assessment in the era of large models: A survey
Zhang, Z., Zhou, Y., Li, C., Zhao, B., Liu, X., & Zhai, G. (2025) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3722559 · Cited by 23
Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation
Yuan, X., Wang, T., Wang, Y.-H., Fine, E., Abdelghani, R., Sauzéon, H., & Oudeyer, P.-Y. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.820 · Cited by 21
Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
Li, M., Zhao, S., Wang, Q., Wang, K., Zhou, Y., Srivastava, S., Gokmen, C., Lee, T., Li, L., Zhang, R., et al. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3188 · Cited by 21
Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models
Kodandaram, S. R., Uckun, U., Bi, X., Ramakrishnan, IV, & Ashok, V. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675605 · Cited by 24
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
Roy, S., Khatua, A., Ghoochani, F., Hadler, U., Nejdl, W., & Ganguly, N. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657882 · Cited by 17
Lv, S., Peng, L., Wu, W., Yao, Y., Jiao, S., & Hu, W. (2025) · Materials Genome Engineering Advances · doi:10.1002/mgea.70013 · Cited by 18
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 24
Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions
Cao, S., Zhang, J., Shi, J., Lv, X., Yao, Z., Tian, Q., Hou, L., & Li, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.835 · Cited by 17
ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models
Xu, B., Peng, Z., Lei, B., Mukherjee, S., Liu, Y., & Xu, D. (2023) · arXiv · doi:10.48550/arxiv.2305.18323 · Cited by 17
Small, W. R., Malhotra, K., Major, V. J., Wiesenfeld, B., Lewis, M., Grover, H., Tang, H., Banerjee, A., Jabbour, M. J., Aphinyanaphongs, Y., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000394 · Cited by 17
Wang, K., Duan, F., Wang, S., Li, P., Xian, Y., Yin, C., Rong, W., & Xiong, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.13259 · Cited by 16
Kian, M., Zong, M., Fischer, K., Singh, A., Velentza, A.-M., O'Connell, A., Ignatova, L., Sang, P., Upadhyay, S., Gupta, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.17937 · Cited by 15
Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Thalken, R., Stiglitz, E., Mimno, D., & Wilkens, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.575 · Cited by 13
Masikisiki, B., Marivate, V., & Hlophe, Y. (2023) · Proceedings of the 4th African Human Computer Interaction Conference · doi:10.1145/3628096.3628747 · Cited by 13
ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory
Hu, C., Fu, J., Du, C., Luo, S., Zhao, J., & Zhao, H. (2023) · arXiv · doi:10.48550/arxiv.2306.03901 · Cited by 13
Automatic biomarker discovery and enrichment with BRAD
Pickard, J., Prakash, R., Choi, M. A., Oliven, N., Stansbury, C., Cwycyshyn, J., Galioto, N., Gorodetsky, A., Velasquez, A., & Rajapakse, I. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf159 · Cited by 14
Smit, R., Smits, T., & Merrill, S. (2024) · Memory Studies Review · doi:10.1163/29498902-202400015 · Cited by 12
LLexus: an AI agent system for incident management
Las-Casas, P., Kumbhare, A. G., Fonseca, R., & Agarwal, S. (2024) · ACM SIGOPS Operating Systems Review · doi:10.1145/3689051.3689056 · Cited by 14
Yager, K. G. (2024) · Digital Discovery · doi:10.1039/d4dd00178h · Cited by 12
How Many Demonstrations Do You Need for In-context Learning?
Chen, J., Chen, L., Zhu, C., & Zhou, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.745 · Cited by 12
Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning
Li, R., & Du, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.452 · Cited by 12
An LLM Compiler for Parallel Function Calling
Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., & Gholami, A. (2023) · arXiv · doi:10.48550/arxiv.2312.04511 · Cited by 12
“…reasoning capability of Large Language Models (LLMs) have expanded the applicability of LLMs beyond content generation to solving complex problems [51, 23, 57, 5, 50, 63, 8, 55, 12]; and recent works have shown how this reasoning capability could be helpful in improving accuracy for solving…”
BackgroundAssessing citation integrity in biomedical publications: corpus annotation and NLP models
Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae420 · Cited by 12
“Prompting the model to reason about the steps to arrive at a conclusion is shown to further improve performance for complex reasoning tasks (Wei et al. 2022).”
BackgroundJustice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.02736 · Cited by 11
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2022) · arXiv · doi:10.48550/arxiv.2209.14958 · Cited by 10
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10
UISGPT: Automated Mobile UI Design Smell Detection with Large Language Models
Yang, B., & Li, S. (2024) · Electronics · doi:10.3390/electronics13163127 · Cited by 10
Novel AI applications in systematic review: gPT-4 assisted data extraction, analysis, review of bias
Kim, J. K., Chua, M. E., Li, T. G., Rickard, M., & Lorenzo, A. J. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-113066 · Cited by 11
Enhancing LLM Reasoning Capabilities Through Brokered Multi-Expert Reflection
Sheokand, T., Jain, G., Bahga, A., & Madisetti, V. K. (2025) · IEEE Access · doi:10.1109/access.2025.3561693 · Cited by 7
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
ChatLogic: Integrating Logic Programming with Large Language Models for Multi-Step Reasoning
Wang, Z., Liu, J., Bao, Q., Rong, H., & Zhang, J. (2024) · 2024 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650138 · Cited by 8
“However, the most impactful method is the Chain of Thought (CoT) [12], which reveals the intermediate reasoning steps used by these models in problem-solving, allowing for continual self-correction of logical thinking, thereby greatly enhancing the rationality of their reasoning capabilities.”
MethodA knowledge-driven framework for Robotic Odor Source Localization using large language models
Mahmud, K. R., Wang, L., Hassan, S., & Zhang, Z. (2025) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2025.104915 · Cited by 10
DAWN: Designing Distributed Agents in a Worldwide Network
Aminiranjbar, Z., Tang, J., Wang, Q., Pant, S., & Viswanathan, M. (2024) · arXiv · doi:10.48550/arxiv.2410.22339 · Cited by 8
Dutta Roy, T. P. (2025) · Cureus · doi:10.7759/cureus.80494 · Cited by 8
Gui, X., Lv, H., Wang, X., Lv, L., Xiao, Y., & Wang, L. (2025) · BioData Mining · doi:10.1186/s13040-025-00458-5 · Cited by 10
Self-citationGPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Hsu, T.-Y., Huang, C.-Y., Rossi, R., Kim, S., Giles, C., & Huang, T.-H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.363 · Cited by 8
Jones, C. R., & Bergen, B. K. (2024) · arXiv · doi:10.48550/arxiv.2412.17128 · Cited by 7
GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.19457 · Cited by 7
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1
Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Chen, X., Sun, Z., Wenjin, G., Zhang, M., Chen, Y., Sun, Y., Su, H., Pan, Y., Klakow, D., Li, W., et al. (2025) · Findings of the Association for Computational Linguistics: ACL 2025 · doi:10.18653/v1/2025.findings-acl.782 · Cited by 7
Sharma, M., Tong, M., Mu, J., Wei, J., Kruthoff, J., Goodfriend, S., Ong, E., Peng, A., Agarwal, R., Anil, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.18837 · Cited by 7
Self-citationFreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. (2023) · arXiv · Cited by 7
Self-citationAlario-Hoyos, C., Kemcha, R., Kloos, C. D., Callejo, P., Estévez-Ayres, I., Santín-Cristóbal, D., Cruz-Argudo, F., & López-Sánchez, J. L. (2024) · 2024 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) · doi:10.1109/tale62452.2024.10834365 · Cited by 8
Anthropocentric Bias in Language Model Evaluation
Millière, R., & Rathkopf, C. (2026) · Computational Linguistics · doi:10.1162/coli.a.582 · Cited by 6
“The simplest example involves prompting a language model to generate a “chain of thought” (CoT) before answering (Wei et al. 2023).”
Guiding Clinical Reasoning with Large Language Models via Knowledge Seeds
Wu, J., Wu, X., & Yang, J. (2024) · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/829 · Cited by 7
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.911 · Cited by 9
L’italiano delle intelligenze artificiali generative
Cicero, F. (2023) · Italiano LinguaDue · doi:10.54103/2037-3597/21990 · Cited by 5
Vakharia, P., Kufeldt, A., Meyers, M., Lane, I., & Gilpin, L. H. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-71170-1_23 · Cited by 5
“Instead of training from scratch, chain of thought (CoT) prompting leverages large pre-trained models by instructing the model to describe the “chain of thought” it utilized to come to its answer during generation [24].”
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning
Chen, W., Huang, Z., Xie, L., Lin, B., Li, H., Lu, L., Tian, X., Cai, D., Zhang, Y., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.01658 · Cited by 5
ChatGPT and Deepseek: Can They Predict the Stock Market and Macroeconomy?
Chen, J., Tang, G., Zhou, G., & Zhu, W. (2025) · arXiv · doi:10.48550/arxiv.2502.10008 · Cited by 6
Metadata
- Authors
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- OpenAlex
- W4221143046
- Cited by
- 4,263
- References
- 5
- Citations by year
- 2026: 212 · 2025: 1547 · 2024: 1537 · 2023: 900 · 2022: 66
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, prompt engineering, AI alignment, failure modes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
