Self-Consistency Improves Chain of Thought Reasoning in Language Models
Open accessAbstract
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer. Our extensive empirical evaluation shows that self-consistency boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In more general tasks without verification tools, techniques like majority vote, self-consistency, and best-of-n can choose the most promising response from candidate responses.
“In more general tasks without verification tools, we can conduct techniques like majority vote, self-consistency, and best-of-n to choose the most promising one from candidate responses”
From Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? (2025) · cites this paper as Wang et al., 2022
Section 2 (pp. 2–3) describes sampling candidate reasoning paths and choosing the most consistent final answer by majority vote, without a separate verifier. It does not establish best-of-n selection, and it limits the described method to tasks with answers from a fixed set rather than general response-selection tasks. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMaj@N is naturally suited to closed-ended tasks; extending it to open-ended generation typically requires an additional aggregation step.
“It is naturally suited to closed-ended tasks (e.g., multiple choice); extending it to open-ended generation typically requires an additional aggregation step-prompting the model (or a judge) to read the N candidates and synthesize a single answer.”
From From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs (2025) · cites this paper as Wang et al., 2022
Section 2 (pp. 2–3) describes aggregating sampled answers and says the method applies to problems with a fixed answer set. It says extension to open-text generation is possible if a good consistency metric can be defined, but does not propose prompting a model or judge to read the candidates and synthesize one answer. (LLM, conf=0.86)
Experimental — accuracy not yet measuredCoT prompting has been proposed to bolster LLM reasoning capabilities, demonstrating improved performance across various reasoning and NLP tasks.
“For instance, CoT prompting (Wei et al., 2022;Wang et al., 2022) has been proposed to bolster LLM reasoning capabilities, demonstrating improved performance across various reasoning and natural language processing tasks.”
From Large language models as tool makers (2023) · cites this paper as Wang et al., 2022
The Introduction (p. 1) reports that Wei et al. proposed CoT prompting to address limitations in language-model reasoning and that it improved performance on multi-step reasoning tasks. But §3.3 (p. 6) says CoT prompting hurt performance on some common NLP tasks; it was Wang et al.’s self-consistency method that boosted performance there. The claim’s broader attribution of improved NLP-task performance to CoT prompting is therefore not supported. (LLM, conf=0.85)
Experimental — accuracy not yet measuredOne iteration of MAGICORE outperforms Self-Consistency by 3.4% when Self-Consistency uses k=120.
“just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2% and Self-Consistency (Wang et al., 2022) by 3.4%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Wang et al., 2022
Self-Consistency generates k reasoning chains and marginalizes over the chains to obtain answer clusters.
“Self-Consistency (SC; Wang et al., 2022) generates k reasoning chains and marginalizes over the chains to obtain answer clusters”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Wang et al., 2022
The self-consistency baseline, selecting the most common answer from the 15 generated ones, shows no improvement over standard prompting.
“This baseline is listed under 'S.C.” and shows no improvement over standard prompting.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [46]
Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding.
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023), as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Wang et al., 2022
For self-consistency, a framework with 10 distinct reasoning paths and a temperature parameter of 0.7 was implemented.
“For self-consistency, following established literature, we implemented a framework with 10 distinct reasoning paths and set the temperature parameter to 0.723.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [23]
Showing models many of their own T = 1 samples, along with a single sample to evaluate as True/False, can significantly improve their performance.
“Showing models many of their own T = 1 samples, along with a single sample to evaluate as True/False, can significantly improve their performance (this is somewhat reminiscent of self-consistency prompting [Wang et al., 2022]).”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Wang et al., 2022
Encouraging language models to first generate internal thoughts or reasoning chains before responding improves performance on reasoning tasks.
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023)”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Wang et al., 2022
Multi-agent debate's efficacy is no better than self-consistency when considering an equivalent number of responses.
“However, our results reveal that its efficacy is no better than self-consistency (Wang et al., 2022) when considering an equivalent number of responses, highlighting the limitations of such an approach.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Wang et al., 2022
Recent studies have shown that increasing inference-time compute can improve performance in reasoning, mathematics, and code generation via meta-generation strategies such as parallel sampling, tree search, and refinement.
“Recent studies have shown that increasing inference-time compute can improve performance in reasoning, mathematics, and code generation via meta-generation strategies (Welleck et al., 2024) such as parallel sampling (Wang et al., 2022;Aggarwal et al., 2023;Sun et al., 2024), tree search (Yao et al., 2024;Wu et al., 2024a), and refinement (Welleck et al., 2023;Madaan et al., 2023;Snell et al., 2024”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Wang et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 16× · see citing works
“…have found that even when the desired answer is fixed, introducing diversity in the reasoning processes can be highly beneficial; therefore we leverage sampling, as commonly used for open-ended text generation (Radford et al., 2019; Brown et al., 2020; Thoppilan et al., 2022), to achieve this goal.”
BackgroundInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Self-citationA Learning Algorithm for Boltzmann Machines
Ackley, D., Hinton, G. E., & Sejnowski, T. (1985) · Cognitive Sciences · doi:10.1016/b978-0-08-051581-6.50053-2 · Cited by 4,462 · cited 6× · see citing works
“Multiple decoding strategies for language models have been proposed in the literature, e.g., temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes…”
BackgroundMethodInfluentialKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 32× · see citing works
“In an effort to address this shortcoming, Wei et al. (2022) have proposed chain-of-thought prompting , where a language model is prompted to generate a series of short sentences that mimic the reasoning process a person might employ in solving a task.”
BackgroundMethodInfluentialSelf-citationA learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402 · see citing works
Individual differences in reasoning: Implications for the rationality debate?
Stanovich, K., & West, R. (2000) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00003435 · Cited by 2,987 · see citing works
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,805 · cited 4× · see citing works
“For these tasks we used the CommonsenseQA (Talmor et al., 2019) benchmark for commonsense reasoning, StrategyQA (Geva et al.”
MethodInfluentialPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Self-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3× · see citing works
“Prompt with correct chain-of-thought 17.1 Prompt with imperfect chain-of-thought 14.9 + Self-consistency (40 paths) 23.4 Prompt with equations 5.0 + Self-consistency (40 paths) 6.5 PaLM-540B Zero-shot CoT (Kojima et al., 2022) 43.0 + Self-consistency (40 paths) 69.2 Table 8: Self-consistency works under imperfect prom…”
ResultInfluentialHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 3× · see citing works
“Here we perform a study using self-consistency to see if it can help fill in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al., 2020) and RTE…”
BackgroundThe PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 3× · see citing works
“We use two public engines code-davinci-001 and code-davinci-002 from the Codex series (Chen et al., 2021) to aid reproducibility; • LaMDA-137B (Thoppilan et al., 2022) is a dense left-to-right, decoder-only language model with 137-billion parameters, pre-trained on a mixture of web documents,…”
BackgroundMethodAre NLP Models really able to Solve Simple Math Word Problems?
Patel, A., Bhattamishra, S., & Goyal, N. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.07191 · Cited by 1,423 · cited 5×
“For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes significant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv.”
BackgroundMethodInfluentialMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 2× · see citing works
“, 2021) show that model performance could be sensitive to the prompt order, and (2) multiple sets of prompts (Gao et al., 2021).”
BackgroundMethodAdversarial NLI: A New Benchmark for Natural Language UnderstandingPreprint
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1910.14599 · Cited by 1,319 · cited 4× · see citing works
“, 2018), ANLI (Nie et al., 2020) and RTE (Dagan et al.”
BackgroundInfluentialThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · cited 7× · see citing works
“…to sampling strategies and parameters, by varying T in temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), k in top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), and p in nucleus sampling (Holtzman et al., 2020), over PaLM-540B in Figure 4 (left).”
BackgroundMethodInfluentialHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 6× · see citing works
“, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
BackgroundMethodInfluentialMathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., & Hajishirzi, H. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1245 · Cited by 956 · see citing works
Parker, L. (2011) · doi:10.1108/09513571211191815 · Cited by 895
UnifiedQA: Crossing Format Boundaries With a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings · doi:10.48550/arxiv.2005.00700 · Cited by 849 · cited 1×
“The previous SoTA baselines are obtained from: a : DeBERTaV3-large + KEAR (Xu et al., 2021b), b : Chowdhery et al. (2022), c : UnifiedQA-FT (Khashabi et al., 2020).”
MethodThe Third PASCAL Recognizing Textual Entailment Challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · ACL-PASCAL@ACL · doi:10.3115/1654536.1654538 · Cited by 801 · cited 1× · see citing works
“…including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al., 2020) and RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009).”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 14× · see citing works
“…have found that even when the desired answer is fixed, introducing diversity in the reasoning processes can be highly beneficial; therefore we leverage sampling, as commonly used for open-ended text generation (Radford et al., 2019; Brown et al., 2020; Thoppilan et al., 2022), to achieve this goal.”
BackgroundMethodInfluentialSelf-citationSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1× · see citing works
“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
MethodAdversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 2× · see citing works
“…we consider ensembling by: (1) prompt order permutation: we randomly permute the exemplars in the prompt 40 times to mitigate model’s sensitivity to prompt order (Zhao et al., 2021; Lu et al., 2021); and (2) multiple sets of prompts (Gao et al., 2021): we manually write 3 different sets of prompts.”
MethodTowards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · cited 6× · see citing works
“In addition to sampling, re-ranking is also a common approach to improving generation quality in language models (Adiwardana et al., 2020; Shen et al., 2021).”
BackgroundMethodInfluentialSelf-citationTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 19× · see citing works
“Self-consistency is far simpler than prior approaches that either train an additional verifier (Cobbe et al., 2021) or train a re-ranker given additional human annotations to improve generation quality (Thoppilan et al.”
BackgroundMethodInfluentialBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 1× · see citing works
“Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al., 2021; BIG-bench collaboration, 2021, inter alia ).”
BackgroundSelf-citationMAWPS: A Math Word Problem Repository
Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., & Hajishirzi, H. (2016) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n16-1136 · Cited by 551 · cited 2× · see citing works
“We use the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), with the following datasets based on question types: AddSub (Hosseini et al.”
MethodUNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514 · see citing works
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 2× · see citing works
“…we consider ensembling by: (1) prompt order permutation: we randomly permute the exemplars in the prompt 40 times to mitigate model’s sensitivity to prompt order (Zhao et al., 2021; Lu et al., 2021); and (2) multiple sets of prompts (Gao et al., 2021): we manually write 3 different sets of prompts.”
MethodThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 6× · see citing works
“, 2021) for open-domain question-answering with implicit multi-hop reasoning, and the AI2 Reasoning Challenge (ARC) (Clark et al., 2018).”
BackgroundMethodInfluentialControlling Linguistic Style Aspects in Neural Language Generation
Ficler, J., & Goldberg, Y. (2017) · arXiv.org · doi:10.18653/v1/w17-4912 · Cited by 432 · cited 3× · see citing works
“…and Scaling We show self-consistency is robust to sampling strategies and parameters, by varying T in temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), k in top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), and p in nucleus sampling (Holtzman et…”
BackgroundMethodBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · cited 3× · see citing works
“Here we perform a study using self-consistency to see if it can help fill in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al.”
MethodIntuition and Reasoning: A Dual-Process Perspective
Evans, J. S. B. T. (2010) · Psychological Inquiry · doi:10.1080/1047840x.2010.521057 · Cited by 385 · cited 3× · see citing works
“The more that deliberate thinking and analysis is required for a problem (Evans, 2010), the greater the diversity of reasoning paths that can recover the answer.”
BackgroundInfluentiale-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367 · cited 2× · see citing works
“, 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al.”
BackgroundProgram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 9× · see citing works
“For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes significant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv.”
BackgroundMethodInfluentialLearning to Solve Arithmetic Word Problems with Verb Categorization
Hosseini, M. J., Hajishirzi, H., Etzioni, O., & Kushman, N. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1058 · Cited by 299 · cited 2× · see citing works
“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
BackgroundMethodSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · arXiv · doi:10.48550/arxiv.1608.01413 · Cited by 275 · cited 1×
“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
MethodGurevych, I., & Miyao, Y. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-1 · Cited by 265 · cited 6× · see citing works
“, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
InfluentialA Simple, Fast Diverse Decoding Algorithm for Neural GenerationPreprint
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv.org · doi:10.48550/arxiv.1611.08562 · Cited by 261 · cited 1× · see citing works
“Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
BackgroundInjecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.04487 · Cited by 251 · cited 2×
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
Method<i>Did Aristotle Use a Laptop?</i> A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · cited 7× · see citing works
“In addition, we observed that language models can sometimes generate incorrect or nonsensical reasoning paths (e.g., the StrategyQA example in Table 4, the two population numbers are not exactly correct), and further work is needed to better ground models’ rationale generations.”
BackgroundMethodInfluentialScaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 1× · see citing works
“Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al., 2021; BIG-bench collaboration, 2021, inter alia ).”
BackgroundMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · cited 2× · see citing works
“, 2020), and factual knowledge extraction (Elazar et al., 2021).”
BackgroundDiverse Beam Search for Improved Description of Complex Scenes
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12340 · Cited by 219 · cited 1× · see citing works
“Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
BackgroundUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 194 · see citing works
Self-citationLearning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192 · cited 6× · see citing works
“, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
BackgroundMethodInfluentialIs MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
Eikema, B., & Aziz, W. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2005.10283 · Cited by 186 · cited 1×
“…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
BackgroundInjecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 166 · see citing works
Generate & Rank: A Multi-task Framework for Math Word ProblemsPreprint
Shen, J., Yin, Y., Li, L., Shang, L.-F., Jiang, X., Zhang, M., & Liu, Q. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2109.03034 · Cited by 163 · cited 2× · see citing works
“In addition to sampling, re-ranking is also a common approach to improving generation quality in language models (Adiwardana et al., 2020; Shen et al., 2021).”
BackgroundUNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark
Lourie, N., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2103.13009 · Cited by 154 · cited 1×
“, 2021) or using multitask learning to improve commonsense reasoning (Lourie et al., 2021).”
BackgroundLearning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1911.10470 · Cited by 152 · cited 2× · see citing works
“…reasoning paths, such as constructing semantic graphs (Xu et al., 2021a), learning an RNN to retrieve reasoning paths over the Wikipedia graph (Asai et al., 2020), fine-tuning with human annotated reasoning paths on math problems (Cobbe et al., 2021), or training an extractor with…”
BackgroundShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 1× · see citing works
“Nye et al. (2021) improve the logical consistency of samples from a System 1 model by adding a System 2-inspired logical reasoning module.”
MethodDiverse M-Best Solutions in Markov Random Fields
Batra, D., Yadollahpour, P., Guzman-Rivera, A., & Shakhnarovich, G. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_1 · Cited by 139 · cited 1× · see citing works
“Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
BackgroundA Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
Miao, S.-Y., Liang, C.-C., & Su, K.-Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.92 · Cited by 135 · cited 3× · see citing works
“, 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
MethodMutual Information and Diverse Decoding Improve Neural Machine TranslationPreprint
Li, J., & Jurafsky, D. (2016) · arXiv.org · doi:10.48550/arxiv.1601.00372 · Cited by 131 · cited 2× · see citing works
“The reason is that beam search yields a lower diversity in the outputs (Li & Jurafsky, 2016), while in self-consistency the diversity of the reasoning paths is the key to a better performance.”
BackgroundNumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108 · cited 2× · see citing works
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
MethodMake Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1910.03065 · Cited by 108 · cited 2×
“Some prior work has shown that language models can suffer from inconsistency in conversation (Adiwardana et al., 2020), explanation generation (Camburu et al., 2020), and factual knowledge extraction (Elazar et al., 2021).”
BackgroundGenerate & Rank: A Multi-task Framework for Math Word Problems
Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., & Liu, Q. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.195 · Cited by 88 · see citing works
MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem SolversPreprint
Lan, Y.-H., Wang, L., Zhang, Q., Lan, Y.-S., Dai, B., Wang, Y., Zhang, D., & Lim, E.-P. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2109.00799 · Cited by 79 · cited 1× · see citing works
“The supervised baselines are obtained from: a: Relevance and LCA operation classifier (Roy & Roth, 2015), b: GPT-2 finetuned with k-fold CV (Lan et al., 2021), c: Ling et al.”
MethodConsistency of a Recurrent Language Model with Respect to Incomplete Decoding
Welleck, S., Kulikov, I., Kim, J., Pang, R. Y., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2002.02492 · Cited by 79 · cited 1×
“Welleck et al. (2020) use “consistency” to refer to generating an infinite-length sequence in recurrent language models.”
BackgroundThe Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Ye, X., & Durrett, G. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2202 · Cited by 75 · cited 1× · see citing works
“In additional experiments, we show self-consistency can robustly boost performance on NLP tasks where adding a chain-of-thought might hurt performance compared to standard prompting (Ye & Durrett, 2022).”
BackgroundMulti-hop Question Answering via Reasoning ChainsPreprint
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 73 · cited 4× · see citing works
“More recently, the importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths (Chen et al., 2019), or by the introduction of latent variables in a commonsense knowledge graph (Yu et al.”
BackgroundMethodInfluentialGiving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension
Andor, D., He, L., Lee, K., & Pitler, E. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1609 · Cited by 72 · cited 2× · see citing works
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
MethodIs MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
Eikema, B., & Aziz, W. F. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.398 · Cited by 69 · see citing works
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Xu, Y.-C., Zhu, C., Wang, S.-H., Sun, S.-Q., Cheng, H., Liu, X.-D., Gao, J.-F., He, P., Zeng, M., & Huang, X.-D. (2021) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2112.03254 · Cited by 66 · cited 2×
“The previous SoTA baselines are obtained from: a : DeBERTaV3-large + KEAR (Xu et al., 2021b), b : Chowdhery et al. (2022), c : UnifiedQA-FT (Khashabi et al., 2020).”
MethodNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 65 · see citing works
Typical Decoding for Natural Language Generation
Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2022) · arXiv.org · doi:10.48550/arxiv.2202.00666 · Cited by 53 · cited 2×
“…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
BackgroundNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49 · cited 1×
“…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
BackgroundConsistency of a Recurrent Language Model With Respect to Incomplete Decoding
Welleck, S., Kulikov, I., Kim, J., Pang, R. Y., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.448 · Cited by 46 · see citing works
Nye, M., Tessler, M. H., Tenenbaum, J. B., & Lake, B. M. (2021) · arXiv · doi:10.48550/arxiv.2107.02794 · Cited by 40 · cited 1× · see citing works
“Nye et al. (2021) improve the logical consistency of samples from a System 1 model by adding a System 2-inspired logical reasoning module.”
MethodDiversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph ExpertsPreprint
Yu, W., Zhu, C.-G., Qin, L., Zhang, Z., Zhao, T., & Jiang, M. (2022) · Findings · doi:10.48550/arxiv.2203.07285 · Cited by 39 · cited 2× · see citing works
“…importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths (Chen et al., 2019), or by the introduction of latent variables in a commonsense knowledge graph (Yu et al., 2022).”
BackgroundThe Unreliability of Explanations in Few-Shot In-Context LearningPreprint
Ye, X., & Durrett, G. (2022) · arXiv.org · doi:10.48550/arxiv.2205.03401 · Cited by 37 · cited 1× · see citing works
“In additional experiments, we show self-consistency can robustly boost performance on NLP tasks where adding a chain-of-thought might hurt performance compared to standard prompting (Ye & Durrett, 2022).”
BackgroundExploiting Reasoning Chains for Multi-hop Science Question Answering
Xu, W., Deng, Y., Zhang, H., Cai, D., & Lam, W. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2109.02905 · Cited by 33 · cited 2×
“Some previous work has considered task-specific approaches for identifying reasoning paths, such as constructing semantic graphs (Xu et al., 2021a), learning an RNN to retrieve reasoning paths over the Wikipedia graph (Asai et al., 2020), fine-tuning with human annotated reasoning paths on math…”
BackgroundMethodMake Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.382 · Cited by 12 · see citing works
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning.
Piękos, P., Malinowski, M., & Michalewski, H. (2021) · ReIn UW (University of Warsaw) · doi:10.18653/v1/2021.acl-short.49 · Cited by 5 · cited 2× · see citing works
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
Method
Flagged first, then by citations
2 of the 81 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment , 2006.
Cited by
Sorted by citations
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Self-citationLarge language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“A recent technique for improving the accuracy of language models on reasoning tasks is called chain-of-thought prompting (Wei et al., 2022b; Wang et al., 2022).”
MethodSelf-citationChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
“We sample from the models via greedy decoding (though follow-up work shows chain of thought prompting can be improved by taking the majority final answer over many sampled generations (Wang et al., 2022a)).”
MethodSelf-citationA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
Self-citationUnifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · see citing works
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158 · see citing works
Walter, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00448-3 · Cited by 1,113 · see citing works
“Hence, the addendum “Take a deep breath and do it step-by-step” has become a popular addendum to any prompt (Wei et al., 2023).”
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
Self-citationGraph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556 · see citing works
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
Manakul, P., Liusie, A., & Gales, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.557 · Cited by 557 · see citing works
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497 · see citing works
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428 · see citing works
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337 · see citing works
“Reasoning is one of the most actively discussed and debated abilities of LLMs as scaling the model parameter size also increases the implicit knowledge in LLMs (Wei et al., 2022a; Wang et al., 2022; Huang and Chang, 2022).”
BackgroundPlan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., & Lim, E.-P. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.147 · Cited by 344 · see citing works
Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., & Pfister, T. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.507 · Cited by 316 · see citing works
Empowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · see citing works
“Generating and combining many CoTs bridges the gap with human-level performances CoTs can be combined and/or filtered using human or automated feedback (Wang et al., 2022; Cobbe et al., 2021).”
BackgroundMethodInfluentialEncouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
Liang, T., He, Z., Jiao, W., Wang, X., Wang, Y., Wang, R., Yang, Y., Shi, S., & Tu, Z. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.992 · Cited by 274 · see citing works
Self-citationA Survey on Model Compression for Large Language Models
Zhu, X., Li, J., Liu, Y., Ma, C., & Wang, W. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00704 · Cited by 273 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274 · see citing works
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282 · see citing works
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
Self-citationVisual Programming: Compositional visual reasoning without training
Gupta, T., & Kembhavi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01436 · Cited by 230 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2022) · arXiv · doi:10.48550/arxiv.2210.11610 · Cited by 230 · see citing works
Self-citationWhen LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.00352 · Cited by 214 · see citing works
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210 · see citing works
Large Language Models Can Self-Improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.67 · Cited by 230 · see citing works
Self-citationA generalist medical language model for disease diagnosis assistance
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03416-6 · Cited by 183 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · see citing works
Applying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186 · see citing works
PAL: Program-aided Language Models
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., & Neubig, G. (2022) · arXiv · doi:10.48550/arxiv.2211.10435 · Cited by 155 · see citing works
From local to global: a graph rag approach to query-focused summarization
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., & Larson, J. (2024) · arXiv · doi:10.48550/arxiv.2404.16130 · Cited by 159 · see citing works
Large Language Models as Optimizers
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., & Chen, X. (2023) · arXiv · doi:10.48550/arxiv.2309.03409 · Cited by 151 · see citing works
Self-citationAnh-Hoang, D., Tran, V., & Nguyen, L.-M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1622292 · Cited by 146 · see citing works
Reasoning with Language Model is Planning with World Model
Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., & Hu, Z. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.507 · Cited by 168 · see citing works
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“Typical methods include chain-ofthought prompting (Wei et al., 2022b; Wang et al., 2022b) that generates explanations before the final answer and the least-to-most prompting (Zhou et al.”
BackgroundMethodSelf-citationTowards building multilingual language model for medicine
Qiu, P., Wu, C., Zhang, X., Lin, W., Wang, H., Zhang, Y., Wang, Y., & Xie, W. (2024) · Nature Communications · doi:10.1038/s41467-024-52417-z · Cited by 147 · see citing works
Large language models for reticular chemistry
Zheng, Z., Rampal, N., Inizan, T. J., Borgs, C., Chayes, J. T., & Yaghi, O. M. (2025) · Nature Reviews Materials · doi:10.1038/s41578-025-00772-8 · Cited by 134 · see citing works
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning
Tang, X., Zou, A., Zhang, Z., Li, Z., Zhao, Y., Zhang, X., Cohan, A., & Gerstein, M. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.33 · Cited by 133 · see citing works
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · see citing works
“This means that in addition to improving model performance on clean arithmetic reasoning tasks (Wang et al., 2022c), SC also substantially reduces the distractibility of large language models to irrelevant context.”
BackgroundInfluentialSelf-citationEnhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 117 · see citing works
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · see citing works
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.153 · Cited by 120 · see citing works
Capabilities of Gemini Models in Medicine
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.18416 · Cited by 117 · see citing works
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 96 · see citing works
“Prior work also has explored how to take the majority vote across different models [15, 3, 29, 28] while in this work, we use the power of a language model to combine different answers.”
MethodScaling Reasoning Efficiently via Relaxed On-Policy Distillation
Ko, J., Abdali, S., Kim, Y. J., Chen, T., & Cameron, P. (2026) · arXiv.org · doi:10.48550/arxiv.2603.11137 · Cited by 86
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
Distilling Reasoning Capabilities into Smaller Language Models
Shridhar, K., Stolfo, A., & Sachan, M. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.441 · Cited by 79 · see citing works
Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs
Yeh, C.-H., Wang, C., Tong, S., Cheng, T.-Y., Wang, R.-Y., Chu, T.-Z., Zhai, Y., Chen, Y.-B., Gao, S.-H., & Ma, Y. (2026) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i14.38188 · Cited by 79
“• Self-Consistency: Following the Self Consistency approach [50], we prompt MLLMs to generate multiple independent responses for each question.”
Evaluating the ChatGPT family of models for biomedical reasoning and classification
Chen, S., Li, Y., Lu, S., Van, H., Aerts, H. J. W. L., Savova, G. K., & Bitterman, D. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad256 · Cited by 91 · see citing works
Teaching Large Language Models to Self-Debug
Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2304.05128 · Cited by 77 · see citing works
“Prior works demonstrate that decoding multiple samples can significantly improve the performance of large language models (Wang et al., 2023; Shi et al., 2022).”
BackgroundSelf-citationOn Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.244 · Cited by 76 · see citing works
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning
Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022) · arXiv · doi:10.48550/arxiv.2212.08061 · Cited by 76 · see citing works
Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
Qin, Y.-M., Wei, B., Ge, J.-X., Kallidromitis, K., Fu, S., Darrell, T., & Wang, X.-D. (2025) · European Conference on Computer Vision · doi:10.48550/arxiv.2511.19418 · Cited by 75
“Text Space Reasoning Text space reasoning methods, such as Chain-of-Thought [28], has achieved big success in language reasoning [35, 51, 52, 55], solving problems like math, science, and logical reasoning.”
Self-citationCognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“Recent expansion-based techniques (Nogueira et al., 2019; Wang et al., 2023c; Tang et al., 2023a) could allow agents to reason about when certain knowledge would be useful, and store this as metadata to facilitate later recall.”
BackgroundMethodInfluentialLarge language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71 · see citing works
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Making Language Models Better Reasoners with Step-Aware Verifier
Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.291 · Cited by 69 · see citing works
Reasoning with large language models for medical question answering
Lucas, M. M., Yang, J., Pomeroy, J. K., & Yang, C. C. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae131 · Cited by 66 · see citing works
Scaling Large Language Models for Next-Generation Single-Cell AnalysisPreprint
Rizvi, S. A., Levine, D., Patel, A. M., Zhang, S., Wang, E., Perry, C. J., Vrkic, I., Constante, N. M., Fu, Z., He, S., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.04.14.648850 · Cited by 49 · see citing works
“Emerging evidence from language modeling demonstrates that explicit reasoning steps can compensate for causal attention limitations [66, 67, 68].”
Rethinking the Evaluation of Harness Evolution for Agents
Wang, Y., Zhu, H.-S., Hu, Z.-Y., Yuan, Y.-G., Chen, Z.-Y., Senthil, S., Hajishirzi, H., Tsvetkov, Y., Smith, N. A., Dasigi, P., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.12227 · Cited by 62
“Given a fixed agent policy, a task distribution, and a compute budget, each method is specified by what it updates and what feedback it observes.”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 60 · see citing works
“…lightweight finetuning techniques—for adapting LMs to new tasks (Kojima et al., 2022), eliciting systematic reasoning from them (Wei et al., 2022; Wang et al., 2022b), and augmenting them with retrieved sources (Guu et al., 2020; Lazaridou et al., 2022) or with tools (Yao et al., 2022).”
BackgroundKim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58 · see citing works
“As documented in prior work, inconsistencies can occur within a response; they are sometimes referred to as logical fallacies or self-inconsistency in the NLP community [47, 116].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
Step-Audio-R1 Technical Report
Tian, F., Zhang, X., Zhang, Y., Zhang, H., Li, Y., Liu, D., Deng, Y.-Y., Wu, D., Chen, J., Zhao, L., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2511.15848 · Cited by 57
“Underlying these successes is a fundamental principle known as test-time compute scaling: allocating more computational resources during inference—through longer chains of thought, iterative refinement, or search—predictably improves model performance [8–10].”
Large language models can predict the results of social science experiments
Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2026) · Nature · doi:10.1038/s41586-026-10742-x · Cited by 25 · see citing works
“For each condition and outcome, we generated 120 prompts; for each prompt, we sampled five model responses and averaged them to produce one prompt-level simulated response, then used an ensemble approach averaging across prompts to reduce idiosyncratic responding to any single prompt format 33–36 .”
Ding, P., Kuang, J., Ma, D., Cao, X., Xian, Y., Chen, J., & Huang, S. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.118 · Cited by 58 · see citing works
FLEX: Continuous Agent Evolution via Forward Learning from Experience
Cai, Z.-C., Guo, X.-Y., Pei, Y., Feng, J.-T., Su, J.-S., Chen, J.-J., Zhang, Y.-Q., Ma, W.-Y., Wang, M.-X., & Zhou, H. (2025) · arXiv.org · doi:10.48550/arxiv.2511.06449 · Cited by 55
Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023) · arXiv · doi:10.48550/arxiv.2306.13063 · Cited by 52 · see citing works
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
Baek, J., Aji, A., & Saffari, A. (2023) · Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023) · doi:10.18653/v1/2023.matching-1.7 · Cited by 51 · see citing works
“At first, some work proposes to extract the knowledge in the parameters of LLMs themselves via prompting, and then use the extracted knowledge to answer the question (Kojima et al., 2022; Liu et al., 2022b; Wei et al., 2022b; Wang et al., 2022).”
BackgroundShi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54 · see citing works
“Alternatively, an autonomous agent could facilitate clinicians to communicate with EHRs in nat-ural languages, translating clinical questions into machine-interpretable queries, planning a sequence of actions, and ultimately delivering the final responses.”
BackgroundA Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296 · see citing works
Task Contamination: Language Models May Not Be Few-Shot Anymore
Li, C., & Flanigan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29808 · Cited by 47 · see citing works
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Hou, Z.-Y., Li, Y., Tang, J., & Dong, Y. (2026) · arXiv.org · doi:10.48550/arxiv.2607.07508 · Cited by 44
“Second, group-wise methods such as GRPO [Shao et al., 2024, Wang et al., 2022] are mismatched to asynchronous training.”
Clover: Closed-Loop Verifiable Code Generation
Sun, C., Sheng, Y., Padon, O., & Barrett, C. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-65112-0_7 · Cited by 43 · see citing works
TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
Chen, S., Harrison, C., Lee, Y.-C., Yang, A., Ren, Z.-Z., Ratliff, L. J., Duan, J., Fox, D., & Krishna, R. (2026) · arXiv.org · doi:10.48550/arxiv.2602.19313 · Cited by 43
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023) · arXiv · doi:10.48550/arxiv.2309.11495 · Cited by 42 · see citing works
“…performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lan-chantin et al., 2023), as well as updating their initial response through…”
BackgroundLLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
Yang, Q., Mahns, S., Li, S.-D., Gu, A., Wu, J.-B., & Xu, H. (2025) · arXiv.org · doi:10.48550/arxiv.2510.17638 · Cited by 41
LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
Li, H., Dong, Q., Chen, J., Su, H., Zhou, Y., Ai, Q., Ye, Z., & Liu, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.05579 · Cited by 40 · see citing works
Step-GUI Technical ReportPreprint
GELab-Team, & StepFun (2025) · arXiv.org · doi:10.48550/arxiv.2512.15431 · Cited by 41 · see citing works
“Chain-of-thought (CoT) prompting Wei et al. (2022) demonstrates that eliciting intermediate reasoning steps significantly enhances large language models’ performance on complex reasoning tasks.”
Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm
Tong, J.-Q., Mou, Y.-R., Li, H.-C., Li, M., Yang, Y., Zhang, M., Chen, Q.-G., Liang, T., Hu, X., Zheng, Y., et al. (2025) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.2511.04570 · Cited by 40
“This reveals an underexplored direction: test time scaling in video generation reasoning tasks [35].”
Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning
Liu, J.-Q., Xiong, K., Xia, P., Zhou, Y.-Y., Ji, H.-N., Feng, L., Han, S., Ding, M.-Y., & Yao, H.-X. (2025) · arXiv.org · doi:10.48550/arxiv.2511.19900 · Cited by 39
“For example, self-consistency leverages agreement among multiple generated solutions as a learning signal [56], which has enhanced reasoning in tasks like math and code generation, as seen in TTRL [85], MM-UPT [59], and SRT [43].”
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38 · see citing works
Ke, Z.-X., Ming, Y., Xu, A., Chin, R., Nguyen, X.-P., Jwalapuram, P., Wang, J.-Y., Yavuz, S., Xiong, C., & Joty, S. R. (2026) · arXiv.org · doi:10.48550/arxiv.2601.14652 · Cited by 38
“As a result, they are often limited to relatively simple sub-agents such as sub-agents that are based on Chain-of-Thought (CoT) (Wei et al., 2022) and CoT with Self-Consistency (SC) (Wang et al., 2023).”
Studying and improving reasoning in humans and machines
Yax, N., Anlló, H., & Palminteri, S. (2024) · Communications Psychology · doi:10.1038/s44271-024-00091-8 · Cited by 40 · see citing works
Large Language Models Cannot Self-Correct Reasoning Yet
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2310.01798 · Cited by 36 · see citing works
“For reference, we also report the results of self-consistency (Wang et al., 2022), which prompts models to generate multiple responses and performs majority voting to select the final answer.”
MethodSelf-citationGATES: Self-Distillation under Privileged Context with Consensus Gating
Stein, A., Huang, F., & Goldstein, T. (2026) · arXiv.org · doi:10.48550/arxiv.2602.20574 · Cited by 36
“…filtering are well-established reliability mechanisms in this literature; they have been used for answer selection at inference time [Wang et al., 2023], for filtering training data in self-improvement pipelines [Zelikman et al., 2022], and for offline reinforcement learning on…”
Understanding the planning of LLM agents: A survey
Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., Wang, Y., Tang, R., & Chen, E. (2024) · arXiv · doi:10.48550/arxiv.2402.02716 · Cited by 47 · see citing works
Self-citationReasoning Models Struggle to Control their Chains of ThoughtPreprint
Yueh-Han, C., McCarthy, R., Lee, B. W., He, H., Kivlichan, I., Baker, B., Carroll, M., & Korbak, T. (2026) · arXiv.org · doi:10.48550/arxiv.2603.05706 · Cited by 35 · see citing works
Guo, K., Wong, P. K.-Y., Cheng, J. C. P., Chan, C.-F., Leung, P.-H., & Tao, X. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106490 · Cited by 31 · see citing works
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K.-F. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.806 · Cited by 38 · see citing works
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
Liu, X., Wang, X.-Y., Ma, Y.-Z., Zhang, Y.-C., & Xiao, C.-W. (2026) · arXiv.org · doi:10.48550/arxiv.2605.21606 · Cited by 33
Self-citationBeattie, J., Neufeld, S., Yang, D., Chukwuma, C., Gul, A., Desai, N., Jiang, S., & Dohopolski, M. (2024) · Cureus · doi:10.7759/cureus.60044 · Cited by 30 · see citing works
Loya, M., Sinha, D., & Futrell, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.241 · Cited by 33 · see citing works
Maximum Likelihood Reinforcement Learning
Tajwar, F., Zeng, G.-N., Zhou, Y., Song, Y.-D., Arora, D., Jiang, Y.-D., Schneider, J. G., Salakhutdinov, R., Feng, H.-W., & Zanette, A. (2026) · arXiv.org · doi:10.48550/arxiv.2602.02710 · Cited by 32
“In other words, validation accuracy is the same as majority voting accuracy (Wang et al., 2023a) in traditional LLM chain-of-thought reasoning tasks.”
Rath, A. (2026) · arXiv.org · doi:10.48550/arxiv.2601.04170 · Cited by 32
“LLM Consistency : Recent work examines single-agent behavioral variation across prompt perturbations [9] and fine-tuning impacts [10], but does not address temporal drift in interactive, multi-turn scenarios or multi-agent coordination dynamics.”
- Authors
- Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2203.11171
- OpenAlex
- W4221161695
- Cited by
- 829
- References
- 81
- Citations by year
- 2026: 71 · 2025: 231 · 2024: 262 · 2023: 253 · 2022: 11 · 2021: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (publishedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- self-consistency, chain-of-thought prompting, large language models, decoding strategies, arithmetic reasoning, commonsense reasoning, GSM8K, Aqua, StrategyQA, ARC-Challenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
