The DOI matches its DataCite record; 79 of the 81 references it cites were found in a registry or an index.
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Open accessAbstract
Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer. Our extensive empirical evaluation shows that self-consistency boosts the performance of chain-of-thought prompting with a striking margin on a range of popular arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“an agent team can adopt a self-consistency approach where multiple agents independently work on the same problem and then vote or converge on an answer [19].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In more general tasks without verification tools, we can conduct techniques like majority vote, self-consistency, and best-of-n to choose the most promising one from candidate responses”
From Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? (2025) · cites this paper as Wang et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is naturally suited to closed-ended tasks (e.g., multiple choice); extending it to open-ended generation typically requires an additional aggregation step-prompting the model (or a judge) to read the N candidates and synthesize a single answer.”
From From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs (2025) · cites this paper as Wang et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, CoT prompting (Wei et al., 2022;Wang et al., 2022) has been proposed to bolster LLM reasoning capabilities, demonstrating improved performance across various reasoning and natural language processing tasks.”
From Large language models as tool makers (2023) · cites this paper as Wang et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“just one iteration of MAGICORE on Llama-3-8B already outperforms Best-of-k sampling (Lightman et al., 2023) by 3.2% and Self-Consistency (Wang et al., 2022) by 3.4%”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Self-Consistency (SC; Wang et al., 2022) generates k reasoning chains and marginalizes over the chains to obtain answer clusters”
From MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning (2025) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“This baseline is listed under 'S.C.” and shows no improvement over standard prompting.”
From On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks (2024) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023), as well as updating their initial response through self-critique (Press et al., 2022; Madaan et al., 2023).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“For self-consistency, following established literature, we implemented a framework with 10 distinct reasoning paths and set the temperature parameter to 0.723.”
From Enhancing diagnostic capability with multi-agents conversational large language models (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Showing models many of their own T = 1 samples, along with a single sample to evaluate as True/False, can significantly improve their performance (this is somewhat reminiscent of self-consistency prompting [Wang et al., 2022]).”
From Language Models (Mostly) Know What They Know (2022) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding (Wei et al., 2022; Adolphs et al., 2021; Wang et al., 2022; Lanchantin et al., 2023)”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“However, our results reveal that its efficacy is no better than self-consistency (Wang et al., 2022) when considering an equivalent number of responses, highlighting the limitations of such an approach.”
From Large Language Models Cannot Self-Correct Reasoning Yet (2023) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Recent studies have shown that increasing inference-time compute can improve performance in reasoning, mathematics, and code generation via meta-generation strategies (Welleck et al., 2024) such as parallel sampling (Wang et al., 2022;Aggarwal et al., 2023;Sun et al., 2024), tree search (Yao et al., 2024;Wu et al., 2024a), and refinement (Welleck et al., 2023;Madaan et al., 2023;Snell et al., 2024”
From AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (2024) · cites this paper as Wang et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“For example, to improve the performance, we equip DAIL-SQL with self-consistency [53], which achieves a performance of 86 . 6% execution accuracy.”
From Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2024) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 16×
“…have found that even when the desired answer is fixed, introducing diversity in the reasoning processes can be highly beneficial; therefore we leverage sampling, as commonly used for open-ended text generation (Radford et al., 2019; Brown et al., 2020; Thoppilan et al., 2022), to achieve this goal.”
BackgroundInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Self-citationIndividual differences in reasoning: Implications for the rationality debate?
Stanovich, K. E., & West, R. F. (2000) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00003435 · Cited by 4,481
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 32×
“In an effort to address this shortcoming, Wei et al. (2022) have proposed chain-of-thought prompting , where a language model is prompted to generate a series of short sentences that mimic the reasoning process a person might employ in solving a task.”
BackgroundMethodInfluentialSelf-citationA learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Self-citationLarge language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 3×
“Prompt with correct chain-of-thought 17.1 Prompt with imperfect chain-of-thought 14.9 + Self-consistency (40 paths) 23.4 Prompt with equations 5.0 + Self-consistency (40 paths) 6.5 PaLM-540B Zero-shot CoT (Kojima et al., 2022) 43.0 + Self-consistency (40 paths) 69.2 Table 8: Self-consistency works under imperfect prom…”
ResultInfluentialHotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 3×
“Here we perform a study using self-consistency to see if it can help fill in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al., 2020) and RTE…”
BackgroundThe PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 3×
“We use two public engines code-davinci-001 and code-davinci-002 from the Codex series (Chen et al., 2021) to aid reproducibility; • LaMDA-137B (Thoppilan et al., 2022) is a dense left-to-right, decoder-only language model with 137-billion parameters, pre-trained on a mixture of web documents,…”
BackgroundMethodAre NLP Models really able to Solve Simple Math Word Problems?
Patel, A., Bhattamishra, S., & Goyal, N. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.07191 · Cited by 1,423 · cited 5×
“For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes significant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv.”
BackgroundMethodInfluentialMaking Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 2×
“, 2021) show that model performance could be sensitive to the prompt order, and (2) multiple sets of prompts (Gao et al., 2021).”
BackgroundMethodAdversarial NLI: A New Benchmark for Natural Language UnderstandingPreprint
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1910.14599 · Cited by 1,319 · cited 4×
“, 2018), ANLI (Nie et al., 2020) and RTE (Dagan et al.”
BackgroundInfluentialThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · cited 7×
“…to sampling strategies and parameters, by varying T in temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), k in top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), and p in nucleus sampling (Holtzman et al., 2020), over PaLM-540B in Figure 4 (left).”
BackgroundMethodInfluentialHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 6×
“, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
BackgroundMethodInfluentialMathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., & Hajishirzi, H. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1245 · Cited by 956
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1421 · Cited by 919 · cited 4×
“For these tasks we used the CommonsenseQA (Talmor et al., 2019) benchmark for commonsense reasoning, StrategyQA (Geva et al.”
MethodInfluentialParker, L. (2011) · doi:10.1108/09513571211191815 · Cited by 895
UnifiedQA: Crossing Format Boundaries With a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings · doi:10.48550/arxiv.2005.00700 · Cited by 849 · cited 1×
“The previous SoTA baselines are obtained from: a : DeBERTaV3-large + KEAR (Xu et al., 2021b), b : Chowdhery et al. (2022), c : UnifiedQA-FT (Khashabi et al., 2020).”
MethodThe Third PASCAL Recognizing Textual Entailment Challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · ACL-PASCAL@ACL · doi:10.3115/1654536.1654538 · Cited by 801 · cited 1×
“…including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al., 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al., 2020) and RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009).”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 14×
“…have found that even when the desired answer is fixed, introducing diversity in the reasoning processes can be highly beneficial; therefore we leverage sampling, as commonly used for open-ended text generation (Radford et al., 2019; Brown et al., 2020; Thoppilan et al., 2022), to achieve this goal.”
BackgroundMethodInfluentialSelf-citationSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1202 · Cited by 702 · cited 1×
“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
MethodAdversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 655 · cited 3×
“Here we perform a study using self-consistency to see if it can help fill in the gap, over a set of common NLP tasks, including (1) Closed-Book Question Answering: BoolQ (Clark et al., 2019), HotpotQA (Yang et al.”
MethodFantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615 · cited 2×
“…we consider ensembling by: (1) prompt order permutation: we randomly permute the exemplars in the prompt 40 times to mitigate model’s sensitivity to prompt order (Zhao et al., 2021; Lu et al., 2021); and (2) multiple sets of prompts (Gao et al., 2021): we manually write 3 different sets of prompts.”
MethodTowards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · cited 6×
“In addition to sampling, re-ranking is also a common approach to improving generation quality in language models (Adiwardana et al., 2020; Shen et al., 2021).”
BackgroundMethodInfluentialSelf-citationTraining Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · cited 19×
“Self-consistency is far simpler than prior approaches that either train an additional verifier (Cobbe et al., 2021) or train a re-ranker given additional human annotations to improve generation quality (Thoppilan et al.”
BackgroundMethodInfluentialBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 1×
“Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al., 2021; BIG-bench collaboration, 2021, inter alia ).”
BackgroundSelf-citationMAWPS: A Math Word Problem Repository
Koncel-Kedziorski, R., Roy, S., Amini, A., Kushman, N., & Hajishirzi, H. (2016) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n16-1136 · Cited by 551 · cited 2×
“We use the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), with the following datasets based on question types: AddSub (Hosseini et al.”
MethodUNIFIEDQA: Crossing Format Boundaries with a Single QA System
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.171 · Cited by 514
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 2×
“…we consider ensembling by: (1) prompt order permutation: we randomly permute the exemplars in the prompt 40 times to mitigate model’s sensitivity to prompt order (Zhao et al., 2021; Lu et al., 2021); and (2) multiple sets of prompts (Gao et al., 2021): we manually write 3 different sets of prompts.”
MethodThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 6×
“, 2021) for open-domain question-answering with implicit multi-hop reasoning, and the AI2 Reasoning Challenge (ARC) (Clark et al., 2018).”
BackgroundMethodInfluentialIntuition and Reasoning: A Dual-Process Perspective
Evans, J. S. B. T. (2010) · Psychological Inquiry · doi:10.1080/1047840x.2010.521057 · Cited by 385 · cited 3×
“The more that deliberate thinking and analysis is required for a problem (Evans, 2010), the greater the diversity of reasoning paths that can recover the answer.”
BackgroundInfluentiale-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367 · cited 2×
“, 2018), and (2) Natural Language Inference: e-SNLI (Camburu et al., 2018), ANLI (Nie et al.”
BackgroundUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 364
Self-citationControlling Linguistic Style Aspects in Neural Language Generation
Ficler, J., & Goldberg, Y. (2017) · arXiv.org · doi:10.18653/v1/w17-4912 · Cited by 334 · cited 3×
“…and Scaling We show self-consistency is robust to sampling strategies and parameters, by varying T in temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), k in top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), and p in nucleus sampling (Holtzman et…”
BackgroundMethodProgram Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325 · cited 9×
“For larger models that already achieve high accuracy on most tasks (e.g., GPT-3 and PaLM-540B), self-consistency still contributes significant additional gains with +12%-18% absolute accuracy on tasks like AQuA and GSM8K, and +7%-11% on SVAMP and ASDiv.”
BackgroundMethodInfluentialLearning to Solve Arithmetic Word Problems with Verb Categorization
Hosseini, M. J., Hajishirzi, H., Etzioni, O., & Kushman, N. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1058 · Cited by 324 · cited 2×
“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
BackgroundMethodSolving General Arithmetic Word Problems
Roy, S., & Roth, D. (2016) · arXiv · doi:10.48550/arxiv.1608.01413 · Cited by 275 · cited 1×
“For these tasks, we used the Math Word Problem Repository (Koncel-Kedziorski et al., 2016), including AddSub (Hosseini et al., 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
MethodA Simple, Fast Diverse Decoding Algorithm for Neural GenerationPreprint
Li, J., Monroe, W., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.08562 · Cited by 269 · cited 1×
“Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
BackgroundInjecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2004.04487 · Cited by 251 · cited 2×
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
MethodDid Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00370 · Cited by 242 · cited 7×
“In addition, we observed that language models can sometimes generate incorrect or nonsensical reasoning paths (e.g., the StrategyQA example in Table 4, the two population numbers are not exactly correct), and further work is needed to better ground models’ rationale generations.”
BackgroundMethodInfluentialScaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 1×
“Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale (Rae et al., 2021; BIG-bench collaboration, 2021, inter alia ).”
BackgroundMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · cited 2×
“, 2020), and factual knowledge extraction (Elazar et al., 2021).”
BackgroundDiverse Beam Search for Improved Description of Complex Scenes
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12340 · Cited by 219 · cited 1×
“Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
BackgroundInjecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.89 · Cited by 197
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192 · cited 6×
“, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
BackgroundMethodInfluentialIs MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
Eikema, B., & Aziz, W. (2020) · International Conference on Computational Linguistics · doi:10.48550/arxiv.2005.10283 · Cited by 186 · cited 1×
“…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
BackgroundGenerate & Rank: A Multi-task Framework for Math Word ProblemsPreprint
Shen, J., Yin, Y., Li, L., Shang, L.-F., Jiang, X., Zhang, M., & Liu, Q. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2109.03034 · Cited by 163 · cited 2×
“In addition to sampling, re-ranking is also a common approach to improving generation quality in language models (Adiwardana et al., 2020; Shen et al., 2021).”
BackgroundUNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark
Lourie, N., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2103.13009 · Cited by 154 · cited 1×
“, 2021) or using multitask learning to improve commonsense reasoning (Lourie et al., 2021).”
BackgroundLearning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1911.10470 · Cited by 152 · cited 2×
“…reasoning paths, such as constructing semantic graphs (Xu et al., 2021a), learning an RNN to retrieve reasoning paths over the Wikipedia graph (Asai et al., 2020), fine-tuning with human annotated reasoning paths on math problems (Cobbe et al., 2021), or training an extractor with…”
BackgroundShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · cited 1×
“Nye et al. (2021) improve the logical consistency of samples from a System 1 model by adding a System 2-inspired logical reasoning module.”
MethodMutual Information and Diverse Decoding Improve Neural Machine TranslationPreprint
Li, J., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.00372 · Cited by 141 · cited 2×
“The reason is that beam search yields a lower diversity in the outputs (Li & Jurafsky, 2016), while in self-consistency the diversity of the reasoning paths is the key to a better performance.”
BackgroundDiverse M-Best Solutions in Markov Random Fields
Batra, D., Yadollahpour, P., Guzman-Rivera, A., & Shakhnarovich, G. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_1 · Cited by 139 · cited 1×
“Other work has sought to explicitly promote diversity in the decoding process (Batra et al., 2012; Li et al., 2016; Vijayakumar et al., 2018).”
BackgroundA Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
Miao, S.-Y., Liang, C.-C., & Su, K.-Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.92 · Cited by 135 · cited 3×
“, 2014), MultiArith (Roy & Roth, 2015), and ASDiv (Miao et al., 2020).”
MethodNumNet: Machine Reading Comprehension with Numerical Reasoning
Ran, Q., Lin, Y., Li, P., Zhou, J., & Liu, Z. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1251 · Cited by 108 · cited 2×
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
MethodMake Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.1910.03065 · Cited by 108 · cited 2×
“Some prior work has shown that language models can suffer from inconsistency in conversation (Adiwardana et al., 2020), explanation generation (Camburu et al., 2020), and factual knowledge extraction (Elazar et al., 2021).”
BackgroundNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 105
Ackley, D. H., Hinton, G. E., & B, T. J. S. A. (1987) · Elsevier eBooks · doi:10.1016/b978-0-08-051581-6.50053-2 · Cited by 102 · cited 6×
“Multiple decoding strategies for language models have been proposed in the literature, e.g., temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes…”
BackgroundMethodInfluentialMulti-hop Question Answering via Reasoning ChainsPreprint
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 97 · cited 4×
“More recently, the importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths (Chen et al., 2019), or by the introduction of latent variables in a commonsense knowledge graph (Yu et al.”
BackgroundMethodInfluentialGurevych, I., & Miyao, Y. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-1 · Cited by 95 · cited 6×
“, 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al.”
InfluentialGenerate & Rank: A Multi-task Framework for Math Word Problems
Shen, J., Yin, Y., Li, L., Shang, L., Jiang, X., Zhang, M., & Liu, Q. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.195 · Cited by 88
MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem SolversPreprint
Lan, Y.-H., Wang, L., Zhang, Q., Lan, Y.-S., Dai, B., Wang, Y., Zhang, D., & Lim, E.-P. (2021) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2109.00799 · Cited by 79 · cited 1×
“The supervised baselines are obtained from: a: Relevance and LCA operation classifier (Roy & Roth, 2015), b: GPT-2 finetuned with k-fold CV (Lan et al., 2021), c: Ling et al.”
MethodConsistency of a Recurrent Language Model with Respect to Incomplete Decoding
Welleck, S., Kulikov, I., Kim, J., Pang, R. Y., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2002.02492 · Cited by 79 · cited 1×
“Welleck et al. (2020) use “consistency” to refer to generating an infinite-length sequence in recurrent language models.”
BackgroundThe Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Ye, X., & Durrett, G. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2202 · Cited by 75 · cited 1×
“In additional experiments, we show self-consistency can robustly boost performance on NLP tasks where adding a chain-of-thought might hurt performance compared to standard prompting (Ye & Durrett, 2022).”
BackgroundGiving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension
Andor, D., He, L., Lee, K., & Pitler, E. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1609 · Cited by 72 · cited 2×
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
MethodIs MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation
Eikema, B., & Aziz, W. F. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.398 · Cited by 69
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention
Xu, Y.-C., Zhu, C., Wang, S.-H., Sun, S.-Q., Cheng, H., Liu, X.-D., Gao, J.-F., He, P., Zeng, M., & Huang, X.-D. (2021) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2112.03254 · Cited by 66 · cited 2×
“The previous SoTA baselines are obtained from: a : DeBERTaV3-large + KEAR (Xu et al., 2021b), b : Chowdhery et al. (2022), c : UnifiedQA-FT (Khashabi et al., 2020).”
MethodConsistency of a Recurrent Language Model With Respect to Incomplete Decoding
Welleck, S., Kulikov, I., Kim, J., Pang, R. Y., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.448 · Cited by 60
Typical Decoding for Natural Language Generation
Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2022) · arXiv.org · doi:10.48550/arxiv.2202.00666 · Cited by 53 · cited 2×
“…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
BackgroundNatural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49 · cited 1×
“…sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), nucleus sampling (Holtzman et al., 2020), minimum Bayes risk decoding (Eikema & Aziz, 2020; Shi et al., 2022), and typical decoding (Meister et al., 2022).”
BackgroundNye, M., Tessler, M. H., Tenenbaum, J. B., & Lake, B. M. (2021) · arXiv · doi:10.48550/arxiv.2107.02794 · Cited by 40 · cited 1×
“Nye et al. (2021) improve the logical consistency of samples from a System 1 model by adding a System 2-inspired logical reasoning module.”
MethodDiversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph ExpertsPreprint
Yu, W., Zhu, C.-G., Qin, L., Zhang, Z., Zhao, T., & Jiang, M. (2022) · Findings · doi:10.48550/arxiv.2203.07285 · Cited by 39 · cited 2×
“…importance of diversity in the reasoning processes has been noticed, but only leveraged via task-specific training, either through an additional QA model over extracted reasoning paths (Chen et al., 2019), or by the introduction of latent variables in a commonsense knowledge graph (Yu et al., 2022).”
BackgroundThe Unreliability of Explanations in Few-Shot In-Context LearningPreprint
Ye, X., & Durrett, G. (2022) · arXiv.org · doi:10.48550/arxiv.2205.03401 · Cited by 37 · cited 1×
“In additional experiments, we show self-consistency can robustly boost performance on NLP tasks where adding a chain-of-thought might hurt performance compared to standard prompting (Ye & Durrett, 2022).”
BackgroundMake Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.382 · Cited by 36
Exploiting Reasoning Chains for Multi-hop Science Question Answering
Xu, W., Deng, Y., Zhang, H., Cai, D., & Lam, W. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2109.02905 · Cited by 33 · cited 2×
“Some previous work has considered task-specific approaches for identifying reasoning paths, such as constructing semantic graphs (Xu et al., 2021a), learning an RNN to retrieve reasoning paths over the Wikipedia graph (Asai et al., 2020), fine-tuning with human annotated reasoning paths on math…”
BackgroundMethodMeasuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning.
Piękos, P., Malinowski, M., & Michalewski, H. (2021) · ReIn UW (University of Warsaw) · doi:10.18653/v1/2021.acl-short.49 · Cited by 10 · cited 2×
“Previous work has primarily focused on specialized approaches for improving reasoning (Andor et al., 2019; Ran et al., 2019; Geva et al., 2020; Pi˛ekos et al., 2021).”
Method
Research works printed but not linked: 1
- Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment , 2006.
Cited by
Sorted by citations
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citationLarge language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“A recent technique for improving the accuracy of language models on reasoning tasks is called chain-of-thought prompting (Wei et al., 2022b; Wang et al., 2022).”
MethodSelf-citationChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
“We sample from the models via greedy decoding (though follow-up work shows chain of thought prompting can be improved by taking the majority final answer over many sampled generations (Wang et al., 2022a)).”
MethodSelf-citation
Metadata
- Authors
- Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2203.11171
- OpenAlex
- W4221161695
- Cited by
- 829
- References
- 81
- Citations by year
- 2026: 71 · 2025: 231 · 2024: 262 · 2023: 253 · 2022: 11
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (publishedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- self-consistency, chain-of-thought prompting, large language models, decoding strategies, arithmetic reasoning, commonsense reasoning, GSM8K, Aqua, StrategyQA, ARC-Challenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
