CoChat
Sign Up
PreprintCited by 254Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 57 of the 59 references it cites were found in a registry or an index, and 2 are not checked yet.

Large Language Models Can Self-Improve

Open access
Huang, Jiaxin, Gu, Shixiang Shane, Hou, Le, Wu, Yuexin, Wang, Xuezhi, Yu, Hongkun, Han, Jiawei·2022·arXiv
DOI 10.48550/arxiv.2210.11610arXiv 2210.11610
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.11610

Cites

57 of 59 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of our method. With Chain-of-Thought (CoT) examples as demonstration (Wei et al., 2022c), the language model generates multiple CoT reasoning paths and answers (temperature T > 0 ) for each question. The most consistent answer is selected by majority voting (Wang et al., 2022c). The CoT reasoning paths that lead to the answer with the highest confidence are augmented by mixed formats, and are fed back to the model as the final training samples.
Figure 1 Overview of our method. With Chain-of-Thought (CoT) examples as demonstration (Wei et al., 2022c), the language model generates multiple CoT reasoning paths and answers (temperature T > 0 ) for each question. The most consistent answer is selected by majority voting (Wang et al., 2022c). The CoT reasoning paths that lead to the answer with the highest confidence are augmented by mixed formats, and are fed back to the model as the final training samples.CC-BY · From the PDF
Figure 2 The relation of accuracy and confidence of the majority-voted answer after multiple path decoding on GSM8K training-set questions. A recent study (Kadavath et al., 2022) shows that language models are not perfectly-calibrated though their calibration increases with model size, and models with more than 10B parameters are reasonably calibrated on some few-shot tasks. This aligns well with our study and serve as the basis of this self-improving method.
Figure 2 The relation of accuracy and confidence of the majority-voted answer after multiple path decoding on GSM8K training-set questions. A recent study (Kadavath et al., 2022) shows that language models are not perfectly-calibrated though their calibration increases with model size, and models with more than 10B parameters are reasonably calibrated on some few-shot tasks. This aligns well with our study and serve as the basis of this self-improving method.CC-BY · From the PDF
Figure 3 Accuracy results on GSM8K test set using 540B model with multi-path sampling and selfconsistency (Wang et al., 2022c). 'Step-by-Step' is the baseline performance of Kojima et al. (2022) plus selfconsistency (Wang et al., 2022c), while our 'Few-Shot w/ Step-by-Step' uses exemplers self-generated from Step-by-Step (greedy decoding) for few-shot prompting the LLM.
Figure 3 Accuracy results on GSM8K test set using 540B model with multi-path sampling and selfconsistency (Wang et al., 2022c). 'Step-by-Step' is the baseline performance of Kojima et al. (2022) plus selfconsistency (Wang et al., 2022c), while our 'Few-Shot w/ Step-by-Step' uses exemplers self-generated from Step-by-Step (greedy decoding) for few-shot prompting the LLM.CC-BY · From the PDF
Figure 4 Accuracy results of LMSI on GSM8K and DROP test set when different sampling temperatures are applied for Self-Consistency.
Figure 4 Accuracy results of LMSI on GSM8K and DROP test set when different sampling temperatures are applied for Self-Consistency.CC-BY · From the PDF
Figure 5 Accuracy results with or without LMSI on GSM8K test set using different numbers of sampled reasoning path for Self-Consistency.
Figure 5 Accuracy results with or without LMSI on GSM8K test set using different numbers of sampled reasoning path for Self-Consistency.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 7

  1. Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. 2021. Semi-supervised semantic segmenta- tion with cross pseudo supervision. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
  2. Anoop Korattikara Balan, Vivek Rathod, Kevin P Mur- phy, and Max Welling. 2015. Bayesian dark knowl- edge. Advances in neural information processing systems, 28.
  3. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantitative reasoning problems with lan- guage models. ArXiv, abs/2206.14858. · arXiv
  4. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022b. On the advance of making language models better reasoners. ArXiv, abs/2206.02336. · arXiv
  5. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017b. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. In ACL.
  6. Timo Schick and Hinrich Schütze. 2020a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Conference of the Eu- ropean Chapter of the Association for Computational Linguistics.
  7. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022b. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682. · arXiv

Cited by

Sorted by citations

See all 254

Related

See all

Metadata

Authors
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., & Han, J.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
1051-1068
DOI
10.48550/arxiv.2210.11610
arXiv
2210.11610
OpenAlex
W4389523706
Cited by
230
References
59
Citations by year
2026: 36 · 2025: 77 · 2024: 63 · 2023: 53 · 2022: 1
FWCI
40.57
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
large language models, human development, self-supervised fine-tuning, unlabeled data, chain-of-thought prompting, self-consistency, reasoning skills, low-resource languages, Arecaceae, GSM8K, OpenBookQA

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review