CoChat
Sign Up
PreprintCited by 1,350Topic ModelingNatural Language Processing TechniquesSpeech Recognition and Synthesis

The Curious Case of Neural Text Degeneration

Open access
Holtzman, Ari, Buys, Jan, Du, Li, Forbes, Maxwell, Choi, Yejin·2019·arXiv
DOI 10.48550/arxiv.1904.09751arXiv 1904.09751
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.

AI snapshot

Written from the paper’s full text.

TL;DR

Maximization-based decoding such as beam search produces degenerate, repetitive text, while pure sampling from the full distribution produces incoherent text. The paper proposes Nucleus Sampling, which truncates the unreliable tail by sampling from the smallest top-p token set whose cumulative probability exceeds p, and reports it as the best overall decoding strategy for open-ended generation.

Key findings

  • Beam search and greedy decoding lead to degeneration: in Table 1, Beam b=16 has perplexity 1.48 versus human 12.38, Self-BLEU 0.44 versus human 0.31, and repetition 28.94%; Greedy has repetition 73.66%.
  • Pure sampling is incoherent and too unlikely: Table 1 reports Pure Sampling perplexity 22.73 versus human 12.38, with HUSE 0.67.
  • Nucleus Sampling best matches human distributional statistics and wins human+statistical evaluation: Table 1 reports Nucleus p=0.95 perplexity 13.13, Self-BLEU 0.32, Zipf coefficient 0.95, repetition 0.36%, and HUSE 0.97, the highest HUSE score.
  • Top-k sampling can match some statistics only at high k but suffers incoherence or repetition: Top-k=640 has perplexity 13.82 and HUSE 0.94, while Top-k=40, t=0.7 has repetition 8.86% and HUSE 0.08.

Methods

  • Uses GPT-2 Large (762M parameters) trained on WebText, a 40GB web text collection; generates 5,000 text passages conditioned on the initial paragraph (1–40 tokens) of held-out WebText documents, ending at an end-of-document token or maximum length of 200 tokens.
  • Compares decoding strategies including greedy, beam search, stochastic beam, pure sampling, temperature sampling, top-k sampling, and Nucleus Sampling using perplexity, Zipf coefficient, Self-BLEU, repetition, and HUSE.
  • HUSE combines human and statistical evaluation: 200 generations per decoding algorithm are annotated for typicality by 20 annotators each, totaling 4,000 annotations per scheme; a KNN classifier with k=13 is used, and for top-k and Nucleus Sampling the truncated distribution is interpolated with mass 0.1.

Limitations

  • The paper states that future work should dynamically characterize the model’s region of confidence and include a more semantic utility function to guide decoding.
  • The paper states that initial HUSE application gave top-k and Nucleus Sampling near-zero scores due to truncation, so HUSE was computed with an interpolation proxy rather than the truncated distribution directly.
  • Evident from the text: experiments use only GPT-2 Large on English WebText open-ended generation, with human evaluation limited to 200 generations per decoding scheme; repetition is measured as a phrase of minimum length 2 repeating at least three times at the end of the first 200 tokens, and the paper notes beam search repetition decreases partly because average generation length shortens as beam width increases.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1904.09751

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

11 of 29 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Similar issues of repetition and genericness have been reported when using large beam sizes and with exact inference.

    “Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Koehn & Knowles, 2017 source

      Section 3.6 (Beam Search; page not provided) reports that translation quality declines with large beam sizes despite better model scores. It identifies shorter translations as the main cause, but does not report repetition or genericness as the problems observed with large beams. The exact-inference clause is attributed to a different citation and is not judged here. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Stahlberg & Byrne, 2019 source

      The passages available from our copy are bibliographic records without paper sections or page text. Searches for exact inference, repetition, genericness, and beam-search errors returned no passage addressing the reported issues, so the attribution cannot be verified from our copy. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Premisecitation

    Several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text.

    “However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018;Holtzman et al., 2018).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Fan et al., 2018 source
    • cites Holtzman et al., 2018 source
  • Premisecitation

    Finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable.

    “Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;Shen et al., 2017;Wiseman et al., 2017).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chen et al., 2018 source

      Section 4 (page unavailable) states that the best-string problem for RNNs is undecidable; Section 7 adds that the polynomial-length version for consistent RNNs is NP-complete. This supports the claim for the RNNs studied, but the paper does not establish the same result for Transformers. (LLM, conf=0.89)

      Experimental — accuracy not yet measured
  • Findingnone given

    Generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.

    “Yet, generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    The unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.

    “This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 14
  • May combine two claims. 25

  • Classified as probably not a substantial claim.

    Premisecitation

    Recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity.

    “A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2019;Holtzman et al., 2018;Fan et al., 2018).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Kulikov et al., 2019 source

      In “Diversity: distinct-n” (page unavailable), the paper reports that greedy search produces less diverse responses than the other strategies; “2.2 Inference (generation)” also describes greedy search as choosing the most likely token at each step. The copy does not establish that these outputs have high grammaticality, so that part of the claim is unsupported. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Holtzman et al., 2018 source

      The Background (§2; page unavailable) says RNN language models often assign higher probability to repetitive, overly generic sentences, supporting the low-diversity disadvantage of probability-based generation. The Qualitative Analysis (§5.2; page unavailable) likewise describes generic output from a language-model baseline. Our copy does not establish that such output has high grammaticality. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Fan et al., 2018 source

  • May combine two claims.

    Premisecitation

    Lowering the temperature improves generation quality but comes at the cost of decreasing diversity.

    “However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018;Hashimoto et al., 2019).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Caccia et al., 2018 source
    • cites Hashimoto et al., 2019 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GAN-generated text fails to outperform generations from language models when quality and diversity are considered jointly.

    “but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018;Tevet et al., 2019;Semeniuta et al., 2018).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Caccia et al., 2018 source
    • cites Tevet et al., 2019 source

      The available indexed passages contain bibliographic records and an abstract, but no passage from the paper’s full text that can establish the claimed joint assessment of quality and diversity. Searches for that assessment and for evaluation terms returned no body-text passage, so the claim cannot be verified under the full-text-only evidence rule. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Semeniuta et al., 2018 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Transformer architecture has been the most successful in the extremely large-scale training setups in recent literature.

    “the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018;2019).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Vaswani et al., 2017 source
    • cites Radford et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [2019]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingnone given

    Pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality.

    “On the other hand, pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Generations from temperature sampling have more repetition unless very high temperatures are used, which negatively affects coherence.

    “Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Language models generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.

    “Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops.

    “Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in §5 and illustrated in Figure 4.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.

    “Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.

    “The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options, while in other contexts most of the probability mass is concentrated in one or a small number of tokens.

    “As illustrated on the left of Figure 5, in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g.nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The scaling factor p can vary wildly at each time-step, in contrast to Nucleus Sampling.

    “Note that the scaling factor p can vary wildly at each time-step, in contrast to Nucleus Sampling.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Setting the temperature lower creates diversity and repetition issues.

    “Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in §5.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Natural language does not tend to fall into repetition loops, even though the model tends to assign high probability to this.

    “Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Maximization-based decoding methods such as beam search lead to degeneration-output text that is bland, incoherent, or gets stuck in repetitive loops-even though likelihood training leads to high quality models.

    “The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration -output text that is bland, incoherent, or gets stuck in repetitive loops.”
    • Also stated in: § 1 INTRODUCTION · boosted
      “In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large, as shown in Figure 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.

    “The latter reveals that text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity.

    “very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( §4).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Nucleus Sampling can easily match reference perplexity through tuning the value of p, avoiding the incoherence caused by setting k high enough to match distributional statistics.

    “Nucleus Sampling can easily match reference perplexity through tuning the value of p, avoiding the incoherence caused by setting k high enough to match distributional statistics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Top-k sampling immediately veers off into an unrelated event.

    “Yet, top-k sampling immediately veers off into an unrelated event.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Pure sampling has higher perplexity than human text.

    “pure sampling also has higher perplexity than human text.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.

    “the current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The generation produced by Nucleus Sampling is not perfect; the model appears to confuse whales with birds and begins writing about those instead.

    “The generation produced by Nucleus Sampling isn't perfect -the model appears to confuse whales with birds, and begins writing about those instead.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word 'umidauda'.

    “Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Pure sampling is the closest to the human distribution, followed by Nucleus Sampling.

    “pure sampling is the closest to the human distribution, followed by Nucleus Sampling.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Human text has increased variance in the probability assigned to tokens compared with Beam Search, which shows endless repetition.

    “Note the increased variance that characterizes human text, in contrast with the endless repetition of text decoded by Beam Search.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Although beam search becomes less repetitive according to this metric as beam width increases, this is largely because average length gets shorter as b increases.

    “although beam search becomes less repetitive according to this metric as beam width increases, this is largely because average length gets shorter as b increases”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Initial exploration of applying HUSE directly led to top-k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods.

    “Initial exploration of applying HUSE directly led to top-k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Setting t ∈ [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.

    “Setting t ∈ [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.

    “The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.

    “Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone.

    “Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Low temperature sampling has been used to partially alleviate the issues of top-k sampling by shaping the distribution before top-k sampling.

    “Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling (Radford et al., 2018;Fan et al., 2018).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Radford et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Fan et al., 2018 source

  • May combine two claims.

    Interpretationown data

    The model is confusing itself when pure sampling: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text.

    “This indicates that the model is confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as in Figure 1.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    If k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.

    “Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Human evaluation is still required to get a full measure of the quality of generated text.

    “human evaluation is still required to get a full measure of the quality of the generated text.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Likelihood maximizing decoding causes repetition and overly generic language usage.

    “likelihood maximizing decoding causes repetition and overly generic language usage”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Statistical evaluations are unable to measure the coherence of generated text properly.

    “Statistical evaluations are unable to measure the coherence of generated text properly.”
    Quote in the paperNot a substantial claim
37 of 37

  • The authors state this as an assumption.

    Models compute the probability of a token sequence using the common left-to-right decomposition of text probability.

    “We assume that models compute P (x 1:m+n ) using the common left-to-right decomposition of the text probability,”

  • The authors state this as an assumption.

    The model assigns higher probability to higher quality text.

    “Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest”

  • The paper relies on this without support.

    The distribution of human-written text is the reference distribution for desirable generation.

    “we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.”

  • The paper relies on this without support.

    Repetition is an attribute of text quality, so less repetition is better.

    “One attribute of text quality that we can quantify is repetition.”

  • The paper relies on this without support.

    Zipf's law applies to natural language text.

    “Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text.”

  • The paper relies on this without support.

    Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.

    “Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.”

  • The paper relies on this without support.

    Values of k greater than 100 are rarely used in practice.

    “Values of k greater than 100 are rarely used in practice”

  • The paper relies on this without support.

    Values of p are usually in [0.9, 1).

    “values of p are usually in [0.9, 1)”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The issue was particularly pronounced in domain-specific medical models, where the complexity of terminology and extended reasoning chains exacerbate degeneration effects.

    “The issue was particularly pronounced in domain-specific medical models, where the complexity of terminology and extended reasoning chains exacerbate degeneration effects [62].”

    From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]

    The Introduction and Conclusion describe degeneration in general-purpose language generation, including repetition caused by likelihood-maximizing decoding. They do not report that degeneration is particularly pronounced in medical models or attribute it to complex terminology or extended reasoning chains (searched for medical, domain-specific, terminology, and reasoning). (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Text degeneration encompasses neural generation challenges such as repetition loops, semantic incoherence, and topic drift, often arising from exposure bias where early errors compound during sequential prediction.

    “Text degeneration encompasses neural generation challenges such as repetition loops, semantic incoherence, and topic drift, often arising from exposure bias where early errors compound during sequential prediction [56,62].”

    From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]

    The Introduction (page unavailable) describes repetitive beam-search output and pure-sampling output that is incoherent and almost unrelated to its context, supporting repetition, incoherence, and a form of topic drift. It attributes these problems to decoding choices and the unreliable tail of the predicted distribution, not to exposure bias or early errors compounding during sequential prediction; searches for those terms found no supporting passage. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Relying on inference-time methods that are closer in behavior to context-based losses provides a reasonable solution to the trade-off between novelty and consistency when prioritizing control, as long as the language model's distribution is sufficiently confident in its decisions.

    “this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.”

    From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • In the joint method, verification answers conditioning on the initial response may increase the likelihood of repetition.

    “This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”

    From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Holtzman et al., 2019

  • Repetition is a known issue of modern LLMs.

    “This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”

    From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Holtzman et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • GPT-2 Large has two reported parameter counts

    Minor

    The text reports GPT-2 Large as having 774M parameters in one place and 762M parameters in another.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Proceedings of the 2015 International Conference on Learning Representations, 2015.

Cited by

Sorted by citations

Related

Authors
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y.
Affiliations
  • University of Washington (US)
  • University of Cape Town (ZA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1904.09751
arXiv
1904.09751
OpenAlex
W2938704169
Semantic Scholar
cf4aa38ae31b43fd07abe13b4ffdb265babb7be1
Cited by
1,304
References
39
Citations by year
2026: 42 · 2025: 119 · 2024: 172 · 2023: 368 · 2022: 242 · 2021: 213 · 2020: 123 · 2019: 25
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1256082, 1524371)
  • Allen Institute for Artificial Intelligence
  • Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
  • Army Research Office
  • Naval Information Warfare Center Pacific
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
text generation, nucleus sampling, decoding strategies, text diversity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review