The DOI matches its DataCite record; 44 of the 44 references it cites were found in a registry or an index; 11 of the 29 sources its claims cite were checked for support.
The Curious Case of Neural Text Degeneration
Open accessAbstract
Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Maximization-based decoding such as beam search produces degenerate, repetitive text, while pure sampling from the full distribution produces incoherent text. The paper proposes Nucleus Sampling, which truncates the unreliable tail by sampling from the smallest top-p token set whose cumulative probability exceeds p, and reports it as the best overall decoding strategy for open-ended generation.
Key findings
- Beam search and greedy decoding lead to degeneration: in Table 1, Beam b=16 has perplexity 1.48 versus human 12.38, Self-BLEU 0.44 versus human 0.31, and repetition 28.94%; Greedy has repetition 73.66%.
- Pure sampling is incoherent and too unlikely: Table 1 reports Pure Sampling perplexity 22.73 versus human 12.38, with HUSE 0.67.
- Nucleus Sampling best matches human distributional statistics and wins human+statistical evaluation: Table 1 reports Nucleus p=0.95 perplexity 13.13, Self-BLEU 0.32, Zipf coefficient 0.95, repetition 0.36%, and HUSE 0.97, the highest HUSE score.
- Top-k sampling can match some statistics only at high k but suffers incoherence or repetition: Top-k=640 has perplexity 13.82 and HUSE 0.94, while Top-k=40, t=0.7 has repetition 8.86% and HUSE 0.08.
Methods
- Uses GPT-2 Large (762M parameters) trained on WebText, a 40GB web text collection; generates 5,000 text passages conditioned on the initial paragraph (1–40 tokens) of held-out WebText documents, ending at an end-of-document token or maximum length of 200 tokens.
- Compares decoding strategies including greedy, beam search, stochastic beam, pure sampling, temperature sampling, top-k sampling, and Nucleus Sampling using perplexity, Zipf coefficient, Self-BLEU, repetition, and HUSE.
- HUSE combines human and statistical evaluation: 200 generations per decoding algorithm are annotated for typicality by 20 annotators each, totaling 4,000 annotations per scheme; a KNN classifier with k=13 is used, and for top-k and Nucleus Sampling the truncated distribution is interpolated with mass 0.1.
Limitations
- The paper states that future work should dynamically characterize the model’s region of confidence and include a more semantic utility function to guide decoding.
- The paper states that initial HUSE application gave top-k and Nucleus Sampling near-zero scores due to truncation, so HUSE was computed with an interpolation proxy rather than the truncated distribution directly.
- Evident from the text: experiments use only GPT-2 Large on English WebText open-ended generation, with human evaluation limited to 200 generations per decoding scheme; repetition is measured as a phrase of minimum length 2 repeating at least three times at the end of the first 200 tokens, and the paper notes beam search repetition decreases partly because average generation length shortens as beam width increases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 of 29 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 2.2 OPEN-ENDED VS DIRECTED GENERATION
What the sources say
- Partly supportedcites Koehn & Knowles, 2017 sourceExperimental — accuracy not yet measured
Section 3.6 (Beam Search; page not provided) reports that translation quality declines with large beam sizes despite better model scores. It identifies shorter translations as the main cause, but does not report repetition or genericness as the problems observed with large beams. The exact-inference clause is attributed to a different citation and is not judged here. (LLM, conf=0.85)
- Could not verifycites Stahlberg & Byrne, 2019 sourceExperimental — accuracy not yet measured
The passages available from our copy are bibliographic records without paper sections or page text. Searches for exact inference, repetition, genericness, and beam-search errors returned no passage addressing the reported issues, so the attribution cannot be verified from our copy. (LLM, conf=0.72)
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018;Holtzman et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Not checked — no full text of the sourcecites Fan et al., 2018 source
- Supportedcites Holtzman et al., 2018 source
The Background (§2; page unavailable) says RNN language-model probabilities often favor repetitive, overly generic sentences. The Baselines (§4.2) use beam-search decoding, and the Quantitative Results (§5.1) report that the paper’s alternative decoding approach produces higher-quality text than those baselines in human evaluation. This supports the citing sentence’s point about maximization-based decoding in open-ended generation. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;Shen et al., 2017;Wiseman et al., 2017).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Partly supportedcites Chen et al., 2018 sourceExperimental — accuracy not yet measured
Section 4 (page unavailable) states that the best-string problem for RNNs is undecidable; Section 7 adds that the polynomial-length version for consistent RNNs is NP-complete. This supports the claim for the RNNs studied, but the paper does not establish the same result for Transformers. (LLM, conf=0.89)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, all stochastic methods face repetition issues when their tuning parameters are set too low, which tends to overtruncate, mimicking greedy search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“Vocabulary usage and Self-BLEU (Zhu et al., 2018) statistics reveal that high values of k are needed to make top-k sampling match human statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Zhu et al., 2018 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling -sampling directly from the probabilities predicted by the model -results in text that is incoherent and almost unrelated to the context.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even with our relatively fine-grained parameter sweep, Nucleus Sampling obtains closest perplexity to human text, as shown in Table 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 14
- It may state two separate claims as one. 25
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2019;Holtzman et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Partly supportedcites Kulikov et al., 2019 sourceExperimental — accuracy not yet measured
In “Diversity: distinct-n” (page unavailable), the paper reports that greedy search produces less diverse responses than the other strategies; “2.2 Inference (generation)” also describes greedy search as choosing the most likely token at each step. The copy does not establish that these outputs have high grammaticality, so that part of the claim is unsupported. (LLM, conf=0.87)
- Partly supportedcites Holtzman et al., 2018 sourceExperimental — accuracy not yet measured
The Background (§2; page unavailable) says RNN language models often assign higher probability to repetitive, overly generic sentences, supporting the low-diversity disadvantage of probability-based generation. The Qualitative Analysis (§5.2; page unavailable) likewise describes generic output from a language-model baseline. Our copy does not establish that such output has high grammaticality. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites Fan et al., 2018 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018;Hashimoto et al., 2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Supportedcites Caccia et al., 2018 source
The Introduction (p. 2) explicitly states that lower softmax temperatures produce less diverse, higher-quality samples. The synthetic-data experiment (p. 6) illustrates the trade-off: reducing temperature improves the quality metric, while an extremely low temperature can cause repeated output of a single sequence. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites Hashimoto et al., 2019 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018;Tevet et al., 2019;Semeniuta et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Supportedcites Caccia et al., 2018 source
The Empirical Results (p. 5) report that MLE-trained language models outperformed the GAN text generators tested when quality and diversity were assessed together. The Discussion (p. 9) confirms this finding for the GAN variants and tasks considered. (LLM, conf=0.94)
- Could not verifycites Tevet et al., 2019 sourceExperimental — accuracy not yet measured
The available indexed passages contain bibliographic records and an abstract, but no passage from the paper’s full text that can establish the claimed joint assessment of quality and diversity. Searches for that assessment and for evaluation terms returned no body-text passage, so the claim cannot be verified under the full-text-only evidence rule. (LLM, conf=0.72)
- Supportedcites Semeniuta et al., 2018 source
The Abstract (p. 1) says the authors used metrics intended to capture both quality and diversity and found that none of the GAN models they considered performed convincingly better than a conventional language model. Section 5.2 (p. 9) likewise states that no GAN model in their comparison was convincingly better. The finding is limited to the models they tested. (LLM, conf=0.92)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018;2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.1 EXPERIMENTAL SETUP
What the sources say
- Supportedcites Vaswani et al., 2017 source
Vaswani et al. is cited immediately after “the Transformer architecture,” identifying the architecture introduced in this paper. The Model Architecture section (p. 2) describes the Transformer; the separate claim about success in extremely large-scale training is cited to Radford et al., not to Vaswani et al. (LLM, conf=0.94)
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [2019]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the other hand, pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in §5 and illustrated in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated on the left of Figure 5, in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g.nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that the scaling factor p can vary wildly at each time-step, in contrast to Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in §5.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration -output text that is bland, incoherent, or gets stuck in repetitive loops.”
- Also stated in: § 1 INTRODUCTION · boosted
“In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large, as shown in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The latter reveals that text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( §4).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling can easily match reference perplexity through tuning the value of p, avoiding the incoherence caused by setting k high enough to match distributional statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, top-k sampling immediately veers off into an unrelated event.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“pure sampling also has higher perplexity than human text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The generation produced by Nucleus Sampling isn't perfect -the model appears to confuse whales with birds, and begins writing about those instead.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pure sampling is the closest to the human distribution, followed by Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note the increased variance that characterizes human text, in contrast with the endless repetition of text decoded by Beam Search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although beam search becomes less repetitive according to this metric as beam width increases, this is largely because average length gets shorter as b increases”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Initial exploration of applying HUSE directly led to top-k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Setting t ∈ [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling (Radford et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Fan et al., 2018 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that the model is confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human evaluation is still required to get a full measure of the quality of the generated text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“likelihood maximizing decoding causes repetition and overly generic language usage”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Statistical evaluations are unable to measure the coherence of generated text properly.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Models compute the probability of a token sequence using the common left-to-right decomposition of text probability.
“We assume that models compute P (x 1:m+n ) using the common left-to-right decomposition of the text probability,”
The authors state this as an assumption.
The model assigns higher probability to higher quality text.
“Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest”
The paper relies on this without support.
The distribution of human-written text is the reference distribution for desirable generation.
“we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.”
The paper relies on this without support.
Repetition is an attribute of text quality, so less repetition is better.
“One attribute of text quality that we can quantify is repetition.”
The paper relies on this without support.
Zipf's law applies to natural language text.
“Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text.”
The paper relies on this without support.
Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.
“Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.”
The paper relies on this without support.
Values of k greater than 100 are rarely used in practice.
“Values of k greater than 100 are rarely used in practice”
The paper relies on this without support.
Values of p are usually in [0.9, 1).
“values of p are usually in [0.9, 1)”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The issue was particularly pronounced in domain-specific medical models, where the complexity of terminology and extended reasoning chains exacerbate degeneration effects [62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Text degeneration encompasses neural generation challenges such as repetition loops, semantic incoherence, and topic drift, often arising from exposure bias where early errors compound during sequential prediction [56,62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The consequence of unrelated prefixes generating nonsensical outcomes underscores recent proposals like Top-k sampling (Fan et al. 2018) and Nucleus sampling (Holtzman et al. 2019), both employing truncated language model distributions to bias sampling towards the most probable tokens.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Holtzman et al. 2019
The cited source says what the claim attributes to it.
In the citing paper
“However, this is known to create repetitions of phrases or sentences even for large well-trained models (Radford et al., 2019; Holtzman et al., 2019).”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
GPT-2 Large has two reported parameter counts
MinorThe text reports GPT-2 Large as having 774M parameters in one place and 762M parameters in another.
“When b ≥ 64, both GPT-2 Large and XL (774M and 1542M parameters, respectively) prefer to stop generating immediately after the given context.”
“We perform experiments using the Large model (762M parameters).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
(2016) · Cited by 11,679
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,422
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Flagged first, then by citations
Cited by
Sorted by citations
Items 801–900 of 1,351
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“Generative tasks often involve various techniques for controlling the diversity and quality of the generated text, such as sampling with temperature, top-k or top-p (nucleus) sampling (Holtzman et al., 2020), and beam search (Li et al., 2016).”
Written vs generated text: “naturalness” as a textual and psycholinguistic category
Kolmogorova, A. V., Margolina, A. V., & Beograd, A. C. D. O. O. (2024) · RESEARCH RESULT Theoretical and Applied Linguistics · doi:10.18413/2313-8912-2024-10-2-0-4 · Cited by 5
Factuality Guided Diffusion-Based Abstractive Summarization
Shin, J., Park, H., & Song, H.-J. (2024) · IEEE Access · doi:10.1109/access.2024.3467063 · Cited by 5
Improving Machine Translation Formality with Large Language Models
Yang, M., & Li, F. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.058248 · Cited by 9
Smaller But Better: Unifying Layout Generation with Smaller Large Language Models
Zhang, P., Zhang, J., Cao, J., Li, H., & Jin, L. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02353-2 · Cited by 9
González, P. M., Huertas‐Tato, J., Martín, A. G., & Camacho, D. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103465 · Cited by 11
A comparative study of AI and human programming on environmental sustainability
Woo, N. H. (2025) · Scientific Reports · doi:10.1038/s41598-025-24658-5 · Cited by 7
Alberga, D., Nanna, V., Giovanna, M., Papadopoulos, E., Ancona, A., Lomuscio, M. C., & Mangiatordi, G. F. (2026) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6c00397 · Cited by 5
Adaptive Bridge between Training and Inference for Dialogue Generation
Xu, H., Zhang, H., Zou, Y., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.198 · Cited by 5
Massive-scale Decoding for Text Generation using Lattices
Xu, J., Jonnalagadda, S., & Durrett, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.344 · Cited by 7
MIL-Decoding: Detoxifying Language Models at Token-Level via Multiple Instance Learning
Zhang, X., & Wan, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.11 · Cited by 6
A Modern Turkish Poet: Fine-Tuned GPT-2
Kurt, U., & Çayır, A. (2023) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk59864.2023.10286720 · Cited by 5
Test-Time Self-Adaptive Small Language Models for Question Answering
Jeong, S., Baek, J., Cho, S., Hwang, S., & Park, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1033 · Cited by 6
Predictive typing method for Persian office automation
Nouraei, B., Shanbehzadeh, J., & Asghari, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107792 · Cited by 5
FREmax: A Simple Method Towards Truly Secure Generative Linguistic Steganography
Pang, K., Bai, M., Yang, J., Wang, H., Jiang, M., & Huang, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446027 · Cited by 5
TCLNet: Turn-level contrastive learning network with reranking for dialogue state tracking
You, C., & Xiong, D. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112308 · Cited by 5
Qu, Y., Huang, S., & Nie, P. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107707 · Cited by 5
Learning Evaluation Models From Large Language Models for Sequence Generation
Wang, C., Zhou, H., Chang, K., Liu, T., Zhang, C., Yang, M., Du, Q., Xiao, T., Zhang, Y., & Zhu, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587460 · Cited by 11
Tanaka, R., Ozeki, A., Kato, S., & Lee, A. (2020) · Computer Speech & Language · doi:10.1016/j.csl.2020.101070 · Cited by 6
On Reducing Repetition in Abstractive Summarization
Nair, P., & Singh, A. K. (2021) · Student Research Workshop .../Proceedings of the Student Research Workshop ... · doi:10.26615/issn.2603-2821.2021_018 · Cited by 5
Tuckute, G., Sathe, A., Wang, M., Yoder, H., Shain, C., & Fedorenko, E. G. (2022) · doi:10.18653/v1/2022.naacl-demo.11 · Cited by 5
PGTCN: A novel password-guessing model based on temporal convolution network
Wu, Y., Wan, X., Guan, X., et al. (2023) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2023.103592 · Cited by 5
Guan, J., Yang, Z., Zhang, R., Hu, Z., & Huang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26509 · Cited by 6
Tang, Z., Wang, P., Zhou, K., Li, J., Cao, Z., & Zhang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.721 · Cited by 14
Residual Learning of Neural Text Generation with n-gram Language Model
Li, H., Cai, D., Xu, J., & Watanabe, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.109 · Cited by 9
Sequentially Controlled Text Generation
Spangher, A., Ming, Y., Hua, X., & Peng, N. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.509 · Cited by 10
Reyes-Angulo, A. A., Paheding, S., Rajaneesh, A., Sajinkumar, K. S., & Oommen, T. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00041 · Cited by 5
PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation
Lu, H., & Lam, W. P. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.5 · Cited by 5
CogNLG: Cognitive graph for KG‐to‐text generation
Lai, P., Ye, F., Fu, Y.-G., Chen, Z., Wu, Y., Wang, Y., & Chang, V. I. C. (2023) · Expert Systems · doi:10.1111/exsy.13461 · Cited by 5
Jackson, I., & Rolf, B. (2023) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2023.10.1572 · Cited by 5
kNN-LM Does Not Improve Open-ended Text Generation
Wang, S., Song, Y., Drozdov, A., Garimella, A., Manjunatha, V., & Iyyer, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.929 · Cited by 12
Ersoy, A., Vizcarra, G., Mayeesha, T. T., & Müller, B. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.175 · Cited by 5
Domain Adaptation for Conversational Query Production with the RAG Model Feedback
Wang, A., Song, L., Ge, X., & Su, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.612 · Cited by 6
Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples
Ren, Z., Zhao, Y., & Zong, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.210 · Cited by 5
MedRedQA for Medical Consumer Question Answering: Dataset, Tasks, and Neural Baselines
Nguyen, V., Karimi, S., Rybinski, M., & Xing, Z. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.42 · Cited by 5
Conformal Autoregressive Generation: Beam Search with Coverage Guarantees
Deutschmann, N., Alberts, M., & Martínez, M. R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29062 · Cited by 11
From Deep Neural Language Models to LLMs
Kucharavy, A. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_1 · Cited by 5
Table-to-Text Generation With Pretrained Diffusion Models
Krylov, A. S., & Somov, O. (2024) · IEEE Access · doi:10.1109/access.2024.3440006 · Cited by 5
Large-Scale Bidirectional Training for Zero-Shot Image Captioning
Kim, T., Marsden, M. A., Ahn, P., Kim, S., Lee, S., Sala, A., & Kim, S. H. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00733 · Cited by 6
Bodaghi, A., Fung, B. C. M., & Schmitt, K. (2024) · ACM Transactions on the Web · doi:10.1145/3700791 · Cited by 8
Generating Dialogue Responses from a Semantic Latent Space
Ko, W.-J., Ray, A., Shen, Y., & Jin, H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.352 · Cited by 7
ConTest: A Unit Test Completion Benchmark featuring Context
Villmow, J., Depoix, J., & Ulges, A. (2021) · doi:10.18653/v1/2021.nlp4prog-1.2 · Cited by 5
Towards improving coherence and diversity of slogan generation
Jin, Y., Bhatia, A., Wanvarie, D., & Le, P. T. V. (2022) · Natural Language Engineering · doi:10.1017/s1351324921000474 · Cited by 6
LaMemo: Language Modeling with Look-Ahead Memory
Ji, H., Zhang, R., Yang, Z., Hu, Z., & Huang, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.422 · Cited by 5
Entailment Relation Aware Paraphrase Generation
Sancheti, A., Srinivasan, B. V., & Rudinger, R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21376 · Cited by 6
Explainable Natural Language Inference via Identifying Important Rationales
Yang, Z., Dong, S., & Hu, J. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3178391 · Cited by 5
Sparks: Inspiration for Science Writing using Language Models
Gero, K. I., Liu, V., & Chilton, L. B. (2022) · Workshop on Intelligent and Interactive Writing Assistants (In2Writing) · doi:10.18653/v1/2022.in2writing-1.12 · Cited by 7
Are Experts Needed? On Human Evaluation of Counselling Reflection Generation
Wu, Z., Balloccu, S., Reiter, E., Helaoui, R., Recupero, D. R., & Riboni, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.382 · Cited by 6
Click: Controllable Text Generation with Sequence Likelihood Contrastive Learning
Zheng, C., Ke, P., Zhang, Z., & Huang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.65 · Cited by 27
Query-Efficient Black-Box Red Teaming via Bayesian Optimization
Lee, D., Lee, J., Ha, J.-W., Kim, J.-H., Lee, S.-W., Lee, H., & Song, H. O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.646 · Cited by 28
Unsupervised Opinion Summarisation in the Wasserstein Space
Song, J., Bilal, I. M., Tsakalidis, A., Procter, R., & Liakata, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.589 · Cited by 6
GePSAn: Generative Procedure Step Anticipation in Cooking Videos
Abdelslam, M. A., Rangrej, S. B., Hadji, I., Dvornik, N., Derpanis, K. G., & Fazly, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00279 · Cited by 5
Proceedings of the 2020 Joint Conference on AI Music Creativity
Bazin, T., Hadjeres, G., Esling, P., & Malt, M. (2021) · KTH Royal Institute of Technology eBooks · doi:10.30746/978-91-519-5560-5 · Cited by 6
Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/747 · Cited by 5
Designing Precise and Robust Dialogue Response EvaluatorsPreprint
Zhao, T., Lala, D., & Kawahara, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04908 · Cited by 5
GenerRNA: A generative pre-trained language model for de novo RNA designPreprint
Zhao, Y., Oono, K., Takizawa, H., & Kotera, M. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.02.01.578496 · Cited by 5
DocSpider: a dataset of cross-domain natural language querying for MongoDB
Özer, A. G., Çekinel, R. F., Toroslu, İ. H., & Karagöz, P. (2025) · Natural language processing. · doi:10.1017/nlp.2024.63 · Cited by 6
Evaluating the Test Adequacy of Benchmarks for LLMs on Code Generation
Liu, X., Sun, X., Bo, L., Hu, Y., Liu, X., & Ye, Z. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70034 · Cited by 4
Chemical language models for natural product discovery
Sakano, K., Furui, K., Kengkanna, A., Kikuchi, Y., & Ohue, M. (2026) · Natural Product Reports · doi:10.1039/d6np00002a · Cited by 4
IMPACTO DE LOS MODELOS GENERATIVOS DE LENGUAJE DE INTELIGENCIA ARTIFICIAL EN LA EDUCACIÓN SUPERIOR
Lara-Colón, R., Castañón-Ayala, L., & Romo-Rodríguez, P. (2024) · TLATEMOANI Revista Académica de Investigación · doi:10.51896/tlatemoani/taru9220 · Cited by 8
Significant ASR Error Detection for Conversational Voice Assistants
Harvill, J. B., Khaziev, R., Li, S., Cogill, R., Wang, L., Chennupati, G., & Thadakamalla, H. P. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448230 · Cited by 4
Semi-supervised Chinese poem-to-painting generation via cycle-consistent adversarial networks
Lu, Z., Guo, T., & Wang, F. (2024) · Journal of Electronic Imaging · doi:10.1117/1.jei.33.5.053056 · Cited by 6
Technische Grundlagen großer Sprachmodelle
Blüthgen, C. (2025) · Die Radiologie · doi:10.1007/s00117-025-01427-z · Cited by 6
Zhao, J., Song, Y., & Cohen, E. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112493 · Cited by 4
Assessing and enhancing the reliability of Chinese large language models in dental implantology
Zhu, G., Zhang, X., & Chen, C. (2025) · BMC Oral Health · doi:10.1186/s12903-025-06648-1 · Cited by 5
A Generative Approach to Enrich Arabic Story Text with Visual Aids
Zakraoui, J., Maadeed, S. A., El-Seoud, S. A., et al. (2021) · International Conference on Software and Information Engineering (ICSIE) · doi:10.1145/3512716.3512725 · Cited by 4
SURF: Semantic-level Unsupervised Reward Function for Machine Translation
Anuchitanukul, A., & Ive, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.334 · Cited by 5
Suri, K., Saha, S., & Singh, A. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.50 · Cited by 3
Towards Pragmatic Production Strategies for Natural Language Generation Tasks
Giulianelli, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.544 · Cited by 4
GAP-Gen: Guided Automatic Python Code Generation
Zhao, J., Song, Y., Wang, J., & Harris, I. G. (2023) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.18653/v1/2023.eacl-srw.4 · Cited by 6
Text-Only Training for Visual Storytelling
Wang, Y., Zhou, W., Lu, Z., & Li, H. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612179 · Cited by 5
Sherstinova, T. Y., Kolobov, R., & Mikhaylovskiy, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48309-7_4 · Cited by 4
Strategies for framing argumentative conclusion generation
Heinisch, P., Frank, A., Opitz, J., et al. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.20 · Cited by 4
Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System
Ippolito, D., Carlini, N., Lee, K., Nasr, M., & Yu, Y. W. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.28 · Cited by 7
Screening through a broad pool: Towards better diversity for lexically constrained text generation
Yuan, C., Huang, H., Cao, Y., & Cao, Q. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103602 · Cited by 5
Cowap, A., Graham, Y., & Foster, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.665 · Cited by 4
Knowledge Graph Compression Enhances Diverse Commonsense Generation
Hwang, E., Thost, V., Shwartz, V., & Ma, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.37 · Cited by 5
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models
Farinhas, A., de Souza, J. G. C., & Martins, A. F. T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.733 · Cited by 14
Liu, J., Liu, J., Wang, Q., Wang, J., Cai, X., Zhao, D., Wang, R., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.578 · Cited by 4
Natural Language Decompositions of Implicit Content Enable Better Text Representations
Hoyle, A., Sarkar, R., Goel, P., & Resnik, P. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.815 · Cited by 10
Enhanced Workflow Management using an Artificial Intelligence ChatBot
Reitenbach, S., Siggel, M., & Bolemant, M. (2024) · AIAA SciTech Forum · doi:10.2514/6.2024-0917 · Cited by 5
Varshney, D., & Ekbal, A. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101645 · Cited by 5
An exploratory study on dialect density estimation for children and adult's African American English
Johnson, A., Shankar, N. B., Mari Ostendorf, E., & Alwan, A. A. (2024) · The Journal of the Acoustical Society of America · doi:10.1121/10.0025771 · Cited by 4
Fang, X. F., Yang, H., Shi, L., Wang, Y., & Li, L. (2024) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2024.3412273 · Cited by 4
Exploring the Effectiveness of Abstract Syntax Tree Patterns for Algorithm Recognition
Neumüller, D., Sihler, F., Straub, R., & Tichy, M. (2024) · International Conference on Code Quality (ICCQ) · doi:10.1109/iccq60895.2024.10576984 · Cited by 4
AI as a Tool for Fair Journalism: Case Studies from Malta
Seychell, D., Hili, G., Attard, J., & Makantatis, K. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00032 · Cited by 7
COT: A Generative Approach for Hate Speech Counter-Narratives via Contrastive Optimal Transport
Zhang, L., Jin, L., Xu, G., et al. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3406691 · Cited by 4
Evaluating the Robustness of ASR Systems in Adverse Acoustic Conditions
Katkov, S., Liotta, A., & Vietti, A. (2024) · International Conference on Intelligent Data Science Technologies and Applications (IDSTA) · doi:10.1109/idsta62194.2024.10746999 · Cited by 4
Kim, B., Seo, S., Han, S., Erdenee, E., & Chang, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.286 · Cited by 6
Reddy, R. G., Sultan, M. A., Franz, M., Sil, A., & Ji, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531878 · Cited by 5
Dependency-based Mixture Language Models
Yang, Z., & Wan, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.535 · Cited by 4
Sagirova, A., & Burtsev, M. (2022) · Cognitive Systems Research · doi:10.1016/j.cogsys.2022.05.002 · Cited by 4
A generalized decoding method for neural text generation
Gong, N., & Yao, N. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101503 · Cited by 4
Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets
Laban, P., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.135 · Cited by 3
Conditional set generation using Seq2seq models
Madaan, A., Rajagopal, D., Tandon, N., Yang, Y., & Bosselut, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.324 · Cited by 5
Characterizing verbatim short-term memory in neural language models
Armeni, K., Honey, C., & Linzen, T. (2022) · Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2022.conll-1.28 · Cited by 4
Decoding a Neural Retriever’s Latent Space for Query Suggestion
Adolphs, L., Huebscher, M. C., Buck, C., Girgin, S., Bachem, O., Ciaramita, M., & Hofmann, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.601 · Cited by 9
Priya, P., Mishra, K., Totala, P., & Ekbal, A. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/686 · Cited by 4
Metadata
- Authors
- Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y.
- Affiliations
- University of Washington (US)
- University of Cape Town (ZA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1904.09751
- OpenAlex
- W2938704169
- Semantic Scholar
- cf4aa38ae31b43fd07abe13b4ffdb265babb7be1
- Cited by
- 1,304
- References
- 39
- Citations by year
- 2026: 42 · 2025: 119 · 2024: 172 · 2023: 368 · 2022: 242 · 2021: 213 · 2020: 123 · 2019: 25
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1256082, 1524371)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Army Research Office
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- text generation, nucleus sampling, decoding strategies, text diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
