The DOI matches its DataCite record; 44 of the 44 references it cites were found in a registry or an index; 11 of the 29 sources its claims cite were checked for support.
The Curious Case of Neural Text Degeneration
Open accessAbstract
Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Maximization-based decoding such as beam search produces degenerate, repetitive text, while pure sampling from the full distribution produces incoherent text. The paper proposes Nucleus Sampling, which truncates the unreliable tail by sampling from the smallest top-p token set whose cumulative probability exceeds p, and reports it as the best overall decoding strategy for open-ended generation.
Key findings
- Beam search and greedy decoding lead to degeneration: in Table 1, Beam b=16 has perplexity 1.48 versus human 12.38, Self-BLEU 0.44 versus human 0.31, and repetition 28.94%; Greedy has repetition 73.66%.
- Pure sampling is incoherent and too unlikely: Table 1 reports Pure Sampling perplexity 22.73 versus human 12.38, with HUSE 0.67.
- Nucleus Sampling best matches human distributional statistics and wins human+statistical evaluation: Table 1 reports Nucleus p=0.95 perplexity 13.13, Self-BLEU 0.32, Zipf coefficient 0.95, repetition 0.36%, and HUSE 0.97, the highest HUSE score.
- Top-k sampling can match some statistics only at high k but suffers incoherence or repetition: Top-k=640 has perplexity 13.82 and HUSE 0.94, while Top-k=40, t=0.7 has repetition 8.86% and HUSE 0.08.
Methods
- Uses GPT-2 Large (762M parameters) trained on WebText, a 40GB web text collection; generates 5,000 text passages conditioned on the initial paragraph (1–40 tokens) of held-out WebText documents, ending at an end-of-document token or maximum length of 200 tokens.
- Compares decoding strategies including greedy, beam search, stochastic beam, pure sampling, temperature sampling, top-k sampling, and Nucleus Sampling using perplexity, Zipf coefficient, Self-BLEU, repetition, and HUSE.
- HUSE combines human and statistical evaluation: 200 generations per decoding algorithm are annotated for typicality by 20 annotators each, totaling 4,000 annotations per scheme; a KNN classifier with k=13 is used, and for top-k and Nucleus Sampling the truncated distribution is interpolated with mass 0.1.
Limitations
- The paper states that future work should dynamically characterize the model’s region of confidence and include a more semantic utility function to guide decoding.
- The paper states that initial HUSE application gave top-k and Nucleus Sampling near-zero scores due to truncation, so HUSE was computed with an interpolation proxy rather than the truncated distribution directly.
- Evident from the text: experiments use only GPT-2 Large on English WebText open-ended generation, with human evaluation limited to 200 generations per decoding scheme; repetition is measured as a phrase of minimum length 2 repeating at least three times at the end of the first 200 tokens, and the paper notes beam search repetition decreases partly because average generation length shortens as beam width increases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 of 29 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 2.2 OPEN-ENDED VS DIRECTED GENERATION
What the sources say
- Partly supportedcites Koehn & Knowles, 2017 sourceExperimental — accuracy not yet measured
Section 3.6 (Beam Search; page not provided) reports that translation quality declines with large beam sizes despite better model scores. It identifies shorter translations as the main cause, but does not report repetition or genericness as the problems observed with large beams. The exact-inference clause is attributed to a different citation and is not judged here. (LLM, conf=0.85)
- Could not verifycites Stahlberg & Byrne, 2019 sourceExperimental — accuracy not yet measured
The passages available from our copy are bibliographic records without paper sections or page text. Searches for exact inference, repetition, genericness, and beam-search errors returned no passage addressing the reported issues, so the attribution cannot be verified from our copy. (LLM, conf=0.72)
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018;Holtzman et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Not checked — no full text of the sourcecites Fan et al., 2018 source
- Supportedcites Holtzman et al., 2018 source
The Background (§2; page unavailable) says RNN language-model probabilities often favor repetitive, overly generic sentences. The Baselines (§4.2) use beam-search decoding, and the Quantitative Results (§5.1) report that the paper’s alternative decoding approach produces higher-quality text than those baselines in human evaluation. This supports the citing sentence’s point about maximization-based decoding in open-ended generation. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;Shen et al., 2017;Wiseman et al., 2017).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Partly supportedcites Chen et al., 2018 sourceExperimental — accuracy not yet measured
Section 4 (page unavailable) states that the best-string problem for RNNs is undecidable; Section 7 adds that the polynomial-length version for consistent RNNs is NP-complete. This supports the claim for the RNNs studied, but the paper does not establish the same result for Transformers. (LLM, conf=0.89)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, all stochastic methods face repetition issues when their tuning parameters are set too low, which tends to overtruncate, mimicking greedy search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“Vocabulary usage and Self-BLEU (Zhu et al., 2018) statistics reveal that high values of k are needed to make top-k sampling match human statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Zhu et al., 2018 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling -sampling directly from the probabilities predicted by the model -results in text that is incoherent and almost unrelated to the context.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even with our relatively fine-grained parameter sweep, Nucleus Sampling obtains closest perplexity to human text, as shown in Table 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 14
- It may state two separate claims as one. 25
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2019;Holtzman et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Partly supportedcites Kulikov et al., 2019 sourceExperimental — accuracy not yet measured
In “Diversity: distinct-n” (page unavailable), the paper reports that greedy search produces less diverse responses than the other strategies; “2.2 Inference (generation)” also describes greedy search as choosing the most likely token at each step. The copy does not establish that these outputs have high grammaticality, so that part of the claim is unsupported. (LLM, conf=0.87)
- Partly supportedcites Holtzman et al., 2018 sourceExperimental — accuracy not yet measured
The Background (§2; page unavailable) says RNN language models often assign higher probability to repetitive, overly generic sentences, supporting the low-diversity disadvantage of probability-based generation. The Qualitative Analysis (§5.2; page unavailable) likewise describes generic output from a language-model baseline. Our copy does not establish that such output has high grammaticality. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites Fan et al., 2018 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018;Hashimoto et al., 2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Supportedcites Caccia et al., 2018 source
The Introduction (p. 2) explicitly states that lower softmax temperatures produce less diverse, higher-quality samples. The synthetic-data experiment (p. 6) illustrates the trade-off: reducing temperature improves the quality metric, while an extremely low temperature can cause repeated output of a single sequence. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites Hashimoto et al., 2019 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018;Tevet et al., 2019;Semeniuta et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Supportedcites Caccia et al., 2018 source
The Empirical Results (p. 5) report that MLE-trained language models outperformed the GAN text generators tested when quality and diversity were assessed together. The Discussion (p. 9) confirms this finding for the GAN variants and tasks considered. (LLM, conf=0.94)
- Could not verifycites Tevet et al., 2019 sourceExperimental — accuracy not yet measured
The available indexed passages contain bibliographic records and an abstract, but no passage from the paper’s full text that can establish the claimed joint assessment of quality and diversity. Searches for that assessment and for evaluation terms returned no body-text passage, so the claim cannot be verified under the full-text-only evidence rule. (LLM, conf=0.72)
- Supportedcites Semeniuta et al., 2018 source
The Abstract (p. 1) says the authors used metrics intended to capture both quality and diversity and found that none of the GAN models they considered performed convincingly better than a conventional language model. Section 5.2 (p. 9) likewise states that no GAN model in their comparison was convincingly better. The finding is limited to the models they tested. (LLM, conf=0.92)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018;2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.1 EXPERIMENTAL SETUP
What the sources say
- Supportedcites Vaswani et al., 2017 source
Vaswani et al. is cited immediately after “the Transformer architecture,” identifying the architecture introduced in this paper. The Model Architecture section (p. 2) describes the Transformer; the separate claim about success in extremely large-scale training is cited to Radford et al., not to Vaswani et al. (LLM, conf=0.94)
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [2019]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the other hand, pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in §5 and illustrated in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated on the left of Figure 5, in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g.nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that the scaling factor p can vary wildly at each time-step, in contrast to Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in §5.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration -output text that is bland, incoherent, or gets stuck in repetitive loops.”
- Also stated in: § 1 INTRODUCTION · boosted
“In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large, as shown in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The latter reveals that text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( §4).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling can easily match reference perplexity through tuning the value of p, avoiding the incoherence caused by setting k high enough to match distributional statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, top-k sampling immediately veers off into an unrelated event.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“pure sampling also has higher perplexity than human text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The generation produced by Nucleus Sampling isn't perfect -the model appears to confuse whales with birds, and begins writing about those instead.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pure sampling is the closest to the human distribution, followed by Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note the increased variance that characterizes human text, in contrast with the endless repetition of text decoded by Beam Search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although beam search becomes less repetitive according to this metric as beam width increases, this is largely because average length gets shorter as b increases”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Initial exploration of applying HUSE directly led to top-k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Setting t ∈ [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling (Radford et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Fan et al., 2018 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that the model is confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human evaluation is still required to get a full measure of the quality of the generated text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“likelihood maximizing decoding causes repetition and overly generic language usage”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Statistical evaluations are unable to measure the coherence of generated text properly.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Models compute the probability of a token sequence using the common left-to-right decomposition of text probability.
“We assume that models compute P (x 1:m+n ) using the common left-to-right decomposition of the text probability,”
The authors state this as an assumption.
The model assigns higher probability to higher quality text.
“Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest”
The paper relies on this without support.
The distribution of human-written text is the reference distribution for desirable generation.
“we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.”
The paper relies on this without support.
Repetition is an attribute of text quality, so less repetition is better.
“One attribute of text quality that we can quantify is repetition.”
The paper relies on this without support.
Zipf's law applies to natural language text.
“Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text.”
The paper relies on this without support.
Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.
“Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.”
The paper relies on this without support.
Values of k greater than 100 are rarely used in practice.
“Values of k greater than 100 are rarely used in practice”
The paper relies on this without support.
Values of p are usually in [0.9, 1).
“values of p are usually in [0.9, 1)”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The issue was particularly pronounced in domain-specific medical models, where the complexity of terminology and extended reasoning chains exacerbate degeneration effects [62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Text degeneration encompasses neural generation challenges such as repetition loops, semantic incoherence, and topic drift, often arising from exposure bias where early errors compound during sequential prediction [56,62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The consequence of unrelated prefixes generating nonsensical outcomes underscores recent proposals like Top-k sampling (Fan et al. 2018) and Nucleus sampling (Holtzman et al. 2019), both employing truncated language model distributions to bias sampling towards the most probable tokens.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Holtzman et al. 2019
The cited source says what the claim attributes to it.
In the citing paper
“However, this is known to create repetitions of phrases or sentences even for large well-trained models (Radford et al., 2019; Holtzman et al., 2019).”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
GPT-2 Large has two reported parameter counts
MinorThe text reports GPT-2 Large as having 774M parameters in one place and 762M parameters in another.
“When b ≥ 64, both GPT-2 Large and XL (774M and 1542M parameters, respectively) prefer to stop generating immediately after the given context.”
“We perform experiments using the Large model (762M parameters).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
(2016) · Cited by 11,679
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,422
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Flagged first, then by citations
Cited by
Sorted by citations
Items 201–300 of 1,351
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
Ng, E., Romero, J., Bagautdinov, T., Bai, S., Darrell, T., Kanazawa, A., & Richard, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00101 · Cited by 71
Large Language Models Produce Responses Perceived to be Empathic
Lee, Y. K., Suh, J., Zhan, H., Li, J. J., & Ong, D. C. (2024) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii63134.2024.00012 · Cited by 104
Shen, T., Quach, V., Barzilay, R., & Jaakkola, T. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.420 · Cited by 58
Uncertainty Estimation in Autoregressive Structured Prediction
Gales, M., & Malinin, A. (2021) · Apollo - University of Cambridge Repository · doi:10.17863/cam.63497 · Cited by 48
Generating Question-Answer Hierarchies
Krishna, K., & Iyyer, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1224 · Cited by 44
Das, A., Selek, S., Warner, A. R., Zuo, X., Hu, Y., Kuttichi Keloth, V., Li, J., Zheng, W. J., & Xu, H. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.27 · Cited by 47
ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning
Boratko, M., Li, X. L., O’Gorman, T., Das, R., Le, D., & McCallum, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.85 · Cited by 56
EmpTransfo: A Multi-head Transformer Architecture for Creating Empathetic Dialog SystemsPreprint
Zandie, R., & Mahoor, M. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02958 · Cited by 53
Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation
Huang, K.-H., McKeown, K. R., Nakov, P., Choi, Y., & Ji, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.815 · Cited by 68
Self-citationRLIPv2: Fast Scaling of Relational Language-Image Pre-training
Yuan, H., Zhang, S., Wang, X., Albanie, S., Pan, Y., Feng, T., Jiang, J., Ni, D., Zhang, Y., & Zhao, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01979 · Cited by 53
Computer-Aided Design as LanguagePreprint
Ganin, Y., Bartunov, S., Li, Y., Keller, E. A., & Saliceti, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.02769 · Cited by 56
Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation
Huang, K.-H., McKeown, K., Nakov, P., Choi, Y., & Ji, H. (2022) · arXiv · doi:10.48550/arxiv.2203.05386 · Cited by 46
Self-citationHoldout-Based Empirical Assessment of Mixed-Type Synthetic Data
Platzer, M., & Reutterer, T. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.679939 · Cited by 57
An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction
Zaratiana, U., Tomeh, N., Holat, P., & Charnois, T. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29919 · Cited by 51
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models
Aghakhani, H., Dai, W., Manoel, A., Fernandes, X., Kharkar, A., Kruegel, C., Vigna, G., Evans, D., Zorn, B., & Sim, R. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00140 · Cited by 72
Less is More: Learning to Refine Dialogue History for Personalized Dialogue Generation
Zhong, H., Dou, Z., Zhu, Y., Qian, H., & Wen, J.-R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.426 · Cited by 46
Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing
Chakrabarty, T., Padmakumar, V., & He, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.460 · Cited by 86
Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins
Bhatnagar, A., Jain, S., Beazer, J., Curran, S. C., Hoffnagle, A. M., Ching, K., Martyn, M., Nayfach, S., Ruffolo, J. A., & Madani, A. (2025) · neural information processing systems · doi:10.52202/085713-1536 · Cited by 55
“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models
Shwartz, V., Rudinger, R., & Tafjord, O. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.556 · Cited by 52
Explaining Question Answering Models through Text GenerationPreprint
Latcinnik, V., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05569 · Cited by 60
Knowledge-Aware Language Model PretrainingPreprint
Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P. N., & Tiwary, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00655 · Cited by 84
Semnani, S. J., Yao, V., Zhang, H., & Lam, M. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.157 · Cited by 78
PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
Hua, X., & Wang, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.57 · Cited by 48
Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
Welleck, S., Kulikov, I., Kim, J., Pang, R. Y., & Cho, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.448 · Cited by 60
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43
Trading Off Diversity and Quality in Natural Language GenerationPreprint
Zhang, H., Duckworth, D., Ippolito, D., & Neelakantan, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10450 · Cited by 68
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023) · arXiv · doi:10.48550/arxiv.2309.11495 · Cited by 42
“An alternative to using the confidence scores is to leverage inconsistencies in the LLMs output to detect hallucination.”
BackgroundInfluentialDugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26501 · Cited by 54
Shen, S., WELCH, C. A., Mihalcea, R. F., & Pérez‐Rosas, V. (2020) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.2 · Cited by 42
On Decoding Strategies for Neural Text Generators
Wiher, G., Meister, C., & Cotterell, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00502 · Cited by 67
Chen, H., & Ding, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.858 · Cited by 60
Diversifying Dialogue Generation with Non-Conversational Text
Su, H., Shen, X., Zhao, S., Zhou, X., Hu, P., Zhong, R., Niu, C., & Zhou, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.634 · Cited by 49
We’re Afraid Language Models Aren’t Modeling Ambiguity
Liu, A., Wu, Z., Michael, J., Suhr, A., West, P., Koller, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.51 · Cited by 113
Self-citationSimplifying Paragraph-Level Question Generation via Transformer Language Models
Lopez, L. E., Cruz, D. K., Cruz, J. C. B., & Cheng, C. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-89363-7_25 · Cited by 41
Hajipour, H., Haßler, K., Holz, T., Schönherr, L., & Fritz, M. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00040 · Cited by 87
Parallel Refinements for Lexically Constrained Text Generation with BART
He, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.681 · Cited by 54
Jiang, Z., Guo, J., Sun, S., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01765 · Cited by 40
Do Neural Language Models Overcome Reporting Bias?
Shwartz, V., & Choi, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.605 · Cited by 48
Self-citationAnalyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models
He, T., Liu, J., Cho, K., Ott, M., Liu, B., Glass, J. R., & Peng, F. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.95 · Cited by 43
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Generating Synthetic Training Data for Supervised De-Identification of Electronic Health Records
Libbi, C. A., Trienes, J., Trieschnigg, D., & Seifert, C. (2021) · Future Internet · doi:10.3390/fi13050136 · Cited by 45
Core Challenges in Embodied Vision-Language Planning
Francis, J., Kitamura, N., Labelle, F., Lu, X., Navarro, I., & Oh, J. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13646 · Cited by 44
Quality-Aware Decoding for Neural Machine Translation
Fernandes, P., Farinhas, A., Rei, R., de Souza, J. G. C., Ogayo, P., Neubig, G., & Martins, A. F. T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.100 · Cited by 61
Beyond [CLS] through Ranking by Generation
dos Santos, C. N., Ma, X., Nallapati, R. M., Huang, Z., & Xiang, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.134 · Cited by 44
Evaluating Factuality in Text Simplification
Devaraj, A., Sheffield, W., Wallace, B., & Li, J. J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.506 · Cited by 40
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42
“…been employed for detection, including entropy (Gehrmann et al., 2019, Lavergne et al., 2008), perplexity (Beres-neva, 2016, Hans et al., 2024), average log-probability score (So-laiman et al., 2019), fluency (Holtzman et al., 2020), and Zipf’s word frequency law (Piantadosi, 2014, Zipf, 2016) 2 .”
Gaur, M., Gunaratna, K., Srinivasan, V., & Jin, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21312 · Cited by 57
Si, W. M., Michael Backes, A., Blackburn, J., Cristofaro, E. D., Stringhini, G., Zannettou, S., & Zhang, Y. (2022) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3548606.3560599 · Cited by 52
Chorus: Foundation Models for Unified Data Discovery and Exploration
Kayali, M., Lykov, A., Fountalis, I., Vasiloglou, N., Olteanu, D., & Suciu, D. M. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3659437.3659461 · Cited by 70
Designing Precise and Robust Dialogue Response Evaluators
Zhao, T., Lala, D., & Kawahara, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.4 · Cited by 41
“Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.60 · Cited by 48
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37
Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models
Zhu, Y., Li, J., Li, G., Zhao, Y., Li, J., Jin, Z. G., & Mei, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27798 · Cited by 69
Simple and Effective Retrieve-Edit-Rerank Text Generation
Hossain, N., Ghazvininejad, M., & Zettlemoyer, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.228 · Cited by 37
AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data
Xu, S., Semnani, S. J., Campagna, G., & Lam, M. S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.31 · Cited by 44
Understanding Neural Abstractive Summarization Models via Uncertainty
Xu, J., Desai, S., & Durrett, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.508 · Cited by 43
LightSeq: A High Performance Inference Library for Transformers
Wang, X., Xiong, Y., Wei, Y., Wang, M., & Li, L. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers · doi:10.18653/v1/2021.naacl-industry.15 · Cited by 36
It’s not Rocket Science: Interpreting Figurative Language in Narratives
Chakrabarty, T., Choi, Y., & Shwartz, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00478 · Cited by 42
Self-citationWei, T., Yuan, W., Luo, J., Zhang, W., & Lu, L. (2023) · Journal of Systems Engineering and Electronics · doi:10.23919/jsee.2023.000035 · Cited by 36
Video Background Music Generation: Dataset, Method and Evaluation
Zhuo, L., Wang, Z., Wang, B., Liao, Y., Bao, C., Peng, S., Han, S., Zhang, A., Fang, F., & Liu, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01433 · Cited by 36
A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations
Colombo, P., Piantanida, P., & Clavel, C. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.511 · Cited by 42
Personalized Response Generation via Generative Split Memory Network
Wu, Y., Ma, X., & Yang, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.157 · Cited by 40
EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training
Gu, Y., Wen, J., Sun, H., Song, Y., Ke, P., Zheng, C., Zhang, Z., Yao, J., Liu, L., Zhu, X., et al. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1387-3 · Cited by 44
Automatic story and item generation for reading comprehension assessments with transformers
Bulut, O., & Yildirim‐Erbasli, S. N. (2022) · International Journal of Assessment Tools in Education · doi:10.21449/ijate.1124382 · Cited by 40
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
Liu, J., Hallinan, S., Lu, X., He, P., Welleck, S., Hajishirzi, H., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.611 · Cited by 48
Self-citationSynthetic Text Generation with Differential Privacy: A Simple and Practical Recipe
Yue, X., Inan, H. A., Li, X., Kumar, G., McAnallen, J., Shajari, H., Sun, H., Levitan, D., & Sim, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.74 · Cited by 101
Symbolic Music Generation Conditioned on Continuous-Valued Emotions
Sulun, S., Davies, M. E. P., & Viana, P. (2022) · IEEE Access · doi:10.1109/access.2022.3169744 · Cited by 43
Hessel, J., Marasović, A., Hwang, J. D., Lee, L., Da, J., Zellers, R., Mankoff, R., & Choi, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.41 · Cited by 79
Self-citationDistributed Inference and Fine-tuning of Large Language Models Over The Internet
Borzunov, A., Ryabinin, M., Chumachenko, A., Baranchuk, D., Dettmers, T., Belkada, Y., Samygin, P., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2312.08361 · Cited by 35
Li, H., Xu, M., & Song, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.881 · Cited by 79
Demi̇r, G. B., Sükut, Y., Duran, G. S., Topsakal, K. G., & Görgülü, S. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae011 · Cited by 34
Chen, Z., Huang, H., Andrusenko, A., Hrinchuk, O., Puvvada, K. C., Li, J., Ghosh, S., Balam, J., & Ginsburg, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447553 · Cited by 44
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 41
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
A Critical Evaluation of Evaluations for Long-form Question Answering
Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.181 · Cited by 33
EmoMusicTV: Emotion-Conditioned Symbolic Music Generation With Hierarchical Transformer VAE
Ji, S., & Yang, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3276177 · Cited by 34
Conformal Prediction for Natural Language Processing: A Survey
Campos, M., Farinhas, A., Zerva, C., Figueiredo, M. A. T., & Martins, A. F. T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00715 · Cited by 63
Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models
Pang, J., Ye, F., Wong, D. F., Yu, D., Shi, S., Tu, Z., & Wang, L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00730 · Cited by 71
GPT-based Generation for Classical Chinese PoetryPreprint
Liao, Y., Wang, Y., Liu, Q., & Jiang, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00151 · Cited by 46
Improving dense retrieval models with LLM augmented data for dataset search
Silva, L., & Barbosa, L. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111740 · Cited by 37
Point2Building: Reconstructing buildings from airborne LiDAR point clouds
Liu, Y., Obukhov, A., Wegner, J. D., & Schindler, K. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.07.012 · Cited by 32
Fine-Tuning Large Language Models to Improve Accuracy and Comprehensibility of Automated Code Review
Yu, Y., Rong, G., Shen, H., Zhang, H., Shao, D., Wang, M., Wei, Z., Xu, Y., & Wang, J. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695993 · Cited by 47
Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling
Chen, H., Huang, Y., Takamura, H., & Nakayama, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16184 · Cited by 32
Controllable Open-ended Question Generation with A New Question Type Ontology
Cao, S., & Wang, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.502 · Cited by 42
Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology
Halperin, B. A., & Lukin, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580744 · Cited by 37
PromptEHR: Conditional Electronic Healthcare Records Generation with Prompt Learning
Wang, Z., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.185 · Cited by 38
Santis, E. D., Martino, A., & Rizzi, A. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3358168 · Cited by 38
Arora, K., Asri, L. E., Bahuleyan, H., & Cheung, J. C. K. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.58 · Cited by 100
Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective
Uchendu, A., Le, T., & Lee, D. (2023) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3606274.3606276 · Cited by 44
Dale, D. C., Voita, E., Barrault, L., & Costa‐jussà, M. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.3 · Cited by 61
Faithfulness-Aware Decoding Strategies for Abstractive Summarization
Wan, D., Liu, M., McKeown, K. R., Dreyer, M., & Bansal, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.210 · Cited by 46
CALYPSO: LLMs as Dungeon Master's Assistants
Zhu, A., Martin, L., Head, A., & Callison-Burch, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v19i1.27534 · Cited by 31
I can’t believe there’s no images! : Learning Visual Tasks Using Only Language Supervision
Gu, S., Clark, C., & Kembhavi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00252 · Cited by 43
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection
Fung, Y., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., & Sil, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.133 · Cited by 52
Tell2Design: A Dataset for Language-Guided Floor Plan Generation
Leng, S., Zhou, Y. M., Dupty, M. H., Lee, W. S., Joyce, S. C., & Lu, J. W. Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.820 · Cited by 41
Three Bricks to Consolidate Watermarks for Large Language Models
Fernandez, P., Chaffin, A., Tit, K., Chappelier, V., & Furon, T. (2023) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs58808.2023.10374576 · Cited by 63
Red Teaming Language Model Detectors with Language Models
Shi, Z., Wang, Y., Yin, F., Chen, X., Chang, K.-W., & Hsieh, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00639 · Cited by 65
TextGAIL: Generative Adversarial Imitation Learning for Text Generation
Wu, Q., Li, L., & Yu, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17656 · Cited by 31
G2GT: Retrosynthesis Prediction with Graph-to-Graph Attention Neural Network and Self-Training
Lin, Z., Yin, S., Shi, L., Zhou, W., & Zhang, Y. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c01302 · Cited by 34
Metadata
- Authors
- Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y.
- Affiliations
- University of Washington (US)
- University of Cape Town (ZA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1904.09751
- OpenAlex
- W2938704169
- Semantic Scholar
- cf4aa38ae31b43fd07abe13b4ffdb265babb7be1
- Cited by
- 1,304
- References
- 39
- Citations by year
- 2026: 42 · 2025: 119 · 2024: 172 · 2023: 368 · 2022: 242 · 2021: 213 · 2020: 123 · 2019: 25
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1256082, 1524371)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Army Research Office
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- text generation, nucleus sampling, decoding strategies, text diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
