The DOI matches its DataCite record; 44 of the 44 references it cites were found in a registry or an index; 11 of the 29 sources its claims cite were checked for support.
The Curious Case of Neural Text Degeneration
Open accessAbstract
Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Maximization-based decoding such as beam search produces degenerate, repetitive text, while pure sampling from the full distribution produces incoherent text. The paper proposes Nucleus Sampling, which truncates the unreliable tail by sampling from the smallest top-p token set whose cumulative probability exceeds p, and reports it as the best overall decoding strategy for open-ended generation.
Key findings
- Beam search and greedy decoding lead to degeneration: in Table 1, Beam b=16 has perplexity 1.48 versus human 12.38, Self-BLEU 0.44 versus human 0.31, and repetition 28.94%; Greedy has repetition 73.66%.
- Pure sampling is incoherent and too unlikely: Table 1 reports Pure Sampling perplexity 22.73 versus human 12.38, with HUSE 0.67.
- Nucleus Sampling best matches human distributional statistics and wins human+statistical evaluation: Table 1 reports Nucleus p=0.95 perplexity 13.13, Self-BLEU 0.32, Zipf coefficient 0.95, repetition 0.36%, and HUSE 0.97, the highest HUSE score.
- Top-k sampling can match some statistics only at high k but suffers incoherence or repetition: Top-k=640 has perplexity 13.82 and HUSE 0.94, while Top-k=40, t=0.7 has repetition 8.86% and HUSE 0.08.
Methods
- Uses GPT-2 Large (762M parameters) trained on WebText, a 40GB web text collection; generates 5,000 text passages conditioned on the initial paragraph (1–40 tokens) of held-out WebText documents, ending at an end-of-document token or maximum length of 200 tokens.
- Compares decoding strategies including greedy, beam search, stochastic beam, pure sampling, temperature sampling, top-k sampling, and Nucleus Sampling using perplexity, Zipf coefficient, Self-BLEU, repetition, and HUSE.
- HUSE combines human and statistical evaluation: 200 generations per decoding algorithm are annotated for typicality by 20 annotators each, totaling 4,000 annotations per scheme; a KNN classifier with k=13 is used, and for top-k and Nucleus Sampling the truncated distribution is interpolated with mass 0.1.
Limitations
- The paper states that future work should dynamically characterize the model’s region of confidence and include a more semantic utility function to guide decoding.
- The paper states that initial HUSE application gave top-k and Nucleus Sampling near-zero scores due to truncation, so HUSE was computed with an interpolation proxy rather than the truncated distribution directly.
- Evident from the text: experiments use only GPT-2 Large on English WebText open-ended generation, with human evaluation limited to 200 generations per decoding scheme; repetition is measured as a phrase of minimum length 2 repeating at least three times at the end of the first 200 tokens, and the paper notes beam search repetition decreases partly because average generation length shortens as beam width increases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 of 29 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 2.2 OPEN-ENDED VS DIRECTED GENERATION
What the sources say
- Partly supportedcites Koehn & Knowles, 2017 sourceExperimental — accuracy not yet measured
Section 3.6 (Beam Search; page not provided) reports that translation quality declines with large beam sizes despite better model scores. It identifies shorter translations as the main cause, but does not report repetition or genericness as the problems observed with large beams. The exact-inference clause is attributed to a different citation and is not judged here. (LLM, conf=0.85)
- Could not verifycites Stahlberg & Byrne, 2019 sourceExperimental — accuracy not yet measured
The passages available from our copy are bibliographic records without paper sections or page text. Searches for exact inference, repetition, genericness, and beam-search errors returned no passage addressing the reported issues, so the attribution cannot be verified from our copy. (LLM, conf=0.72)
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018;Holtzman et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Not checked — no full text of the sourcecites Fan et al., 2018 source
- Supportedcites Holtzman et al., 2018 source
The Background (§2; page unavailable) says RNN language-model probabilities often favor repetitive, overly generic sentences. The Baselines (§4.2) use beam-search decoding, and the Quantitative Results (§5.1) report that the paper’s alternative decoding approach produces higher-quality text than those baselines in human evaluation. This supports the citing sentence’s point about maximization-based decoding in open-ended generation. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;Shen et al., 2017;Wiseman et al., 2017).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Partly supportedcites Chen et al., 2018 sourceExperimental — accuracy not yet measured
Section 4 (page unavailable) states that the best-string problem for RNNs is undecidable; Section 7 adds that the polynomial-length version for consistent RNNs is NP-complete. This supports the claim for the RNNs studied, but the paper does not establish the same result for Transformers. (LLM, conf=0.89)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, all stochastic methods face repetition issues when their tuning parameters are set too low, which tends to overtruncate, mimicking greedy search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIn the paper
“Vocabulary usage and Self-BLEU (Zhu et al., 2018) statistics reveal that high values of k are needed to make top-k sampling match human statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Zhu et al., 2018 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling -sampling directly from the probabilities predicted by the model -results in text that is incoherent and almost unrelated to the context.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even with our relatively fine-grained parameter sweep, Nucleus Sampling obtains closest perplexity to human text, as shown in Table 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 14
- It may state two separate claims as one. 25
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2019;Holtzman et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Partly supportedcites Kulikov et al., 2019 sourceExperimental — accuracy not yet measured
In “Diversity: distinct-n” (page unavailable), the paper reports that greedy search produces less diverse responses than the other strategies; “2.2 Inference (generation)” also describes greedy search as choosing the most likely token at each step. The copy does not establish that these outputs have high grammaticality, so that part of the claim is unsupported. (LLM, conf=0.87)
- Partly supportedcites Holtzman et al., 2018 sourceExperimental — accuracy not yet measured
The Background (§2; page unavailable) says RNN language models often assign higher probability to repetitive, overly generic sentences, supporting the low-diversity disadvantage of probability-based generation. The Qualitative Analysis (§5.2; page unavailable) likewise describes generic output from a language-model baseline. Our copy does not establish that such output has high grammaticality. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites Fan et al., 2018 source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018;Hashimoto et al., 2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Supportedcites Caccia et al., 2018 source
The Introduction (p. 2) explicitly states that lower softmax temperatures produce less diverse, higher-quality samples. The synthetic-data experiment (p. 6) illustrates the trade-off: reducing temperature improves the quality metric, while an extremely low temperature can cause repeated output of a single sequence. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites Hashimoto et al., 2019 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018;Tevet et al., 2019;Semeniuta et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Supportedcites Caccia et al., 2018 source
The Empirical Results (p. 5) report that MLE-trained language models outperformed the GAN text generators tested when quality and diversity were assessed together. The Discussion (p. 9) confirms this finding for the GAN variants and tasks considered. (LLM, conf=0.94)
- Could not verifycites Tevet et al., 2019 sourceExperimental — accuracy not yet measured
The available indexed passages contain bibliographic records and an abstract, but no passage from the paper’s full text that can establish the claimed joint assessment of quality and diversity. Searches for that assessment and for evaluation terms returned no body-text passage, so the claim cannot be verified under the full-text-only evidence rule. (LLM, conf=0.72)
- Supportedcites Semeniuta et al., 2018 source
The Abstract (p. 1) says the authors used metrics intended to capture both quality and diversity and found that none of the GAN models they considered performed convincingly better than a conventional language model. Section 5.2 (p. 9) likewise states that no GAN model in their comparison was convincingly better. The finding is limited to the models they tested. (LLM, conf=0.92)
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018;2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.1 EXPERIMENTAL SETUP
What the sources say
- Supportedcites Vaswani et al., 2017 source
Vaswani et al. is cited immediately after “the Transformer architecture,” identifying the architecture introduced in this paper. The Model Architecture section (p. 2) describes the Transformer; the separate claim about success in extremely large-scale training is cited to Radford et al., not to Vaswani et al. (LLM, conf=0.94)
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [2019]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the other hand, pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in §5 and illustrated in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated on the left of Figure 5, in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g.nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that the scaling factor p can vary wildly at each time-step, in contrast to Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in §5.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration -output text that is bland, incoherent, or gets stuck in repetitive loops.”
- Also stated in: § 1 INTRODUCTION · boosted
“In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large, as shown in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Abstract
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The latter reveals that text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( §4).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling can easily match reference perplexity through tuning the value of p, avoiding the incoherence caused by setting k high enough to match distributional statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes? Based: unclearIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, top-k sampling immediately veers off into an unrelated event.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“pure sampling also has higher perplexity than human text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The generation produced by Nucleus Sampling isn't perfect -the model appears to confuse whales with birds, and begins writing about those instead.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pure sampling is the closest to the human distribution, followed by Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note the increased variance that characterizes human text, in contrast with the endless repetition of text decoded by Beam Search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although beam search becomes less repetitive according to this metric as beam width increases, this is largely because average length gets shorter as b increases”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Initial exploration of applying HUSE directly led to top-k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Setting t ∈ [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling (Radford et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Fan et al., 2018 source
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that the model is confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human evaluation is still required to get a full measure of the quality of the generated text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“likelihood maximizing decoding causes repetition and overly generic language usage”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Statistical evaluations are unable to measure the coherence of generated text properly.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Models compute the probability of a token sequence using the common left-to-right decomposition of text probability.
“We assume that models compute P (x 1:m+n ) using the common left-to-right decomposition of the text probability,”
The authors state this as an assumption.
The model assigns higher probability to higher quality text.
“Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest”
The paper relies on this without support.
The distribution of human-written text is the reference distribution for desirable generation.
“we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.”
The paper relies on this without support.
Repetition is an attribute of text quality, so less repetition is better.
“One attribute of text quality that we can quantify is repetition.”
The paper relies on this without support.
Zipf's law applies to natural language text.
“Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text.”
The paper relies on this without support.
Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.
“Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.”
The paper relies on this without support.
Values of k greater than 100 are rarely used in practice.
“Values of k greater than 100 are rarely used in practice”
The paper relies on this without support.
Values of p are usually in [0.9, 1).
“values of p are usually in [0.9, 1)”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The issue was particularly pronounced in domain-specific medical models, where the complexity of terminology and extended reasoning chains exacerbate degeneration effects [62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Text degeneration encompasses neural generation challenges such as repetition loops, semantic incoherence, and topic drift, often arising from exposure bias where early errors compound during sequential prediction [56,62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The consequence of unrelated prefixes generating nonsensical outcomes underscores recent proposals like Top-k sampling (Fan et al. 2018) and Nucleus sampling (Holtzman et al. 2019), both employing truncated language model distributions to bias sampling towards the most probable tokens.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Holtzman et al. 2019
The cited source says what the claim attributes to it.
In the citing paper
“However, this is known to create repetitions of phrases or sentences even for large well-trained models (Radford et al., 2019; Holtzman et al., 2019).”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
GPT-2 Large has two reported parameter counts
MinorThe text reports GPT-2 Large as having 774M parameters in one place and 762M parameters in another.
“When b ≥ 64, both GPT-2 Large and XL (774M and 1542M parameters, respectively) prefer to stop generating immediately after the given context.”
“We perform experiments using the Large model (762M parameters).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
(2016) · Cited by 11,679
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,422
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Flagged first, then by citations
Cited by
Sorted by citations
Items 401–500 of 1,351
Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification
Li, B., Xu, X., Wang, X., Hou, Y., Feng, Y., Wang, F., Zhang, X., Zhu, Q., & Che, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28084 · Cited by 21
Watermarking for Large Language Models: A Survey
Yang, Z., Zhao, G., & Wu, H. (2025) · Mathematics · doi:10.3390/math13091420 · Cited by 27
Reverse Engineering Configurations of Neural Text Generation Models
Tay, Y., Bahri, D., Zheng, C., Brunk, C. A., Metzler, D., & Tomkins, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.25 · Cited by 31
Attention Is Indeed All You Need: Semantically Attention-Guided Decoding for Data-to-Text NLG
Juraska, J., & Walker, M. A. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.45 · Cited by 20
Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet
He, X., & Li, V. O. K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17536 · Cited by 19
Li, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04195 · Cited by 25
Data Augmentation for Spoken Language Understanding via Pretrained ModelsPreprint
Peng, B., Zhu, C., Zeng, M., & Gao, J. (2020) · arXiv (Cornell University) · Cited by 18
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15343 · Cited by 152
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 33
A Cognitive Regularizer for Language Modeling
Wei, J., Meister, C., & Cotterell, R. (2021) · doi:10.18653/v1/2021.acl-long.404 · Cited by 18
DeepAltTrip: Top-K Alternative Itineraries for Trip Recommendation
Rashid, S. M. M., Ali, M. E., & Cheema, M. A. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3239595 · Cited by 18
Comparative Explanations of Recommendations
Yang, A., Wang, N., Cai, R., Deng, H., & Wang, H. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3512031 · Cited by 19
CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering
Yue, X., Zhang, X. F., Yao, Z., Lin, S. M., & Sun, H. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669300 · Cited by 22
Towards an open-domain chatbot for language practice
Tyen, G., Brenchley, M., Caines, A., & Buttery, P. J. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.28 · Cited by 23
Sparsing and Smoothing for the seq2seq Models
Zhao, S., Liang, Z., Wen, J., & Chen, J. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3207982 · Cited by 20
How Far are We from Robust Long Abstractive Summarization?
Koh, H. Y., Ju, J., Zhang, H., Liu, M. Z., & Pan, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.172 · Cited by 21
He, C., Tan, T.-P., Xue, S., & Tan, Y. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101868 · Cited by 20
Aligning Large Language Models through Synthetic Feedback
Kim, S., Bae, S., Shin, J., Kang, S., Kwak, D., Yoo, K. M., & Seo, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.844 · Cited by 80
Generative Artificial Intelligence in Clinical Medicine and Impact on Gastroenterology
Soroush, A., Giuffrè, M., Chung, S., & Shung, D. L. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.03.038 · Cited by 23
Deep Learning for Predictive Business Process Monitoring: Review and Benchmark
Rama-Maneiro, E., Vidal, J. C., & Lama, M. (2021) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2021.3139807 · Cited by 129
Learning to Select External Knowledge With Multi-Scale Negative Sampling
Huang, H., Lu, H., Bao, S., Wang, F., Wu, H. R., Niu, Z.-Y., & Wang, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3301222 · Cited by 22
Model for Cooking Recipe Generation using Reinforcement Learning
Fujita, J., Sato, M., & Nobuhara, H. (2021) · IEEE International Conference on Data Engineering Workshops (ICDEW) · doi:10.1109/icdew53142.2021.00007 · Cited by 18
DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer
Ji, H., & Huang, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.347 · Cited by 21
A Knowledge-Enhanced Pretraining Model for Commonsense Story GenerationPreprint
Guan, J., Huang, F., Zhao, Z., Zhu, X., & Huang, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.05139 · Cited by 18
An Overview on Language Models: Recent Developments and OutlookPreprint
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05759 · Cited by 15
Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity
Shuster, K., Urbanek, J., Szlam, A., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.182 · Cited by 20
AI-generated text detection: A comprehensive review of methods, datasets, and applications
Kehkashan, T., Riaz, R. A., Al‐Shamayleh, A. S., Akhunzada, A., Ali, N., Hamza, M., & Akbar, F. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100793 · Cited by 24
Limits of Detecting Text Generated by Large-Scale Language Models
Varshney, L. R., Keskar, N. S., & Socher, R. (2020) · Information Theory and Applications Workshop (ITA) · doi:10.1109/ita50056.2020.9245012 · Cited by 21
Generating Empathetic Responses by Injecting Anticipated Emotion
Liu, Y., Du, J., Li, X., & Xu, R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413596 · Cited by 17
Educational Multi-Question Generation for Reading Comprehension
Rathod, M., Tu, T., & Stasaski, K. (2022) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2022.bea-1.26 · Cited by 17
Enhancing Knowledge Selection for Grounded Dialogues via Document Semantic Graphs
Li, S., Namazifar, M., Jin, D., Bansal, M., Ji, H., Liu, Y., & Hakkani-Tür, D. Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.202 · Cited by 19
Semantic Diversity in Dialogue with Natural Language Inference
Stasaski, K., & Hearst, M. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.6 · Cited by 30
Controlling Perceived Emotion in Symbolic Music Generation with Monte Carlo Tree Search
Ferreira, L. N., Mou, L., Whitehead, J., & Lelis, L. H. S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v18i1.21960 · Cited by 17
Uncertainty-Aware Unlikelihood Learning Improves Generative Aspect Sentiment Quad Prediction
Hu, M., Bai, Y., Wu, Y., Zhang, Z., Zhang, L., Gao, H., Zhao, S., & Huang, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.851 · Cited by 19
Zhang, Z., Wen, J., & Huang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.709 · Cited by 34
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation
Bhagavatula, C., Hwang, J. D., Downey, D., Bras, R. J. L., Lu, X., Qin, L., Sakaguchi, K., Swayamdipta, S., West, P., & Choi, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.535 · Cited by 27
Self-citationBenchmarking and Improving Text-to-SQL Generation under Ambiguity
Bhaskar, A., Tomar, T., Sathe, A., & Sarawagi, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.436 · Cited by 22
DLT: Conditioned layout generation with Joint Discrete-Continuous Diffusion Layout Transformer
Levi, E., Brosh, E., Mykhailych, M., & Perez, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00201 · Cited by 19
Awal, M. R., & Asaduzzaman (2024) · Higher Education Skills and Work-based Learning · doi:10.1108/heswbl-09-2023-0249 · Cited by 19
Deep learning for code generation: a survey
Zhang, H., Zhang, K., Li, Z., Li, J., Li, J., Li, Y., Zhao, Y., Zhu, Y. S., Liu, F., Li, G., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3956-3 · Cited by 20
Ceron, T., Falk, N., Barić, A., Nikolaev, D., & Padó, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00710 · Cited by 23
Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers
Shi, Y., Zhang, H., Wan, C., & Gu, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00005 · Cited by 38
Relevance-Promoting Language Model for Short-Text Conversation
Li, X., Li, P., Bi, W., Liu, X., & Lam, W. P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6340 · Cited by 16
Gopalakrishnan, K., Hedayatnia, B., Wang, L., Liu, Y., & Hakkani-Tür, D. Z. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1508 · Cited by 24
NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation
Kim, S., Chang, M., & Lee, S.-W. (2021) · doi:10.18653/v1/2021.acl-long.287 · Cited by 20
Isonuma, M., Mori, J., Bollegala, D., & Sakata, I. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00406 · Cited by 18
Using Question Answering Rewards to Improve Abstractive Summarization
Gunasekara, C., Feigenblat, G., Sznajder, B., Aharonov, R., & Joshi, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.47 · Cited by 17
Guo, Z., Kang, J., & Herremans, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25635 · Cited by 24
One-Class Learning for AI-Generated Essay Detection
Corizzo, R., & Leal-Arenas, S. (2023) · Applied Sciences · doi:10.3390/app13137901 · Cited by 16
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Mohammadshahi, A., Scialom, T., Yazdani, M., Yanki, P., Fan, A., Henderson, J., & Saeidi, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.428 · Cited by 28
Differentially Private Language Models for Secure Data Sharing
Mattern, J., Jin, Z., Weggenmann, B., Schölkopf, B., & Sachan, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.323 · Cited by 49
Natural Language Deduction through Search over Statement Compositions
Bostrom, K., Sprague, Z., Chaudhuri, S., & Durrett, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.358 · Cited by 35
Fusing Pre-Trained Language Models with Multimodal Prompts through Reinforcement Learning
Yu, Y., Chung, J., Yun, H., Hessel, J., Park, J. S., Lu, X., Zellers, R., Ammanabrolu, P., Bras, R. J. L., Kim, G., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01044 · Cited by 17
Self-citationAI to Train AI: Using ChatGPT to Improve the Accuracy of a Therapeutic Dialogue System
Gabor-Siatkowska, K., Sowański, M., Rzatkiewicz, R., Stefaniak, I., Kozłowski, M., & Janicki, A. (2023) · Electronics · doi:10.3390/electronics12224694 · Cited by 16
Revisiting Machine Translation for Cross-lingual Classification
Artetxe, M., Goswami, V., Bhosale, S., Fan, A., & Zettlemoyer, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.399 · Cited by 33
Li, J., Luo, Y., Li, G., et al. (2024) · Geophysics · doi:10.1190/geo2023-0205.1 · Cited by 16
Automatic Medical Report Generation: Methods and Applications
Li, G., Tahir, A. M., Zhang, D., Wang, Z. J., & Ward, R. K. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.20240044 · Cited by 20
Generation of Hip-Hop Lyrics with Hierarchical Modeling and Conditional Templates
Manjavacas, E., Kestemont, M., & Karsdorp, F. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8638 · Cited by 18
Fˆ2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmax
Choi, B.-J., Hong, J., Park, D., & Lee, S. W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.737 · Cited by 16
Shakeri, S., Constant, N., Kale, M., & Xue, L. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.4 · Cited by 22
DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation
Hua, X., Sreevatsa, A., & Wang, L. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.501 · Cited by 20
Tekiroğlu, S. S., Bonaldi, H., Fanton, M., & Guerini, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.245 · Cited by 25
Manakul, P., Gales, M., & Wang, L. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1683 · Cited by 15
Persona-Guided Planning for Controlling the Protagonist’s Persona in Story Generation
Zhang, Z., Wen, J., Guan, J., & Huang, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.245 · Cited by 19
Li, H., Song, Y., & Fan, L. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.429 · Cited by 17
Wang, W., Ge, L., Zhang, J., & Yang, C. Z. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531823 · Cited by 19
Coherence boosting: When your pretrained language model is not paying enough attention
Malkin, N., Wang, Z., & Jojić, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.565 · Cited by 30
Studying Psychosis Using Natural Language Generation: A Review of Emerging Opportunities
Palaniyappan, L. K., Benrimoh, D. A., Voppel, A., & Rocca, R. (2023) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2023.04.009 · Cited by 14
Mishra, K., Priya, P., & Ekbal, A. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26685 · Cited by 16
A Practical Toolkit for Multilingual Question and Answer Generation
Ushio, A., Alva-Manchego, F. E., & Camacho-Collados, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.8 · Cited by 17
Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking
King, B., & Flanigan, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.344 · Cited by 22
RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation
Liu, S., Cho, H., Freedman, M., Ma, X., & May, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.468 · Cited by 19
EtriCA: Event-Triggered Context-Aware Story Generation Augmented by Cross Attention
Tang, C., Lin, C., Huang, H., Guérin, F., & Zhang, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.403 · Cited by 19
Crossing the Gap: Domain Generalization for Image Captioning
Ren, Y., Mao, Z., Fang, S., Lu, Y., He, T., Du, H., Zhang, Y., & Ouyang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00281 · Cited by 16
Explaining Data Patterns in Natural Language with Language Models
Singh, C., Morris, J. X., Aneja, J., Rush, A. M., & Gao, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.3 · Cited by 17
Evaluating Large Language Models on Controlled Generation Tasks
Sun, J., Tian, Y., Zhou, W., Xu, N., Hu, Q., Gupta, R., Wieting, J., Peng, N., & Ma, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.190 · Cited by 29
Self-Detoxifying Language Models via Toxification Reversal
Leong, C. T., Cheng, Y., Wang, J., Wang, J., & Li, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.269 · Cited by 35
MapPrior: Bird’s-Eye View Map Layout Estimation with Generative Models
Zhu, X., Zyrianov, V., Liu, Z., & Wang, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00756 · Cited by 17
Wu, S.-L., Chang, X., Wichern, G., Jung, J.-W., Germain, F. G., Roux, J. L., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447215 · Cited by 18
APIGen: Generative API Method Recommendation
Chen, Y., Gao, C., Zhu, M., Liao, Q., Wang, Y., & Xu, G. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00025 · Cited by 25
Sieve: Multimodal Dataset Pruning Using Image Captioning Models
Mahmoud, A., Elhoushi, M., Abbas, A., Yu, Y., Ardalani, N., Leather, H., & Morcos, A. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02116 · Cited by 31
Publicly-Detectable Watermarking for Language Models
Fairoze, J., Garg, S., Jha, S., Mahloujifar, S., Mahmoody, M., & Wang, M. (2025) · IACR Communications in Cryptology · doi:10.62056/ahmpdkp10 · Cited by 63
Evaluation of Generative AI Models in Python Code Generation: A Comparative Study
Palla, D., & Slabý, A. (2025) · IEEE Access · doi:10.1109/access.2025.3560244 · Cited by 27
Breaking Writer’s Block: Low-cost Fine-tuning of Natural Language Generation Models
Duval, A., Lamson, T., de Kérouara, G. D. L., & Gallé, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.33 · Cited by 20
End-to-End Learning of Flowchart Grounded Task-Oriented Dialogs
Raghu, D., Agarwal, S., Joshi, S., & Mausam, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.357 · Cited by 23
Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning
Honda, U., Watanabe, T., & Matsumoto, Y. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00118 · Cited by 15
Parallel Scheduled SamplingPreprint
Duckworth, D., Neelakantan, A., Goodrich, B., Kaiser, Ł., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04331 · Cited by 27
Finding Experts in Transformer ModelsPreprint
Suau, X., Zappella, L., & Apostoloff, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.07647 · Cited by 29
Using In-Context Learning to Improve Dialogue Safety
Meade, N., Gella, S., Hazarika, D., Gupta, P., Jin, D., Reddy, S., Liu, Y., & Hakkani-Tur, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.796 · Cited by 13
Warm-Starting for Improving the Novelty of Abstractive Summarization
Alomari, A., Al‐Shamayleh, A. S., Idris, N. B., Sabri, A. Q. M., Alsmadi, I. M., & Omary, D. (2023) · IEEE Access · doi:10.1109/access.2023.3322226 · Cited by 17
Langsmith: An Interactive Academic Text Revision System
Ito, T., Kuribayashi, T., Hidaka, M., Suzuki, J., & Inui, K. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.28 · Cited by 19
Natural-Language Multi-Agent Simulations of Argumentative Opinion Dynamics
Betz, G. (2022) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.4725 · Cited by 19
Leashing the Inner Demons: Self-Detoxification for Language Models
Xu, C., He, Z., He, Z., & McAuley, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21406 · Cited by 17
Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings
Krishna, K., Nathani, D., Garcia, X., Samanta, B., & Talukdar, P. P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.514 · Cited by 22
CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning
Wang, W., Fang, T., Xu, B., Bo, C. Y. L., Song, Y., & Chen, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.733 · Cited by 17
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
Shi, C., Su, Y., Yang, C. H., Yang, Y., & Cai, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.947 · Cited by 24
On the Zero-Shot Generalization of Machine-Generated Text Detectors
Xiao, P., Zhang, J., Han, X., Tsvetkov, Y., & He, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.318 · Cited by 24
Integrating Visual Context Into Language Models for Situated Social Conversation Starters
Janssens, R., Wolfert, P., Demeester, T., & Belpaeme, T. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3428704 · Cited by 15
Metadata
- Authors
- Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y.
- Affiliations
- University of Washington (US)
- University of Cape Town (ZA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1904.09751
- OpenAlex
- W2938704169
- Semantic Scholar
- cf4aa38ae31b43fd07abe13b4ffdb265babb7be1
- Cited by
- 1,304
- References
- 39
- Citations by year
- 2026: 42 · 2025: 119 · 2024: 172 · 2023: 368 · 2022: 242 · 2021: 213 · 2020: 123 · 2019: 25
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1256082, 1524371)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Army Research Office
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- text generation, nucleus sampling, decoding strategies, text diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
