The DOI matches its DataCite record; 44 of the 44 references it cites were found in a registry or an index; 11 of the 29 sources its claims cite were checked for support.
The Curious Case of Neural Text Degeneration
Open accessAbstract
Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Maximization-based decoding such as beam search produces degenerate, repetitive text, while pure sampling from the full distribution produces incoherent text. The paper proposes Nucleus Sampling, which truncates the unreliable tail by sampling from the smallest top-p token set whose cumulative probability exceeds p, and reports it as the best overall decoding strategy for open-ended generation.
Key findings
- Beam search and greedy decoding lead to degeneration: in Table 1, Beam b=16 has perplexity 1.48 versus human 12.38, Self-BLEU 0.44 versus human 0.31, and repetition 28.94%; Greedy has repetition 73.66%.
- Pure sampling is incoherent and too unlikely: Table 1 reports Pure Sampling perplexity 22.73 versus human 12.38, with HUSE 0.67.
- Nucleus Sampling best matches human distributional statistics and wins human+statistical evaluation: Table 1 reports Nucleus p=0.95 perplexity 13.13, Self-BLEU 0.32, Zipf coefficient 0.95, repetition 0.36%, and HUSE 0.97, the highest HUSE score.
- Top-k sampling can match some statistics only at high k but suffers incoherence or repetition: Top-k=640 has perplexity 13.82 and HUSE 0.94, while Top-k=40, t=0.7 has repetition 8.86% and HUSE 0.08.
Methods
- Uses GPT-2 Large (762M parameters) trained on WebText, a 40GB web text collection; generates 5,000 text passages conditioned on the initial paragraph (1–40 tokens) of held-out WebText documents, ending at an end-of-document token or maximum length of 200 tokens.
- Compares decoding strategies including greedy, beam search, stochastic beam, pure sampling, temperature sampling, top-k sampling, and Nucleus Sampling using perplexity, Zipf coefficient, Self-BLEU, repetition, and HUSE.
- HUSE combines human and statistical evaluation: 200 generations per decoding algorithm are annotated for typicality by 20 annotators each, totaling 4,000 annotations per scheme; a KNN classifier with k=13 is used, and for top-k and Nucleus Sampling the truncated distribution is interpolated with mass 0.1.
Limitations
- The paper states that future work should dynamically characterize the model’s region of confidence and include a more semantic utility function to guide decoding.
- The paper states that initial HUSE application gave top-k and Nucleus Sampling near-zero scores due to truncation, so HUSE was computed with an interpolation proxy rather than the truncated distribution directly.
- Evident from the text: experiments use only GPT-2 Large on English WebText open-ended generation, with human evaluation limited to 200 generations per decoding scheme; repetition is measured as a phrase of minimum length 2 repeating at least three times at the end of the first 200 tokens, and the paper notes beam search repetition decreases partly because average generation length shortens as beam width increases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
33 Own result · 22 Own interpretation · 20 No basis given · 5 Partly supported · 5 Not checked yet · 3 No full text of the sources · 2 Supported so far · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 2.2 OPEN-ENDED VS DIRECTED GENERATION
What the sources say
- Partly supportedcites Koehn & Knowles, 2017 sourceExperimental — accuracy not yet measured
Section 3.6 (Beam Search; page not provided) reports that translation quality declines with large beam sizes despite better model scores. It identifies shorter translations as the main cause, but does not report repetition or genericness as the problems observed with large beams. The exact-inference clause is attributed to a different citation and is not judged here. (LLM, conf=0.85)
- Could not verifycites Stahlberg & Byrne, 2019 sourceExperimental — accuracy not yet measured
The passages available from our copy are bibliographic records without paper sections or page text. Searches for exact inference, repetition, genericness, and beam-search errors returned no passage addressing the reported issues, so the attribution cannot be verified from our copy. (LLM, conf=0.72)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018;Holtzman et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Not checked — no full text of the sourcecites Fan et al., 2018 source
- Supportedcites Holtzman et al., 2018 source
The Background (§2; page unavailable) says RNN language-model probabilities often favor repetitive, overly generic sentences. The Baselines (§4.2) use beam-search decoding, and the Quantitative Results (§5.1) report that the paper’s alternative decoding approach produces higher-quality text than those baselines in human evaluation. This supports the citing sentence’s point about maximization-based decoding in open-ended generation. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Since finding the optimum argmax sequence from recurrent neural language models or Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;Shen et al., 2017;Wiseman et al., 2017).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Maximization-based decoding
What the sources say
- Partly supportedcites Chen et al., 2018 sourceExperimental — accuracy not yet measured
Section 4 (page unavailable) states that the best-string problem for RNNs is undecidable; Section 7 adds that the polynomial-length version for consistent RNNs is NP-complete. This supports the claim for the RNNs studied, but the paper does not establish the same result for Transformers. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Further, all stochastic methods face repetition issues when their tuning parameters are set too low, which tends to overtruncate, mimicking greedy search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Vocabulary usage and Self-BLEU (Zhu et al., 2018) statistics reveal that high values of k are needed to make top-k sampling match human statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Zhu et al., 2018 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling -sampling directly from the probabilities predicted by the model -results in text that is incoherent and almost unrelated to the context.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even with our relatively fine-grained parameter sweep, Nucleus Sampling obtains closest perplexity to human text, as shown in Table 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 14
- It may state two separate claims as one. 25
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2019;Holtzman et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Partly supportedcites Kulikov et al., 2019 sourceExperimental — accuracy not yet measured
In “Diversity: distinct-n” (page unavailable), the paper reports that greedy search produces less diverse responses than the other strategies; “2.2 Inference (generation)” also describes greedy search as choosing the most likely token at each step. The copy does not establish that these outputs have high grammaticality, so that part of the claim is unsupported. (LLM, conf=0.87)
- Partly supportedcites Holtzman et al., 2018 sourceExperimental — accuracy not yet measured
The Background (§2; page unavailable) says RNN language models often assign higher probability to repetitive, overly generic sentences, supporting the low-diversity disadvantage of probability-based generation. The Qualitative Analysis (§5.2; page unavailable) likewise describes generic output from a language-model baseline. Our copy does not establish that such output has high grammaticality. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites Fan et al., 2018 source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018;Hashimoto et al., 2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Supportedcites Caccia et al., 2018 source
The Introduction (p. 2) explicitly states that lower softmax temperatures produce less diverse, higher-quality samples. The synthetic-data experiment (p. 6) illustrates the trade-off: reducing temperature improves the quality metric, while an extremely low temperature can cause repeated output of a single sequence. (LLM, conf=0.95)
- Not checked — no full text of the sourcecites Hashimoto et al., 2019 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018;Tevet et al., 2019;Semeniuta et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
What the sources say
- Supportedcites Caccia et al., 2018 source
The Empirical Results (p. 5) report that MLE-trained language models outperformed the GAN text generators tested when quality and diversity were assessed together. The Discussion (p. 9) confirms this finding for the GAN variants and tasks considered. (LLM, conf=0.94)
- Could not verifycites Tevet et al., 2019 sourceExperimental — accuracy not yet measured
The available indexed passages contain bibliographic records and an abstract, but no passage from the paper’s full text that can establish the claimed joint assessment of quality and diversity. Searches for that assessment and for evaluation terms returned no body-text passage, so the claim cannot be verified under the full-text-only evidence rule. (LLM, conf=0.72)
- Supportedcites Semeniuta et al., 2018 source
The Abstract (p. 1) says the authors used metrics intended to capture both quality and diversity and found that none of the GAN models they considered performed convincingly better than a conventional language model. Section 5.2 (p. 9) likewise states that no GAN model in their comparison was convincingly better. The finding is limited to the models they tested. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018;2019).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.1 EXPERIMENTAL SETUP
What the sources say
- Supportedcites Vaswani et al., 2017 source
Vaswani et al. is cited immediately after “the Transformer architecture,” identifying the architecture introduced in this paper. The Model Architecture section (p. 2) describes the Transformer; the separate claim about success in extremely large-scale training is cited to Radford et al., not to Vaswani et al. (LLM, conf=0.94)
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [2019]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“On the other hand, pure sampling produces text that is significantly less likely than the gold, corresponding to lower generation quality.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.3 REPETITION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in §5 and illustrated in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Sampling, Top-k, and Nucleus can all be calibrated to human perplexities, but the first two face coherency issues when their parameters are set this high.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The probability of a repeated phrase increases with each repetition, creating a positive feedback loop.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated on the left of Figure 5, in some contexts the head of the next word distribution can be flat across tens or hundreds of reasonable options (e.g.nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the figure.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that the scaling factor p can vary wildly at each time-step, in contrast to Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in §5.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.3 NATURAL LANGUAGE DOES NOT MAXIMIZE PROBABILITY
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration -output text that is bland, incoherent, or gets stuck in repetitive loops.”
- Also stated in: § 1 INTRODUCTION · boosted
“In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large, as shown in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · boosted
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The latter reveals that text generated by maximization or top-k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( §4).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 5.1 ZIPF DISTRIBUTION ANALYSIS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling can easily match reference perplexity through tuning the value of p, avoiding the incoherence caused by setting k high enough to match distributional statistics.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, top-k sampling immediately veers off into an unrelated event.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“pure sampling also has higher perplexity than human text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“the current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The generation produced by Nucleus Sampling isn't perfect -the model appears to confuse whales with birds, and begins writing about those instead.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
boostedcorrelationalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“pure sampling is the closest to the human distribution, followed by Nucleus Sampling.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note the increased variance that characterizes human text, in contrast with the endless repetition of text decoded by Beam Search.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although beam search becomes less repetitive according to this metric as beam width increases, this is largely because average length gets shorter as b increases”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Initial exploration of applying HUSE directly led to top-k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Setting t ∈ [0, 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The presence of flat distributions makes the use of a small k in top-k sampling problematic, while the presence of peaked distributions makes large k's problematic.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.1 NUCLEUS SAMPLING
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Nucleus Sampling and top-k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Choosing where to truncate can be interpreted as determining the generative model's trustworthy prediction zone.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Low temperature sampling has also been used to partially alleviate the issues of top-k sampling discussed above, by shaping the distribution before top-k sampling (Radford et al., 2018;Fan et al., 2018).”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.3 SAMPLING WITH TEMPERATURE
What the sources say
- Not matchedcites Radford et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Fan et al., 2018 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This indicates that the model is confusing itself: sampling too many unlikely tokens and creating context that makes it difficult to recover the human distribution of text, as in Figure 1.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.2 PERPLEXITY
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top-k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 3.2 TOP-k SAMPLING
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human evaluation is still required to get a full measure of the quality of the generated text.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“likelihood maximizing decoding causes repetition and overly generic language usage”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 7 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Statistical evaluations are unable to measure the coherence of generated text properly.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 6.1 HUMAN UNIFIED WITH STATISTICAL EVALUATION (HUSE)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The authors state this as an assumption.
Models compute the probability of a token sequence using the common left-to-right decomposition of text probability.
“We assume that models compute P (x 1:m+n ) using the common left-to-right decomposition of the text probability,”
The authors state this as an assumption.
The model assigns higher probability to higher quality text.
“Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest”
The paper relies on this without support.
The distribution of human-written text is the reference distribution for desirable generation.
“we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.”
The paper relies on this without support.
Repetition is an attribute of text quality, so less repetition is better.
“One attribute of text quality that we can quantify is repetition.”
The paper relies on this without support.
Zipf's law applies to natural language text.
“Zipf's law suggests that there is an exponential relationship between the rank of a word and its frequency in text.”
The paper relies on this without support.
Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.
“Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements.”
The paper relies on this without support.
Values of k greater than 100 are rarely used in practice.
“Values of k greater than 100 are rarely used in practice”
The paper relies on this without support.
Values of p are usually in [0.9, 1).
“values of p are usually in [0.9, 1)”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The issue was particularly pronounced in domain-specific medical models, where the complexity of terminology and extended reasoning chains exacerbate degeneration effects [62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Text degeneration encompasses neural generation challenges such as repetition loops, semantic incoherence, and topic drift, often arising from exposure bias where early errors compound during sequential prediction [56,62].”
From Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework (2026) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“This may increase the likelihood of repetition, another known issue of modern LLMs (Holtzman et al., 2019).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2024) · cites this paper as Holtzman et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“The consequence of unrelated prefixes generating nonsensical outcomes underscores recent proposals like Top-k sampling (Fan et al. 2018) and Nucleus sampling (Holtzman et al. 2019), both employing truncated language model distributions to bias sampling towards the most probable tokens.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Holtzman et al. 2019
The cited source says what the claim attributes to it.
In the citing paper
“However, this is known to create repetitions of phrases or sentences even for large well-trained models (Radford et al., 2019; Holtzman et al., 2019).”
From CTRL: A Conditional Transformer Language Model for Controllable Generation (2019) · cites this paper as Holtzman et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > Abstract
“When b ≥ 64, both GPT-2 Large and XL (774M and 1542M parameters, respectively) prefer to stop generating immediately after the given context.”
§ THE CURIOUS CASE OF NEURAL TEXT DeGENERATION > 4.1 EXPERIMENTAL SETUP
“We perform experiments using the Large model (762M parameters).”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
(2016) · Cited by 11,679
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,422
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects
Ni, J., Li, J., & McAuley, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1018 · Cited by 1,400
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952
“Recent work in the literature has focused on controlling natural language generation using traditional approaches from machine learning like novel architectures which condition outputs [15, 16], more advanced sampling techniques [6, 11], gradient-based”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Self-citationSelf-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
“…to sampling strategies and parameters, by varying T in temperature sampling (Ackley et al., 1985; Ficler & Goldberg, 2017), k in top-k sampling (Fan et al., 2018; Holtzman et al., 2018; Radford et al., 2019), and p in nucleus sampling (Holtzman et al., 2020), over PaLM-540B in Figure 4 (left).”
BackgroundMethodInfluentialCTRL: A Conditional Transformer Language Model for Controllable Generation
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.05858 · Cited by 751
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764
“Instead, methods that focus on the most likely utterances – while introducing a small amount of variability (e.g. beam search and nucleus sampling) – have been found to produce better results in practice [78]).”
BackgroundConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 701
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161317 · Cited by 613
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531
“(2022) and use nucleus sampling (Holtzman et al., 2020) ( p = 0 .”
BackgroundQLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
“All generations in this section were generated with Nucleus Sampling [25] with p = 0 .”
MethodSelf-citationUnleashing the potential of prompt engineering for large language models
CHEN, B. ‐., Zhang, Z., Langrené, N., & Zhu, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101260 · Cited by 1,097
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
Autoregressive Visual Tracking
Wei, X., Bai, Y., Zheng, Y., Shi, D., & Gong, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00935 · Cited by 378
A Simple Language Model for Task-Oriented DialoguePreprint
Hosseini-Asl, E., McCann, B., Wu, C.-S., Yavuz, S., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00796 · Cited by 398
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Data Augmentation using Pre-trained Transformer Models
Kumar, V., Choudhary, A., & Cho, E. (2020) · doi:10.18653/v1/2020.lifelongnlp-1.3 · Cited by 402
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
Budzianowski, P., & Vulić, I. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5602 · Cited by 339
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262
“4 This bias is inspired by LLMs’ sensitivity to repeated patterns (McKenzie et al., 2023; Holtzman et al., 2020).”
BackgroundA taxonomy of prompt modifiers for text-to-image generation
Oppenlaender, J. (2023) · Behaviour and Information Technology · doi:10.1080/0144929x.2023.2286532 · Cited by 385
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282
Bilingual language model for protein sequence and structure
Heinzinger, M., Weissenow, K., Sanchez, J. G., Henkel, A., Mirdita, M., Steinegger, M., & Rost, B. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae150 · Cited by 311
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
Guan, J., Huang, F., Zhao, Z., Zhu, X., & Huang, M. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00302 · Cited by 243
Towards Emotional Support Dialog Systems
Liu, S., Zheng, C., DeMasi, O., Sabour, S., Yu, L., Yu, Z., Jiang, Y., & Huang, M. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.269 · Cited by 286
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“The rationale for incorporating randomness into decoding strategies stems from the realization that high likelihood sequences often result in surprisingly low-quality text, which is called likelihood trap (Stahlberg and Byrne, 2019; Holtzman et al., 2020; Meister et al., 2020; Zhang et al., 2021).”
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
“Beyond their known proficiency in generating human-like text [48], LLMs, in the realm of software engineering, are poised to offer code suggestions, assist in automated documentation, aid in requirement elicitation, and more.”
BackgroundEnabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 239
Self-citationCompressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 229
Self-citationMitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., & Bing, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01316 · Cited by 199
Budennyy, S., Lazarev, V. D., Zakharenko, N. N., Korovin, A. N., Plosskaya, O. A., Dimitrov, D. V., Akhripkin, V. S., Pavlov, I. V., Oseledets, I. V., Barsola, I. S., et al. (2022) · Doklady Mathematics · doi:10.1134/s1064562422060230 · Cited by 249
Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation
Liu, M., Pinckney, N., Khailany, B., & Ren, H. M. (2023) · doi:10.1109/iccad57390.2023.10323812 · Cited by 346
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187
“For all three inference methods, we obtain generations from the model using Fairseq’s implementation [47] of top-p (nucleus) sampling [33] with p = 0.”
MethodUnsupervised Commonsense Question Answering with Self-Talk
Shwartz, V., West, P., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.373 · Cited by 215
Self-citationA Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196
Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs
Hsiao, W.-Y., Liu, J.-Y., Yeh, Y.-C., & Yang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16091 · Cited by 199
IgLM: Infilling language modeling for antibody sequence design
Shuai, R. W., Ruffolo, J. A., & Gray, J. J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.001 · Cited by 191
Applying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
West, P., Bhagavatula, C., Hessel, J., Hwang, J. D., Jiang, L., Bras, R. J. L., Lu, X., Welleck, S., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.341 · Cited by 209
Self-citationSparks: Inspiration for Science Writing using Language Models
Gero, K. I., Liu, V., & Chilton, L. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533533 · Cited by 161
End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2
Ham, D., Lee, J.-G., Jang, Y., & Kim, K.-E. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.54 · Cited by 161
Learning Video Representations from Large Language Models
Zhao, Y., Misra, I., Krähenbühl, P., & Girdhar, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00637 · Cited by 190
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.522 · Cited by 320
Self-citationVisual Adversarial Examples Jailbreak Aligned Large Language Models
Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., & Mittal, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30150 · Cited by 150
Toxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.64 · Cited by 149
Task-Oriented Dialog Systems That Consider Multiple Appropriate Responses under the Same Context
Zhang, Y., Ou, Z., & Yu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6507 · Cited by 154
LayoutTransformer: Layout Generation and Completion with Self-attention
Gupta, K., Lazarow, J., Achille, A., Davis, L. S., Mahadevan, V., & Shrivastava, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00104 · Cited by 164
Huang, Q., Dong, X., Zhang, P., Wang, B., He, C., Wang, J., Lin, D., Zhang, W., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01274 · Cited by 270
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356
Wang, Y., Wang, L., & Siau, K. L. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2316376 · Cited by 163
ShapeFormer: Transformer-based Shape Completion via Sparse Representation
Yan, X., Lin, L., Mitra, N. J., Lischinski, D., Cohen‐Or, D., & Huang, H. F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00614 · Cited by 156
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
Cao, S., & Wang, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.532 · Cited by 146
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
Xu, C., Guo, D., Duan, N., & McAuley, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.385 · Cited by 348
PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking
Rashkin, H., Çelikyılmaz, A., Choi, Y., & Gao, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.349 · Cited by 160
Self-citationDon’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Li, M., Roller, S., Kulikov, I., Welleck, S., Boureau, Y.-L., Cho, K., & Weston, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.428 · Cited by 151
Exploring and Unleashing the Power of Large Language Models in Automated Code Translation
Yang, Z., Liu, F., Yu, Z., Keung, J. W., Li, J., Liu, S., Hong, Y., Ma, X., Jin, Z., & Li, G. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660778 · Cited by 104
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2023) · Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2023.conll-1.21 · Cited by 152
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131
The Science of Detecting LLM-Generated Text
Tang, R., Chuang, Y.-N., & Hu, X. (2024) · Communications of the ACM · doi:10.1145/3624725 · Cited by 125
Pix2seq: A Language Modeling Framework for Object DetectionPreprint
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 272
Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs
Zhang, H., Liu, Z., Xiong, C., & Liu, Z. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.184 · Cited by 137
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 178
Self-citationOn the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121
INSPIRED: Toward Sociable Recommendation Dialog Systems
Hayati, S. A., Kang, D., Zhu, Q., Shi, W., & Yu, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.654 · Cited by 159
Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP
Esmaeilpour, S., Liu, B., Robertson, E., & Shu, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20610 · Cited by 134
End-to-end generation of multiple-choice questions using text-to-text transfer transformer models
Rodriguez-Torrealba, R., Garcia-Lopez, E., & Garcia-Cabot, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118258 · Cited by 118
A Recipe for Arbitrary Text Style Transfer with Large Language Models
Reif, E., Ippolito, D., Yuan, A., Coenen, A., Callison-Burch, C., & Wei, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.94 · Cited by 123
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113
“To increase the diversity of the generated questions, we tried sampling-based methods (Fan et al., 2018; Holtzman et al., 2020), but obtained low-quality questions.”
MethodResultGenerative Data Augmentation for Commonsense Reasoning
Yang, Y., Malaviya, C., Fernandez, J., Swayamdipta, S., Bras, R. J. L., Wang, J., Bhagavatula, C., Choi, Y., & Downey, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.90 · Cited by 156
Self-citationUnnatural Instructions: Tuning Language Models with (Almost) No Human Labor
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.806 · Cited by 324
Contrastive Decoding: Open-ended Text Generation as Optimization
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J. M., Hashimoto, T., Zettlemoyer, L., & Lewis, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.687 · Cited by 408
Self-citationZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, C., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.801 · Cited by 200
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 394
The limitations of stylometry for detecting machine-generated fake news
Schuster, T., Schuster, R., Shah, D. J., & Barzilay, R. (2020) · Computational Linguistics · doi:10.1162/coli_a_00380 · Cited by 109
Assessment of chemistry knowledge in large language models that generate code
White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. (2023) · Digital Discovery · doi:10.1039/d2dd00087c · Cited by 130
Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106
CPM: A large-scale generative Chinese Pre-trained language model
Zhang, Z., Han, X., Zhou, H., Ke, P., Gu, Y., Ye, D., Qin, Y., Su, Y., Ji, H., Guan, J., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.07.001 · Cited by 119
MotionLM: Multi-Agent Motion Forecasting as Language Modeling
Seff, A., Cera, B., Chen, D., Ng, M., Zhou, A., Nayakanti, N., Refaat, K. S., Al‐Rfou, R., & Sapp, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00788 · Cited by 128
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
SceneGen: Learning to Generate Realistic Traffic Scenes
Tan, S., Wong, K., Wang, S., Manivasagam, S., Ren, M., & Urtasun, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00095 · Cited by 104
“Therefore, we adopt a sampling strategy inspired by nucleus sampling [23].”
BackgroundMethodKdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation
Zhou, H., Zheng, C., Huang, K., Huang, M., & Zhu, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.635 · Cited by 106
Trojaning Language Models for Fun and Profit
Zhang, X., Zhang, Z., Ji, S., & Wang, T. (2021) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp51992.2021.00022 · Cited by 117
A Large-Scale Chinese Short-Text Conversation Dataset
Wang, Y., Ke, P., Zheng, Y., Huang, K., Jiang, Y., Zhu, X., & Huang, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_8 · Cited by 105
Materials science in the era of large language models: a perspective
Lei, G., Docherty, R., & Cooper, S. J. (2024) · Digital Discovery · doi:10.1039/d4dd00074a · Cited by 130
Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding
Abdelnabi, S., & Fritz, M. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00083 · Cited by 121
Pretrained Transformers for Text Ranking: BERT and BeyondPreprint
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101
Metadata
- Authors
- Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y.
- Affiliations
- University of Washington (US)
- University of Cape Town (ZA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1904.09751
- OpenAlex
- W2938704169
- Semantic Scholar
- cf4aa38ae31b43fd07abe13b4ffdb265babb7be1
- Cited by
- 1,304
- References
- 39
- Citations by year
- 2026: 42 · 2025: 119 · 2024: 172 · 2023: 368 · 2022: 242 · 2021: 213 · 2020: 123 · 2019: 25
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1256082, 1524371)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Army Research Office
- Naval Information Warfare Center Pacific
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- text generation, nucleus sampling, decoding strategies, text diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
