Deduplicating Training Data Makes Language Models Better
Open accessAbstract
We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. We release code for reproducing our work and performing dataset deduplication at https://github.com/google-research/deduplicate-text-datasets.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Repeated data is worth less, thus models trained on less unique data (and correspondingly more epochs) have consistently higher loss.
“Conforming to intuition and prior work on deduplication (Lee et al., 2021), repeated data is worth less, thus models trained on less unique data (and, correspondingly, more epochs) have consistently higher loss.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Lee et al., 2021
The paper’s summary of deduplication results says models trained on deduplicated data have no worse perplexity than models trained on the original data and sometimes improve it, supporting the general point that repeated data can be less useful. Section 6.1 (p. 6) does not establish that models trained on less unique data for more epochs consistently have higher loss: it reports similar perplexity on some validation sets and differences that depend on the evaluation set. (LLM, conf=0.82)
Experimental — accuracy not yet measuredPrior work found deduplication was able to improve perplexity, but it did not evaluate on downstream tasks.
“Prior work found deduplication was able to improve perplexity (Lee et al., 2021); however, it did not evaluate on downstream tasks.”
From Scaling Data-Constrained Language Models (2023) · cites this paper as Lee et al., 2021
When encoders learn wrong correlations between different parts of the training data, it could result in erroneous generation that diverges from the input.
“When encoders learn wrong correlations between different parts of the training data, it could result in erroneous generation that diverges from the input [6, 71, 142, 251].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [71]
Section 6.2 (Generated Text) says duplicated training data can bias language-model generations toward copied content. It does not address encoders learning wrong correlations between parts of the training data or generation diverging from its input; searches for those mechanisms found no supporting passage. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
Cited by 11
This paper’s reference list could not be split into single entries.
- Authors
- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N.
- Affiliations
- Google (United States) (US)
- University of Pennsylvania (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2107.06499
- OpenAlex
- W4287077924
- Cited by
- 11
- References
- 0
- Citations by year
- 2026: 5 · 2025: 5 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- training data quality, training efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
