The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Deduplicating Training Data Makes Language Models Better
Open accessAbstract
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Data deduplication has become a key tool for language model projects following research showing that it both improves performance on downstream tasks (Lee et al., 2022; Zhang et al., 2021) and decreases memorization of training data (Kandpal et al., 2022).”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Lee et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To ensure reliable evaluation and improve performance [26], we deduplicate documents by title and utilize the predefined training and validation splits.”
From Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval (2025) · cites this paper as [26]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, the probability of reproducing training data has been shown to correlate with its frequency of appearance [12].”
From Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy (2025) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“Deduplication is a powerful strategy for improving dataset quality (Li et al., 2024; Lee et al., 2022; Penedo et al., 2024b; Fang et al., 2025; Liu et al., 2024).”
From OpenThoughts: Data Recipes for Reasoning Models (2025) · cites this paper as Lee et al., 2022
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Moreover, it is provided only on the complete text generated by the model (trajectory) and not on specific parts of the text (particular state) [92,138].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [92]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Space/time trade-offs in hash coding with allowable errors
Bloom, B. (1970) · CACM · doi:10.1145/362686.362692 · Cited by 8,424
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 4×
“And so because these datasets are up to 19% smaller, even including the deduplication runtime itself, training on dedupli-cated datasets directly reduces the training cost in terms of time, dollar, and the environment (Bender et al., 2021; Strubell et al., 2019; Patterson et al., 2021).”
BackgroundInfluentialMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE. 1
Jaccard, P. (1912) · New Phytologist · doi:10.1111/j.1469-8137.1912.tb05611.x · Cited by 4,561
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Research works printed but not linked: 7
- PMLR.
- David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34.
- Paul Jaccard. 1912. The distribution of the flora in the alpine zone. New phytologist, 11(2):37-50. 10 8433
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
- PMLR.
- Yannick Versley and Yana Panchenko. 2012. Not just bigger: Towards better-quality web corpora. In Pro- ceedings of the seventh Web as Corpus Workshop (WAC7), pages 44-52.
Cited by
Sorted by citations
Items 301–344 of 344
Chahid, Y., Chahid, A., Chahid, I., & Elmiad, A. K. (2026) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2026.0170604
GHP: Hardening Graph‐Based Reranking Defense for Poisoned Retrieval‐Augmented Generation
She, W., Dong, Z., Yang, Y. W., Tian, Z., Liu, W., & Kong, D. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70824
Kulkarni, P. P., & Alam, A. (2026) · Workshop on Quantum Computing and Quantum-Inspired Technology for Data-Intensive Systems and Applications (Q-Data) · doi:10.1145/3811628.3811832
Drivers of Secure and Correct Code: A Factorial Study of Size, Pre-Training, and Data Quality
Liguori, P., Krasniqi, R., & Cotroneo, D. (2026) · International Conference on Dependable Systems and Networks workshops · doi:10.1109/dsn-w70714.2026.00031
Re-KGR: Mitigating Factual Hallucinations via Hybrid Graph Reasoning and Semantic Refinement
Li, Y. Z., Wang, X., Liu, R., Wang, M. Y., & Luo, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2494-4_16
Pelloin, V., Bekkali, L., Dehak, R., & Doukhan, D. (2026) · Journées d'Études sur la Parole (JEP) · doi:10.21437/jep.2026-28
LLM-Oriented Information Retrieval: A Denoising-First Perspective
Dai, L., Sun, L., Cao, F., Rao, Z., Yang, C., Liu, H., & Xiong, H. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808544 · Cited by 2
Data Governance for Generative AI
Armand, K. W. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0368-8.ch006
Capability Degradation in Small Language Models Under Low-Signal Data Corruption
Qamar, R., Appasani, B., Bizon, N., Verma, S., & Jha, A. V. (2026) · International Conference on Electronics, Computers and Artificial Intelligence (ECAI) · doi:10.1109/ecai69016.2026.11613657
SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication
Fang, X., Tong, K., Liu, J., Ning, T., & He, Y. (2026) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1109/caibda70336.2026.11621091
Memming, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10486789/v1
Kiet, D. L. A., Anh, N. T., Thanh, T. T., Bao, V. N. Q., & Tu, N. H. (2026) · IEEE Access · doi:10.1109/access.2026.3718998
Kowalik, M., Poruba, M., Grzymkowski, Ł., & Stefański, T. P. (2026) · IEEE Access · doi:10.1109/access.2026.3720484
CodeCAD: A Parametric CAD Dataset for Programmatic 3D Model Generation
Fresacher, D., & Diepold, K. J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817592
AzText: Curating Web-Scale Pretraining Data for a Low-Resource Language
Mahammadli, E., & Rustamov, S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31319-5_23
DM-RAG: Enhancing User Support in Dameng Databases with Retrieval-Augmented Generation
Huang, Q., Liu, K., Deng, L., Zhang, S., Hu, C., Qian, T., Yan, X., & Jiang, J. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00304
Efficient Banzhaf-Based Data Valuation for k -Nearest Neighbors Classification
Zhang, G., Oettershagen, L., Wang, L., & Gionis, A. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3819518.3819521
Large Language Models and Their Impact on English Teaching Practices
Mutalliyevich, D. A. (2023) · International Journal of English and Education · doi:10.67050/ijee/v12i3/ijee233014
Not All Secrets Are Equal: Type-Aware Unlearning for Language Model Secret Removal
Fakharzadehjahromy, H., Sow, A., Bueff, A., Heintz, F., Geng, J., & Tiger, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37673-2_15
Software Engineering for Foundation Models (SE4FM)
Chen, B., Lin, D., Leung, A., Chen, Z., Rajbahadur, G. K., Oliva, G. A., Chen, Y., Liu, X., Chong, C. Y., & Hassan, A. E. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3776600
Gallaba, K., Fan, Z., Lin, J., Cogo, F. R., Rombaut, B., Lin, D., & Hassan, A. E. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3776599
Biloshchytska, S., Kuchanskyi, O., Andrashko, Y., Amangeldiyev, Z., Kantayeva, D., & Tovt-Kuchanska, M. (2026) · Information · doi:10.3390/info17090897
Inappropriate Content Classification Model for Digital Violence Detection Using Hybrid Data
Zambrano, P., Sánchez, M., Anchundia, C. E., Illicachi, J., Calahorrano, A. D. O., Montenegro, A. E. P., & Espinosa, J. S. L. (2026) · Informatics · doi:10.3390/informatics13090151
Data and semantic reasoning in audio-language learning
Adriaan, W. G. W. (2026) · Research Publications (Maastricht University) · doi:10.26481/dis.20260907gw
What a nano-GPT can (not) tell us about spoken language
Pace-Sigge, M. T., & Mailoa, E. (2026) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.24172.pac
Almjbry, M. A., & Almagrhe, I. M. (2026) · Al-Farooq Journal of Sciences · doi:10.65405/9pvbat43
Representation choice and training-boundary composition in document stream segmentation
Ulybin, D. (2026) · American Impact Review · doi:10.66308/air.e2026086
Abdullahi, R. I., Obunadike, G. N., Abdullahi, A., & Irhebhude, M. E. (2026) · FUDMA Journal of Sciences · doi:10.33003/fjs-2026-1020-6074
Exploring Hybrid Pre-training for Automatic Essay Scoring
Carpi, M. M., Silveira, I. C., Mauá, D. D., & Finger, M. (2026) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2026.26589
Horcas, J.-M., Ballesteros, J., Pinto, M., & Fuentes, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40763-4_6
Nurislom, A. (2026) · Eurasian Journal of Mathematical Theory and Computer Sciences · doi:10.70413/ejmtcs-v06-i10-s92242
Shen, Y., Liu, Y., Dong, C., Pal, S., Yang, Y., & Shen, J. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-026-00277-5
Niu, Z. (2026) · Advances in Economics Management and Political Sciences · doi:10.54254/2754-1169/2026.ld38022
Chen, L., Wang, Z., Li, Z., Li, W., Zhu, H., & Han, D. (2026) · Applied Sciences · doi:10.3390/app162010430
Cheverkalov, V. А., Basinya, E. A., Karapetyants, N., & Ivanenko, V. G. (2026) · Bezopasnost informacionnyh tehnology · doi:10.26583/bit.2026.5.05
Yeşilkaya, Ü. B., & Yeşilkaya, İ. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.09.241
ConRAC-SE: Evidence-Agreement Routing for Cost-Aware Retrieval-Augmented Security Classification
Chang, Y., Lim, W., Kailasa, R., Shim, S. S. Y., & Lee, J. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840065
Rethinking Graph-LLMs: Are Performance Gains Confounded by Data Leakage?
Ye, Y., Qin, T., Guo, X., Wang, W., Peng, Q., & Li, T. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840804
Yeşilkaya, Ü. B., & Yeşilkaya, İ. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.09.240
Non-Identifiability of Reasoning Trajectories in Deterministic Language Model Decoding
Deepthi, K., HEGDE, S. S., & Shastry, A. (2026) · doi:10.1109/nmitcon70171.2026.11754496
Beyond behavioural markers: causal-representational corroboration in AI sentience assessment
Li, Y. (2026) · AI and Ethics · doi:10.1007/s43681-026-01495-4
PowerBIO-NER: Audit-First Evaluation and BIO-Constrained Decoding for Power Engineering NER
Shi, W., Song, N., & Zhang, X. (2026) · International Conference on Energy Materials and Electrical Engineering (ICEMEE) · doi:10.1109/icemee71303.2026.11748103
The More You Tune, the More They Spill?
Salerno, F., Al-Kaswan, A., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10966-5
Metadata
- Authors
- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N.
- Affiliations
- Google (United States) (US)
- University of Pennsylvania (US)
- Corresponding author
- Ippolito, Daphne
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 8424-8445
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3177765786
- Cited by
- 304
- References
- 50
- Citations by year
- 2026: 62 · 2025: 79 · 2024: 65 · 2023: 64 · 2022: 27
- FWCI
- 32.49
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- language model training, training data quality, duplicate removal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
