Deduplicating Training Data Makes Language Models Better
Open accessAbstract
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, Nicholas Carlini. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deduplication improves performance on downstream tasks and decreases memorization of training data.
“Data deduplication has become a key tool for language model projects following research showing that it both improves performance on downstream tasks (Lee et al., 2022; Zhang et al., 2021) and decreases memorization of training data (Kandpal et al., 2022).”
From The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023) · cites this paper as Lee et al., 2022
The Introduction and Conclusion report that deduplication sometimes improves language-model perplexity and helps models reach accuracy targets faster. But the paper’s “Contamination of downstream tasks” section explicitly says it does not study effects on downstream benchmark tasks, so Lee et al. do not establish that part of the citing sentence. The memorization claim is attached to Kandpal et al., not Lee et al. (LLM, conf=0.88)
Experimental — accuracy not yet measuredDocuments are deduplicated by title and predefined training and validation splits are used to ensure reliable evaluation.
“To ensure reliable evaluation and improve performance [26], we deduplicate documents by title and utilize the predefined training and validation splits.”
From Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval (2025) · cites this paper as [26]
The Introduction and §5.3 explain that deduplication can improve model performance and prevent train–validation overlap from inflating evaluation metrics. The paper describes deduplication by substring matching and document similarity, not by title; §5 discusses retaining validation or test copies when duplicates cross splits, but does not establish the claimed use of predefined splits. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe probability of reproducing training data correlates with its frequency of appearance, and information repeated multiple times during pretraining is more likely to be memorized and reproduced verbatim.
“For example, the probability of reproducing training data has been shown to correlate with its frequency of appearance [12].”
From Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy (2025) · cites this paper as [18]
Deduplication is a powerful strategy for improving dataset quality.
“Deduplication is a powerful strategy for improving dataset quality (Li et al., 2024; Lee et al., 2022; Penedo et al., 2024b; Fang et al., 2025; Liu et al., 2024).”
From OpenThoughts: Data Recipes for Reasoning Models (2025) · cites this paper as Lee et al., 2022
Preference-based feedback is provided only on the complete text generated by the model (trajectory) and not on specific parts of the text (particular state).
“Moreover, it is provided only on the complete text generated by the model (trajectory) and not on specific parts of the text (particular state) [92,138].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [92]
Section 6.2, “Generated Text” (page unavailable), examines whether generated text reproduces training data; it does not discuss preference-based feedback. Searches for feedback on complete trajectories versus particular states found no passage addressing that distinction. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Space/time trade-offs in hash coding with allowable errors
Bloom, B. (1970) · CACM · doi:10.1145/362686.362692 · Cited by 8,424 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 4× · see citing works
“And so because these datasets are up to 19% smaller, even including the deduplication runtime itself, training on dedupli-cated datasets directly reduces the training cost in terms of time, dollar, and the environment (Bender et al., 2021; Strubell et al., 2019; Patterson et al., 2021).”
BackgroundInfluentialMembership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE. <sup>1</sup>
Jaccard, P. (1912) · New Phytologist · doi:10.1111/j.1469-8137.1912.tb05611.x · Cited by 4,561 · see citing works
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Suffix Arrays: A New Method for On-Line String Searches
Manber, U., & Myers, E. W. (1993) · SIAM Journal on Computing · doi:10.1137/0222058 · Cited by 1,915 · see citing works
Linear pattern matching algorithms
Weiner, P. D. (1973) · Annual Symposium on Switching & Automata Theory · doi:10.1109/swat.1973.13 · Cited by 1,829 · see citing works
On the resemblance and containment of documents
Broder, A. (2002) · Compression and Complexity of Sequences (SEQUENCES) · doi:10.1109/sequen.1997.666900 · Cited by 1,734 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · see citing works
Adafactor: Adaptive Learning Rates with Sublinear Memory CostPreprint
Shazeer, N., & Stern, M. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1804.04235 · Cited by 1,410 · see citing works
A Closer Look at Memorization in Deep Networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1706.05394 · Cited by 1,307 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · see citing works
mT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 852 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Self-citationUniversal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642 · see citing works
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627 · see citing works
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569 · see citing works
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
Carbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486 · see citing works
Simple Linear Work Suffix Array Construction
Kärkkäinen, J., & Sanders, P. W. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45061-0_73 · Cited by 404 · see citing works
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
A Simple Method for Commonsense Reasoning
Trinh, T. H., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1806.02847 · Cited by 293 · see citing works
The adverse effects of code duplication in machine learning models of code
Allamanis, M. (2019) · Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3359591.3359735 · Cited by 284 · see citing works
Space Efficient Linear Time Construction of Suffix Arrays
Ko, P., & Aluru, S. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44888-8_15 · Cited by 225 · see citing works
Using Suffix Arrays to Compute Term Frequency and Document Frequency for All Substrings in a Corpus
Yamamoto, M., & Church, K. (2001) · Computational Linguistics · doi:10.1162/089120101300346787 · Cited by 177 · see citing works
Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., Zhang, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.12369 · Cited by 172 · see citing works
Self-citationWhat Does it Mean for a Language Model to Preserve Privacy?
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., & Tramèr, F. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3534642 · Cited by 165 · see citing works
Self-citationWhat Neural Networks Memorize and Why: Discovering the Long Tail via Influence EstimationPreprint
Feldman, V., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03703 · Cited by 146 · see citing works
Self-citationLinear Suffix Array Construction by Almost Pure Induced-Sorting
Nong, G., Zhang, S., & Chan, W. H. (2009) · DCC · doi:10.1109/dcc.2009.42 · Cited by 143 · see citing works
A new suffix tree similarity measure for document clustering
Chim, H., & Deng, X. (2007) · International World Wide Web Conference (WWW) · doi:10.1145/1242572.1242590 · Cited by 108 · see citing works
Detecting Overfitting of Deep Generative Networks via Latent Recovery
Webster, R., Rabin, J., Simon, L., & Jurie, F. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01153 · Cited by 101 · see citing works
Towards Controllable Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.291 · Cited by 95 · see citing works
Bandy, J., & Vincent, N. (2021) · arXiv · doi:10.48550/arxiv.2105.05241 · Cited by 36 · see citing works
Wiki-40B: Multilingual Language Model Dataset
Guo, M., Dai, Z., Vrandečić, D., & Al‐Rfou, R. (2020) · Language Resources and Evaluation · Cited by 33 · see citing works
Identifying and characterizing highly similar notes in big clinical note datasets
Gabriel, R. A., Kuo, T.-T., McAuley, J., & Hsu, C. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.04.009 · Cited by 31 · see citing works
On the geometry of generalization and memorization in deep neural networksPreprint
Stephenson, C., Padhy, S., Ganesh, A., Hui, Y., Tang, H., & Chung, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14602 · Cited by 27 · see citing works
(2021) · arXiv (Cornell University) · Cited by 21 · see citing works
Connected Components at Scale via Local ContractionsPreprint
Łącki, J., Mirrokni, V., & Włodarczyk, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10727 · Cited by 19 · see citing works
Towards Controllable Biases in Language GenerationPreprint
Sheng, E., Chang, K.-W., Natarajan, P., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00268 · Cited by 11 · see citing works
Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings
Gyawali, B., Anastasiou, L., & Knoth, P. (2020) · Open Research Online (The Open University) · Cited by 10 · see citing works
Understanding Invariance via Feedforward Inversion of Discriminatively Trained ClassifiersPreprint
Teterwak, P., Zhang, C., Krishnan, D., & Mozer, M. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07470 · Cited by 4 · see citing works
Self-citationDocumenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.08758
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., et al. (2013) · arXiv · doi:10.48550/arxiv.1312.3005
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · arXiv · doi:10.48550/arxiv.1908.07125
Flagged first, then by citations
Research works printed but not linked: 7
- PMLR.
- David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34.
- Paul Jaccard. 1912. The distribution of the flora in the alpine zone. New phytologist, 11(2):37-50. 10 8433
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
- PMLR.
- Yannick Versley and Yana Panchenko. 2012. Not just bigger: Towards better-quality web corpora. In Pro- ceedings of the seventh Web as Corpus Workshop (WAC7), pages 44-52.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
“[96] show that duplicated examples from the pretraining corpus bias the model to favor generating repeats of the memorized phrases from the duplicated examples.”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
“However, the inherent memo-rization capability becomes problematic in the context of duplicated information present within pre-training data (Lee et al., 2022a; Kandpal et al., 2023; Paullada et al., 2021).”
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
TimeLMs: Diachronic Language Models from Twitter
Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., & Camacho-collados, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.25 · Cited by 263 · see citing works
Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation
Liu, M., Pinckney, N., Khailany, B., & Ren, H. M. (2023) · doi:10.1109/iccad57390.2023.10323812 · Cited by 253 · see citing works
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 240 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“However, the inherent memo-rization capability becomes problematic in the context of duplicated information present within pre-training data (Lee et al., 2022a; Kandpal et al., 2023; Paullada et al., 2021).”
Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
Liu, F., Liu, Y., Shi, L., Yang, Z., Zhang, L., Lian, X., Li, Z., & Ma, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3657432 · Cited by 188 · see citing works
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H. B., Vassilvitskii, S., Chien, S., & Thakurta, A. G. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14649 · Cited by 166 · see citing works
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y., Laszlo, S., Fleet, D. J., Soricut, R., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01761 · Cited by 166 · see citing works
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
“Issues in the data include poor quality [31], misinformation [35], bias [41, 47], and being outdated [33, 43] knowledge.”
BackgroundResource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 152 · see citing works
Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation
Zheng, B., Hou, Y., Lu, H., Chen, Y., Zhao, W. X., Chen, M., & Wen, J.-R. (2024) · 2024 IEEE 40th International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00118 · Cited by 143 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Knowledge Unlearning for Mitigating Privacy Risks in Language Models
Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., & Seo, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.805 · Cited by 135 · see citing works
Are Large Pre-Trained Language Models Leaking Your Personal Information?
Huang, J., Shao, H., & Chang, K. C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.148 · Cited by 135 · see citing works
Genomic language models: opportunities and challenges
Benegas, G., Ye, C., Albors, C., Li, J. C., & Song, Y. S. (2025) · Trends in Genetics · doi:10.1016/j.tig.2024.11.013 · Cited by 133 · see citing works
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129 · see citing works
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 124 · see citing works
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 124 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 110 · see citing works
Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 110 · see citing works
Foundation Models and Fair UsePreprint
Henderson, P., Li, X., Jurafsky, D., Hashimoto, T., Lemley, M. A., & Liang, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4404340 · Cited by 108 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting
Guan, X., Liu, Y., Lin, H., Lu, Y., He, B., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29770 · Cited by 101 · see citing works
When Large Language Models Meet Vector Databases: A Survey
Jing, Z., Su, Y., & Han, Y. (2025) · 2025 Conference on Artificial Intelligence x Multimedia (AIxMM) · doi:10.1109/aixmm62960.2025.00008 · Cited by 93 · see citing works
Taiyi: a bilingual fine-tuned large language model for diverse biomedical tasks
Luo, L., Ning, J., Zhao, Y., Wang, Z., Ding, Z., Chen, P., Fu, W., Han, Q., Xu, G., Qiu, Y., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae037 · Cited by 89 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Abadji, J., Suárez, P. O., Romary, L., & Sagot, B. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/579or68a4ybs · Cited by 87 · see citing works
Wang, Y., Pan, Y., Su, Z., Deng, Y., Zhao, Q., Du, L., Luan, T. H., Kang, J., & Niyato, D. T. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3576176 · Cited by 86 · see citing works
Will we run out of data? Limits of LLM scaling based on human-generated data
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2022) · arXiv · doi:10.48550/arxiv.2211.04325 · Cited by 80 · see citing works
“For example, data processing techniques like deduplication (Lee et al., 2022) and data filtering (Gao, 2021) have been shown to improve model performance.”
MethodLee, K., Cooper, A. F., & Grimmelmann, J. (2024) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3614407.3643696 · Cited by 83 · see citing works
Self-citationRLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Han, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 72 · see citing works
TelecomGPT: A Framework to Build Telecom-Specific Large Language Models
Zou, H., Zhao, Q., Tian, Y., Bariah, L., Bader, F., Lestable, T., & Debbah, M. A. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3593184 · Cited by 72 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 68 · see citing works
Deep lexical hypothesis: Identifying personality structure in natural language.
Cutler, A., & Condon, D. M. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000443 · Cited by 65 · see citing works
Large language models in psychiatry: Opportunities and challenges
Volkmer, S., Meyer‐Lindenberg, A., & Schwarz, E. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116026 · Cited by 65 · see citing works
Guerreiro, N. M., Voita, E., & Martins, A. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.75 · Cited by 63 · see citing works
Copyright Safety for Generative AIPreprint
Sag, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4438593 · Cited by 62 · see citing works
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03915 · Cited by 61 · see citing works
“Instead, we applied substring deduplication (Lee et al., 2022) based on Suffix Array (Manber and Myers, 1993) as a complementary method that clusters documents sharing a long substring, for documents with more than 6000 characters.”
BackgroundMethodInfluentialmedBERT.de: A comprehensive German BERT model for the medical domain
Bressem, K. K., Papaioannou, J.-M., Grundmann, P., Borchert, F., Adams, L. C., Liu, L., Busch, F., Xu, L., Loyen, J. P., Niehues, S. M., et al. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121598 · Cited by 60 · see citing works
The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
He, F., Zhu, T., Ye, D., Liu, B., Zhou, W., & Yu, P. S. (2026) · ACM Computing Surveys · doi:10.1145/3773080 · Cited by 58 · see citing works
Contextual feature extraction hierarchies converge in large language models and the brain
Mischler, G., Li, Y. A., Bickel, S., Mehta, A. D., & Mesgarani, N. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00925-4 · Cited by 58 · see citing works
Protein language models are biased by unequal sequence sampling across the tree of life
Ding, F., & Steinhardt, J. (2024) · doi:10.1101/2024.03.07.584001 · Cited by 57 · see citing works
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 55 · see citing works
Copyright Violations and Large Language Models
Karamolegkou, A., Li, J., Zhou, L., & Søgaard, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.458 · Cited by 54 · see citing works
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52 · see citing works
Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe
Yue, X., Inan, H. A., Li, X., Kumar, G., McAnallen, J., Shajari, H., Sun, H., Levitan, D., & Sim, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.74 · Cited by 52 · see citing works
Towards Continual Knowledge Learning of Language Models
Jang, J., Ye, S., Yang, S., Shin, J., Han, J., Kim, G., Choi, S. J., & Seo, M. (2021) · arXiv · doi:10.48550/arxiv.2110.03215 · Cited by 49 · see citing works
Deng, C., Duan, Y., Jin, X., Chang, H., Tian, Y., Liu, H., Wang, Y., Gao, K., Zou, H. P., Jin, Y., et al. (2025) · AI and Ethics · doi:10.1007/s43681-025-00797-3 · Cited by 49 · see citing works
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.674 · Cited by 48 · see citing works
Self-citationAn evaluation on large language model outputs: Discourse and memorization
de Wynter, A., Wang, X., Sokolov, A., Gu, Q., & Chen, S. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100024 · Cited by 39 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
KnowLog: Knowledge Enhanced Pre-trained Language Model for Log Understanding
Ma, L., Yang, W., Xu, B., Jiang, S., Fei, B., Liang, J., Zhou, M., & Xiao, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623304 · Cited by 38 · see citing works
“Note Bloat” impacts deep learning-based NLP models for clinical prediction tasks
Liu, J., Daniel Capurro, N., Nguyen, A. N., & Verspoor, K. M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104149 · Cited by 38 · see citing works
Large language models in clinical trials: applications, technical advances, and future directions
Lin, A., Wang, Z., Jiang, A., Chen, L., Qi, C., Zhu, L., Mou, W., Gan, W., Zeng, D., Xiao, M., et al. (2025) · BMC Medicine · doi:10.1186/s12916-025-04348-9 · Cited by 37 · see citing works
Large Language Models are Not Models of Natural Language: They are Corpus Models
Veres, C. (2022) · IEEE Access · doi:10.1109/access.2022.3182505 · Cited by 37 · see citing works
EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training
Gu, Y., Wen, J., Sun, H., Song, Y., Ke, P., Zheng, C., Zhang, Z., Yao, J., Liu, L., Zhu, X., et al. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1387-3 · Cited by 36 · see citing works
Traces of Memorisation in Large Language Models for Code
Al-Kaswan, A., Izadi, M., & Arie Van Deursen, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639133 · Cited by 35 · see citing works
Parmanto, B., Aryoyudanta, B., Soekinto, T. W., Setiawan, I. M. A., Wang, Y., Hu, H., Saptono, A., & Choi, Y. K. (2024) · JMIR Formative Research · doi:10.2196/54633 · Cited by 35 · see citing works
An outline of Prognostics and health management Large Model: Concepts, Paradigms, and challenges
Tao, L., Li, S., Liu, H., Huang, Q., Ma, L., Ning, G., Chen, Y., Wu, Y., Li, B., Zhang, W., et al. (2025) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2025.112683 · Cited by 33 · see citing works
MethylGPT: a foundation model for the DNA methylomePreprint
Ying, K., Song, J., Cui, H., Zhang, Y., Li, S., Chen, X., Liu, H., Eames, A., McCartney, D. L., Marioni, R. E., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.30.621013 · Cited by 33 · see citing works
Privacy preserving large language models: ChatGPT case study based vision and framework
Ullah, I., Hassan, N., Gill, S. S., Suleiman, B., Ahanger, T. A., Shah, Z., Qadir, J., & Kanhere, S. S. (2024) · IET Blockchain · doi:10.1049/blc2.12091 · Cited by 33 · see citing works
Wang, L., & Li, H. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3215244 · Cited by 29 · see citing works
Stammbach, D., Antoniak, M., & Ash, E. (2022) · doi:10.18653/v1/2022.wnu-1.6 · Cited by 29 · see citing works
Preventing generation of verbatim memorization in language models gives a false sense of privacy
Ippolito, D., Tramer, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette Choo, C., & Carlini, N. (2023) · Proceedings of the 16th International Natural Language Generation Conference · doi:10.18653/v1/2023.inlg-main.3 · Cited by 28 · see citing works
Self-citationExeBench: an ML-scale dataset of executable C functions
Armengol-Estapé, J., Woodruff, J., Brauckmann, A., Magalhães, J. W. D. S., & O’boyle, M. F. P. (2022) · doi:10.1145/3520312.3534867 · Cited by 28 · see citing works
Zheng, Y., Chang, C.-H., Huang, S.-H., Chen, P., & Picek, S. (2024) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2024.3477348 · Cited by 27 · see citing works
Sankar, B. S., Gilliland, D., Rincon, J., Hermjakob, H., Yan, Y., Adam, I., Lemaster, G., Wang, D., Watson, K., Bui, A., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11100984 · Cited by 26 · see citing works
Memorization in Deep Learning: A Survey
Wei, J., Zhang, Y., Zhang, L. Y., Ding, M., Chen, C., Ong, K., Zhang, J., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3769076 · Cited by 26 · see citing works
Training Data Extraction From Pre-trained Language Models: A Survey
Ishihara, S. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.23 · Cited by 26 · see citing works
Esmradi, A., Yip, D. W., & Chan, C. F. (2023) · arXiv · doi:10.48550/arxiv.2312.10982 · Cited by 24 · see citing works
Esmradi, A., Yip, D. W., & Chan, C. F. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-97-1274-8_6 · Cited by 24 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
Licoeval: Evaluating LLMs on License Compliance in Code Generation
Xu, W., Gao, K., He, H., & Zhou, M. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00052 · Cited by 22 · see citing works
TeenyTinyLlama: Open-source tiny language models trained in Brazilian Portuguese
Corrêa, N. K., Falk, S., Fatimah, S., Sen, A., & Oliveira, N. F. D. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100558 · Cited by 22 · see citing works
A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot
Du, F., Ma, X.-J., Yang, J., Liu, Y., Luo, C.-R., Wang, X., Jiang, H.-O., & Jing, X. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-3767-3 · Cited by 22 · see citing works
SoK: Machine Learning GovernancePreprint
Chandrasekaran, V., Jia, H., Thudi, A., Travers, A., Yaghini, M., & Papernot, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10870 · Cited by 22 · see citing works
An archival perspective on pretraining data
Desai, M. A., Pasquetto, I. V., Jacobs, A. Z., & Card, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100966 · Cited by 21 · see citing works
Pal, K. K., Bajaj, A. P., Banerjee, P., Dutcher, A., Nakamura, M., Basque, Z. L., Gupta, H., Sawant, S. A., Anantheswaran, U., Shoshitaishvili, Y., et al. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00152 · Cited by 20 · see citing works
Horawalavithana, S., Ayton, E., Sharma, S. K., Howland, S., Subramanian, M., Vasquez, S., Cosbey, R., Glenski, M., & Volkova, S. (2022) · BIGSCIENCE · doi:10.18653/v1/2022.bigscience-1.12 · Cited by 19 · see citing works
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 19 · see citing works
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 19 · see citing works
Deep Lexical Hypothesis: Identifying personality structure in natural languagePreprint
Cutler, A., & Condon, D. M. (2022) · doi:10.31234/osf.io/gdm5v · Cited by 19 · see citing works
Wei, S., Hu, A., Liang, Y., Yang, J., Yu, L., Li, W., Yang, B., & Qiu, J. (2025) · Radiation Oncology · doi:10.1186/s13014-025-02660-5 · Cited by 18 · see citing works
Impact of Co-occurrence on Factual Knowledge of Large Language Models
Kang, C., & Choi, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.518 · Cited by 18 · see citing works
Demystifying Data Management for Large Language Models
Miao, X., Jia, Z., & Cui, B. (2024) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3626246.3654683 · Cited by 18 · see citing works
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.15076 · Cited by 17 · see citing works
Florensa, A. F., Armenteros, J. J. A., Nielsen, H., Aarestrup, F. M., & Clausen, P. T. L. C. (2024) · NAR Genomics and Bioinformatics · doi:10.1093/nargab/lqae106 · Cited by 17 · see citing works
PreparedLLM: effective pre -pretraining framework for domain-specific large language models
Chen, Z., Lin, M., Wang, Z., Zang, M., & Bai, Y. (2024) · Big Earth Data · doi:10.1080/20964471.2024.2396159 · Cited by 17 · see citing works
Provably Confidential Language Modelling
Zhao, X., Li, L., & Wang, Y.-X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.69 · Cited by 17 · see citing works
Creative Writers' Attitudes on Writing as Training Data for Large Language Models
Gero, K. I., Desai, M., Schnitzler, C., Eom, N., Cushman, J., & Glassman, E. L. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713287 · Cited by 16 · see citing works
Conditional generation of real antigen-specific T cell receptor sequences
Karthikeyan, D., Bennett, S., Reynolds, A. G., Vincent, B. G., & Rubinsteyn, A. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01096-6 · Cited by 16 · see citing works
Ray, U., Arteaga, C., Ahn, Y., & Park, J. (2024) · Engineering Construction & Architectural Management · doi:10.1108/ecam-09-2024-1259 · Cited by 16 · see citing works
- Authors
- Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N.
- Affiliations
- Google (United States) (US)
- University of Pennsylvania (US)
- Corresponding author
- Ippolito, Daphne
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 8424-8445
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3177765786
- Cited by
- 304
- References
- 50
- Citations by year
- 2026: 62 · 2025: 79 · 2024: 65 · 2023: 64 · 2022: 27 · 2021: 7
- FWCI
- 32.49
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- language model training, training data quality, duplicate removal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
