The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Data Contamination: From Memorization to Exploitation
Open accessAbstract
Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation. Experiments with two models and three downstream tasks show that exploitation exists in some cases, but in others the models memorize the contaminated data, but do not exploit it. We show that these two measures are affected by different factors such as the number of duplications of the contaminated data and the model size. Our results highlight the importance of analyzing massive web-scale datasets to verify that progress in NLP is obtained by better language understanding and not better data exploitation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Because data contamination is a well known issue in the field (Balloccu et al. [2024], Magar and Schwartz [2022], Sainz et al. [2023], Jacovi et al. [2023], Xu et al. [2024])”
From A Careful Examination of Large Language Model Performance on Grade School Arithmetic (2024) · cites this paper as Magar and Schwartz [2022]
In the citing paper
“Our framework intentionally combines 23 public/gated benchmarks with 14 private benchmarks to address training data contamination, where models may be optimized for public evaluation datasets27.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 3
- Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposium , SEC'19, page 267-284, USA. USENIX Association.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-totext transformer. Journal of Machine Learning Research , 21(140):1-67.
Cited by
Sorted by citations
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Eight Things to Know about Large Language Models
Bowman, S. R. (2024) · Critical AI · doi:10.1215/2834703x-11556011 · Cited by 305
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 188
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 176
Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges
Ding, B., Qin, C., Zhao, R., Luo, T., Li, X., Chen, G., Xia, W., Hu, J., Luu, A. T., & Joty, S. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.97 · Cited by 126
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129
Natural language processing in the era of large language models
Zubiaga, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1350306 · Cited by 81
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 127
Inductive reasoning in humans and large language models
Han, S. J., Ransom, K., Perfors, A., & Kemp, C. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101155 · Cited by 82
Experimental narratives: A comparison of human crowdsourced storytelling and AI storytelling
Beguš, N. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03868-8 · Cited by 60
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59
Dolma: an open corpus of three trillion tokens for language model pretraining research
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.840 · Cited by 56
Large language models as tax attorneys: a case study in legal capabilities emergence
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2024) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2023.0159 · Cited by 62
Task Contamination: Language Models May Not Be Few-Shot Anymore
Li, C., & Flanigan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29808 · Cited by 47
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 82
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00080 · Cited by 45
Cognitive Biases and Artificial Intelligence
Wang, J., & Redelmeier, D. A. (2024) · NEJM AI · doi:10.1056/aics2400639 · Cited by 43
AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
Schoenegger, P., Park, P. S., Karger, E., Trott, S., & Tetlock, P. E. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3707649 · Cited by 51
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36
Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns
Bian, N., Lin, H., Liu, P., Lu, Y., Zhang, C.-P., He, B., Han, X., & Sun, L. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3476030 · Cited by 60
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Chen, H., Fang, Z., Singla, Y., & Dredze, M. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.182 · Cited by 31
Ho, C., Tian, T., Ayers, A. T., Aaron, R. E., Phillips, V., Wolf, R. M., Mathioudakis, N. N., Dai, T., & Klonoff, D. C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02757-z · Cited by 33
Chhun, C., Suchanek, F. M., & Clavel, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00689 · Cited by 41
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI
Pangakis, N., & Wolken, S. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35883 · Cited by 27
Zheng, Y., Chang, C.-H., Huang, S.-H., Chen, P., & Picek, S. (2024) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2024.3477348 · Cited by 31
Prompting Language Models for Linguistic Structure
Blevins, T., Gonen, H., & Zettlemoyer, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.367 · Cited by 36
Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs
Roberts, J., Lüddecke, T., Sheikh, R., Han, K., & Albanie, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00060 · Cited by 33
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., & de Montjoye, Y.-A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00028 · Cited by 57
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 35
JASMINE: Arabic GPT Models for Few-Shot Learning
Nagoudi, E. M. B., Abdul-Mageed, M., Elmadany, A., Inciarte, A. A., & Khondaker, M. T. I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1040 · Cited by 25
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.00332 · Cited by 19
Impact of Co-occurrence on Factual Knowledge of Large Language Models
Kang, C., & Choi, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.518 · Cited by 38
Weissweiler, L., Hofmann, V., Kantharuban, A., Cai, A., Dutt, R., Hengle, A., Kabra, A., Kulkarni, A., Vijayakumar, A., Yu, H., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.401 · Cited by 27
Text Embedding Inversion Security for Multilingual Language Models
Chen, Y., Lent, H., & Bjerva, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.422 · Cited by 30
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Ma, Z., Sansom, J., Peng, R., & Chai, J. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.72 · Cited by 44
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias
Tal, Y., Magar, I., & Schwartz, R. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.13 · Cited by 25
Self-citationFlows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 34
The mosaic memory of large language models
Shilov, I., Meeus, M., & de Montjoye, Y.-A. (2026) · Nature Communications · doi:10.1038/s41467-026-68603-0 · Cited by 24
Re-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00159 · Cited by 10
Sun, S., Zhang, Y., Yan, J., Gao, Y., Ong, D., Chen, B., & Su, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.750 · Cited by 15
ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search
Wang, S., Zhuang, S., Koopman, B., & Zuccon, G. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715595 · Cited by 15
FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation
Bayat, F. F., Zhang, L., Munir, S., & Wang, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1587 · Cited by 24
Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
Wu, Y., He, Y., Jia, Y., Mihalcea, R. F., Chen, Y., & Deng, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.717 · Cited by 33
Data Leakage in Visual Datasets
Ramos, P., Ramos, R., & García, N. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00661 · Cited by 14
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6
VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination
Wang, Z., Shao, M., Bhandari, J., Mankali, L., Karri, R., Sinanoglu, O., Shafique, M., & Knechtel, J. (2025) · IEEE International Conference on LLM-Aided Design (ICLAD) · doi:10.1109/iclad65226.2025.00017 · Cited by 8
Data Similarity is Not Enough to Explain Language Model Performance
Yauney, G., Reif, E., & Mimno, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.695 · Cited by 13
Koala: An Index for Quantifying Overlaps with Pre-training Corpora
Vu, T.-T., He, X., Haffari, G., & Shareghi, E. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.7 · Cited by 19
Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 7
Impact of Noisy Supervision in Foundation Model Learning
Chen, H., Wang, Z., Tao, R., Wei, H., Xie, X., Sugiyama, M., Raj, B., & Wang, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3552309 · Cited by 14
Detecting Training Data of Large Language Models via Expectation Maximization
Kim, G., Li, Y., Spiliopoulou, E., Ma, J., & Wang, W. Y. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.49 · Cited by 14
Position: Benchmarking is Broken - Don't Let AI be Its Own Judge
Cheng, Z., Wohnig, S., Gupta, R., Alam, S., Abdullahi, T., Alves, J., Nielsen-Garcia, C., Mir, S., Li, S., Orender, J., et al. (2025) · neural information processing systems · doi:10.52202/085713-2433 · Cited by 6
Guimarães, A., Magalhães, J., & Martins, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88717-8_5 · Cited by 4
Diving into Data: Pitfalls and Promises of Data Journalism in Semi-Authoritarian Contexts
Munoriyarwa, A., & Chiumbu, S. H. (2023) · Communicatio · doi:10.1080/02500167.2023.2188471 · Cited by 4
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 7
How Does Memorization Impact LLMs' Social Reasoning? An Assessment using Seen and Unseen Queries
Amirizaniani, M., Sivachenko, M., Lavergne, A., Shah, C. S., & Mashhadi, A. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703576 · Cited by 4
Tseng, Y., Parcollet, T., van Dalen, R. C., Zhang, S., & Bhattacharya, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434665 · Cited by 3
Understanding Domain Learning in Language Models Through Subpopulation Analysis
Zhao, Z., Ziser, Y., & Cohen, S. B. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.16 · Cited by 4
Differentially Private Learning Needs Better Model Initialization and Self-Distillation
Ngong, I. C., Near, J. P., & Mireshghallah, N. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.455 · Cited by 2
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
Cao, J., Chen, S., Zhang, W., Lo, H. C., Li, Y., & Cheung, S.-C. (2025) · Proceedings of the 16th International Conference on Internetware · doi:10.1145/3755881.3755901 · Cited by 2
Should We Evaluate LLM Based Security Analysis Approaches on Open Source Systems?
Dozono, K., Engesser, J., Hummel, B., Roehm, T., & Pretschner, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00265 · Cited by 4
Zhang, P., & Zhang, J. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614963 · Cited by 2
Investigation of the Inference Capabilities and Memorization of Pre-trained Language Models
Sakai, Y., Kamigaito, H., Hayashi, K., & Watanabe, T. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1427 · Cited by 2
Quick Back-Translation for Unsupervised Machine Translation
Brimacombe, B., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.571 · Cited by 2
Razeghi, Y., Mekala, R. R., Logan, R. L., Gardner, M., & Singh, S. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.39 · Cited by 2
HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
He, Y., Wu, Y., Jia, Y., Mihalcea, R., Chen, Y., & Deng, N. (2023) · arXiv · doi:10.48550/arxiv.2310.16755 · Cited by 1
Detecting Interspecific Positive Selection Using Convolutional Neural Networks
West, C., Walker, C. R., Arasti, S., Vasilev, V., Xu, X., De Maio, N., & Goldman, N. (2025) · Molecular Biology and Evolution · doi:10.1093/molbev/msaf154 · Cited by 1
A survey on 1-bit quantized large language models
Tripathi, K., Malik, D., Akshat, A., & Lata, K. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11529-3 · Cited by 1
Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis
Zhu, K., Tu, S., Jin, Z., Hou, L., Li, J., & Zhao, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.192 · Cited by 2
Unmasking the Type Inference Capabilities of LLMs for Java Code Snippets
Dong, Y., Xu, Z., Tian, Y., & Sun, C. P. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3790099 · Cited by 1
Thejll-Madsen, T., Yongsatianchot, N., & Marsella, S. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3702035 · Cited by 1
CONSPIR ED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
Bates, L., Glockner, M., Nakov, P., & Gurevych, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.794 · Cited by 1
Large Language Models for Binary Health-Related Question Answering: A Zero- and Few-Shot Evaluation
Fernández-Pichel, M., Losada, D. E., & Pichel, J. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63772-8_29 · Cited by 2
Exploring Chinese Humor Generation: A Study on Two-part Allegorical Sayings
Xu, R. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651265 · Cited by 2
Exploring prompts to elicit memorization in masked language model-based named entity recognition
Xia, Y., Sedova, A., de Araujo, P. H. L., Kougia, V., Nußbaumer, L., & Roth, B. (2025) · PLoS ONE · doi:10.1371/journal.pone.0330877 · Cited by 1
Cross-Language Assessment of Mathematical Capability of ChatGPT
Sathe, G., Shamraj, A., Surve, A., Patil, N., & Saxena, K. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-64779-6_45
Bilateral Ground Reaction Force Prediction Using Deep Learning Models and Custom Force Plate
Yeo, Y. H., Razali, M. F., Ripin, Z. M., Nur-Akasyah, J., Ridzwan, M. I. Z., Tan, A. W. T., & Tay, J. Y. (2025) · IIUM Engineering Journal · doi:10.31436/iiumej.v26i1.3379
Evidence of conceptual mastery in the application of rules by Large Language ModelsPreprint
Nunes, J. L., Almeida, G., & Flanagan, B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5161877
Preference-Strength-Aware Self-Improving Alignment with Generative Preference Models
Zhai, Y., Zhang, Z., Yang, C., Xu, K., Yu, Y., Li, W., Wang, H., Xu, Z., Feng, D., Ding, B., et al. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730063
Barrit, S., Ranuzzi, G., Fetzer, S., Barajraji, M. A., Hadwe, S. E., Zanello, M., Ortler, M., O’Flaherty, J., Massager, N., Madsen, J. R., et al. (2025) · Acta Neurochirurgica · doi:10.1007/s00701-025-06610-8
Mind the Gap: Static and Interactive Evaluations of Large Audio Models
Li, M., Held, W., Ryan, M. J., Pipatanakul, K., Manakul, P., Zhu, H., & Yang, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.428
MedAID-ML: A Multilingual Dataset of Biomedical Texts for Detecting AI-Generated Content
Styll, P., Llanos, L. C., Fernández-García, J., & Segura-Bédmar, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_5
Learning or Cheating? Assessing Data Contamination in Large Vision-Language Models
Masry, A., Ahmed, M. F., & Mahbub, R. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204323
DAD-PDD: Dual-Aspect Discrepancies-Based Pre-Training Data Detection for LLMs
Fan, X., Mo, F., Miyamoto, R., Aung, E. S., Chen, C., Matsumoto, T., Kido, F., & Yamana, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_7 · Cited by 1
Theories of Inclusion and Social Justice
Malahlela, M. K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-3236-9.ch001
Evaluate LLMs on Code Review Understanding: A Multi-step Reasoning Benchmark
Wei, X., & Xin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4445-5_34
Talebi, S., Tong, E., & Mofrad, M. R. K. (2026) · Discover Computing · doi:10.1007/s10791-026-09940-y
Interactive Visual Assessment for Text-to-Image Generation Models
Mi, X., Tang, F., Cao, J., Sheng, Q., Huang, Z., Li, P., Liu, Y., & Lee, T. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3658714
Analysis of Semantic Benchmark Data Contamination Attack for LLM-Driven Fake News Detection
Xu, C. G., & Kechadi, T. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402039
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
Tan, D., Chen, P., van Genabith, J., & Chowdhury, K. D. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-short.26
Large Language Model Evaluation
坂井, 優. (2024) · Institutional Repositories DataBase (IRDB) · doi:10.34413/dr.02100
Meta-Evaluation Methodology and Benchmark for Automatic Story Generation
Chhun, C. (2024) · theses.fr (ABES) · doi:10.70675/ec296eb8z87a1z4fdbzbbbdz12bb27449b8f
Metadata
- Authors
- Magar, I., & Schwartz, R.
- ORCID
- Schwartz, Roy
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2022
- Pages
- 157-165
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4221145126
- Cited by
- 88
- References
- 0
- Citations by year
- 2026: 21 · 2025: 22 · 2024: 21 · 2023: 19 · 2022: 5
- FWCI
- 9.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Allen Institute for Artificial Intelligence
- Israel Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- contaminated data, memorization, exploitation, pre-trained language models, BERT, Wikipedia, data duplication, model size, language comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
