Data Contamination: From Memorization to Exploitation
Open accessAbstract
Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation. Experiments with two models and three downstream tasks show that exploitation exists in some cases, but in others the models memorize the contaminated data, but do not exploit it. We show that these two measures are affected by different factors such as the number of duplications of the contaminated data and the model size. Our results highlight the importance of analyzing massive web-scale datasets to verify that progress in NLP is obtained by better language understanding and not better data exploitation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MedHELM intentionally combines 23 public/gated benchmarks with 14 private benchmarks to mitigate training data contamination.
“Our framework intentionally combines 23 public/gated benchmarks with 14 private benchmarks to address training data contamination, where models may be optimized for public evaluation datasets27.”
From Holistic evaluation of large language models for medical tasks with MedHELM (2026) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual .
- Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposium , SEC'19, page 267-284, USA. USENIX Association.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-totext transformer. Journal of Machine Learning Research , 21(140):1-67.
Cited by
Sorted by citations
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · see citing works
Eight Things to Know about Large Language Models
Bowman, S. R. (2024) · Critical AI · doi:10.1215/2834703x-11556011 · Cited by 198 · see citing works
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 165 · see citing works
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151 · see citing works
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 110 · see citing works
Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges
Ding, B., Qin, C., Zhao, R., Luo, T., Li, X., Chen, G., Xia, W., Hu, J., Luu, A. T., & Joty, S. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.97 · Cited by 109 · see citing works
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning
Razeghi, Y., Logan IV, R. L., Gardner, M., & Singh, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.59 · Cited by 89 · see citing works
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129 · see citing works
Natural language processing in the era of large language models
Zubiaga, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1350306 · Cited by 81 · see citing works
Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
Wang, B., Yue, X., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.795 · Cited by 71 · see citing works
Inductive reasoning in humans and large language models
Han, S. J., Ransom, K., Perfors, A., & Kemp, C. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101155 · Cited by 68 · see citing works
Experimental narratives: A comparison of human crowdsourced storytelling and AI storytelling
Beguš, N. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03868-8 · Cited by 60 · see citing works
Holistic evaluation of large language models for medical tasks with MedHELM
Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., et al. (2026) · Nature Medicine · doi:10.1038/s41591-025-04151-2 · Cited by 59 · see citing works
Large language models as tax attorneys: a case study in legal capabilities emergence
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2024) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2023.0159 · Cited by 53 · see citing works
Task Contamination: Language Models May Not Be Few-Shot Anymore
Li, C., & Flanigan, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29808 · Cited by 47 · see citing works
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 47 · see citing works
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00080 · Cited by 45 · see citing works
Cognitive Biases and Artificial Intelligence
Wang, J., & Redelmeier, D. A. (2024) · NEJM AI · doi:10.1056/aics2400639 · Cited by 43 · see citing works
AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
Schoenegger, P., Park, P. S., Karger, E., Trott, S., & Tetlock, P. (2024) · ACM Trans. Interact. Intell. Syst. · doi:10.1145/3707649 · Cited by 53 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · see citing works
Influence of External Information on Large Language Models Mirrors Social Cognitive Patterns
Bian, N., Lin, H., Liu, P., Lu, Y., Zhang, C.-P., He, B., Han, X., & Sun, L. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3476030 · Cited by 36 · see citing works
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Chen, H., Fang, Z., Singla, Y., & Dredze, M. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.182 · Cited by 31 · see citing works
Ho, C., Tian, T., Ayers, A. T., Aaron, R. E., Phillips, V., Wolf, R. M., Mathioudakis, N. N., Dai, T., & Klonoff, D. C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02757-z · Cited by 31 · see citing works
Chhun, C., Suchanek, F. M., & Clavel, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00689 · Cited by 30 · see citing works
SpecRover: Code Intent Extraction via LLMs
Ruan, H., Zhang, Y., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2408.02232 · Cited by 28 · see citing works
Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI
Pangakis, N., & Wolken, S. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35883 · Cited by 27 · see citing works
Zheng, Y., Chang, C.-H., Huang, S.-H., Chen, P., & Picek, S. (2024) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2024.3477348 · Cited by 27 · see citing works
Prompting Language Models for Linguistic Structure
Blevins, T., Gonen, H., & Zettlemoyer, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.367 · Cited by 25 · see citing works
Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs
Roberts, J., Lüddecke, T., Sheikh, R., Han, K., & Albanie, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00060 · Cited by 24 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., & de Montjoye, Y.-A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00028 · Cited by 20 · see citing works
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 19 · see citing works
JASMINE: Arabic GPT Models for Few-Shot Learning
Nagoudi, E. M. B., Abdul-Mageed, M., Elmadany, A., Inciarte, A. A., & Khondaker, M. T. I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1040 · Cited by 19 · see citing works
Impact of Co-occurrence on Factual Knowledge of Large Language Models
Kang, C., & Choi, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.518 · Cited by 18 · see citing works
Weissweiler, L., Hofmann, V., Kantharuban, A., Cai, A., Dutt, R., Hengle, A., Kabra, A., Kulkarni, A., Vijayakumar, A., Yu, H., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.401 · Cited by 18 · see citing works
Text Embedding Inversion Security for Multilingual Language Models
Chen, Y., Lent, H., & Bjerva, J. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.422 · Cited by 18 · see citing works
Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Ma, Z., Sansom, J., Peng, R., & Chai, J. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.72 · Cited by 17 · see citing works
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias
Tal, Y., Magar, I., & Schwartz, R. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.13 · Cited by 17 · see citing works
Self-citationFlows: Building Blocks of Reasoning and Collaborating AIPreprint
Josifoski, M., Klein, L., Peyrard, M., Baldwin, N., Li, Y., Geng, S., Schnitzler, J. P., Yao, Y., Wei, J., Paul, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01285 · Cited by 14 · see citing works
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35 · see citing works
The mosaic memory of large language models
Shilov, I., Meeus, M., & de Montjoye, Y.-A. (2026) · Nature Communications · doi:10.1038/s41467-026-68603-0 · Cited by 11 · see citing works
Re-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10 · see citing works
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.00159 · Cited by 10 · see citing works
Sun, S., Zhang, Y., Yan, J., Gao, Y., Ong, D., Chen, B., & Su, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.750 · Cited by 10 · see citing works
ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search
Wang, S., Zhuang, S., Koopman, B., & Zuccon, G. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715595 · Cited by 9 · see citing works
FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation
Bayat, F. F., Zhang, L., Munir, S., & Wang, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1587 · Cited by 9 · see citing works
Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
Wu, Y., He, Y., Jia, Y., Mihalcea, R. F., Chen, Y., & Deng, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.717 · Cited by 8 · see citing works
Data Leakage in Visual Datasets
Ramos, P., Ramos, R., & García, N. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00661 · Cited by 7 · see citing works
Benchmarking Benchmark Leakage in Large Language Models
Xu, R., Wang, Z., Fan, R.-Z., & Liu, P. (2024) · arXiv · doi:10.48550/arxiv.2404.18824 · Cited by 6 · see citing works
VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination
Wang, Z., Shao, M., Bhandari, J., Mankali, L., Karri, R., Sinanoglu, O., Shafique, M., & Knechtel, J. (2025) · IEEE International Conference on LLM-Aided Design (ICLAD) · doi:10.1109/iclad65226.2025.00017 · Cited by 6 · see citing works
Data Similarity is Not Enough to Explain Language Model Performance
Yauney, G., Reif, E., & Mimno, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.695 · Cited by 6 · see citing works
Koala: An Index for Quantifying Overlaps with Pre-training Corpora
Vu, T.-T., He, X., Haffari, G., & Shareghi, E. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.7 · Cited by 6 · see citing works
Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 6 · see citing works
Impact of Noisy Supervision in Foundation Model Learning
Chen, H., Wang, Z., Tao, R., Wei, H., Xie, X., Sugiyama, M., Raj, B., & Wang, J. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3552309 · Cited by 6 · see citing works
Detecting Training Data of Large Language Models via Expectation Maximization
Kim, G., Li, Y., Spiliopoulou, E., Ma, J., & Wang, W. Y. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.49 · Cited by 6 · see citing works
Position: Benchmarking is Broken - Don't Let AI be Its Own Judge
Cheng, Z., Wohnig, S., Gupta, R., Alam, S., Abdullahi, T., Alves, J., Nielsen-Garcia, C., Mir, S., Li, S., Orender, J., et al. (2025) · neural information processing systems · doi:10.52202/085713-2433 · Cited by 5 · see citing works
Guimarães, A., Magalhães, J., & Martins, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88717-8_5 · Cited by 4 · see citing works
Diving into Data: Pitfalls and Promises of Data Journalism in Semi-Authoritarian Contexts
Munoriyarwa, A., & Chiumbu, S. H. (2023) · Communicatio · doi:10.1080/02500167.2023.2188471 · Cited by 4 · see citing works
A Comprehensive Evaluation of Tool-Assisted Generation Strategies
Jacovi, A., Caciularu, A., Herzig, J., Aharoni, R., Bohnet, B., & Geva, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.926 · Cited by 4 · see citing works
How Does Memorization Impact LLMs' Social Reasoning? An Assessment using Seen and Unseen Queries
Amirizaniani, M., Sivachenko, M., Lavergne, A., Shah, C. S., & Mashhadi, A. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703576 · Cited by 4 · see citing works
Tseng, Y., Parcollet, T., van Dalen, R. C., Zhang, S., & Bhattacharya, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434665 · Cited by 3 · see citing works
Understanding Domain Learning in Language Models Through Subpopulation Analysis
Zhao, Z., Ziser, Y., & Cohen, S. B. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.16 · Cited by 3 · see citing works
Differentially Private Learning Needs Better Model Initialization and Self-Distillation
Ngong, I. C., Near, J. P., & Mireshghallah, N. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.455 · Cited by 2 · see citing works
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
Cao, J., Chen, S., Zhang, W., Lo, H. C., Li, Y., & Cheung, S.-C. (2025) · Proceedings of the 16th International Conference on Internetware · doi:10.1145/3755881.3755901 · Cited by 2 · see citing works
Should We Evaluate LLM Based Security Analysis Approaches on Open Source Systems?
Dozono, K., Engesser, J., Hummel, B., Roehm, T., & Pretschner, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00265 · Cited by 2 · see citing works
Zhang, P., & Zhang, J. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614963 · Cited by 2 · see citing works
Investigation of the Inference Capabilities and Memorization of Pre-trained Language Models
Sakai, Y., Kamigaito, H., Hayashi, K., & Watanabe, T. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1427 · Cited by 2 · see citing works
Quick Back-Translation for Unsupervised Machine Translation
Brimacombe, B., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.571 · Cited by 2 · see citing works
Razeghi, Y., Mekala, R. R., Logan, R. L., Gardner, M., & Singh, S. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.39 · Cited by 2 · see citing works
HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
He, Y., Wu, Y., Jia, Y., Mihalcea, R., Chen, Y., & Deng, N. (2023) · arXiv · doi:10.48550/arxiv.2310.16755 · Cited by 1 · see citing works
Detecting Interspecific Positive Selection Using Convolutional Neural Networks
West, C., Walker, C. R., Arasti, S., Vasilev, V., Xu, X., De Maio, N., & Goldman, N. (2025) · Molecular Biology and Evolution · doi:10.1093/molbev/msaf154 · Cited by 1 · see citing works
A survey on 1-bit quantized large language models
Tripathi, K., Malik, D., Akshat, A., & Lata, K. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11529-3 · Cited by 1 · see citing works
Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis
Zhu, K., Tu, S., Jin, Z., Hou, L., Li, J., & Zhao, J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.192 · Cited by 1 · see citing works
Unmasking the Type Inference Capabilities of LLMs for Java Code Snippets
Dong, Y., Xu, Z., Tian, Y., & Sun, C. P. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3790099 · Cited by 1 · see citing works
Thejll-Madsen, T., Yongsatianchot, N., & Marsella, S. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3702035 · Cited by 1 · see citing works
CONSPIR ED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety
Bates, L., Glockner, M., Nakov, P., & Gurevych, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.794 · Cited by 1 · see citing works
Large Language Models for Binary Health-Related Question Answering: A Zero- and Few-Shot Evaluation
Fernández-Pichel, M., Losada, D. E., & Pichel, J. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63772-8_29 · Cited by 1 · see citing works
Exploring Chinese Humor Generation: A Study on Two-part Allegorical Sayings
Xu, R. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651265 · Cited by 1 · see citing works
Exploring prompts to elicit memorization in masked language model-based named entity recognition
Xia, Y., Sedova, A., de Araujo, P. H. L., Kougia, V., Nußbaumer, L., & Roth, B. (2025) · PLoS ONE · doi:10.1371/journal.pone.0330877 · Cited by 1 · see citing works
Cross-Language Assessment of Mathematical Capability of ChatGPT
Sathe, G., Shamraj, A., Surve, A., Patil, N., & Saxena, K. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-64779-6_45
Bilateral Ground Reaction Force Prediction Using Deep Learning Models and Custom Force Plate
Yeo, Y. H., Razali, M. F., Ripin, Z. M., Nur-Akasyah, J., Ridzwan, M. I. Z., Tan, A. W. T., & Tay, J. Y. (2025) · IIUM Engineering Journal · doi:10.31436/iiumej.v26i1.3379
Evidence of conceptual mastery in the application of rules by Large Language ModelsPreprint
Nunes, J. L., Almeida, G., & Flanagan, B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5161877
Preference-Strength-Aware Self-Improving Alignment with Generative Preference Models
Zhai, Y., Zhang, Z., Yang, C., Xu, K., Yu, Y., Li, W., Wang, H., Xu, Z., Feng, D., Ding, B., et al. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730063
Barrit, S., Ranuzzi, G., Fetzer, S., Barajraji, M. A., Hadwe, S. E., Zanello, M., Ortler, M., O’Flaherty, J., Massager, N., Madsen, J. R., et al. (2025) · Acta Neurochirurgica · doi:10.1007/s00701-025-06610-8
Mind the Gap: Static and Interactive Evaluations of Large Audio Models
Li, M., Held, W., Ryan, M. J., Pipatanakul, K., Manakul, P., Zhu, H., & Yang, D. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.428
MedAID-ML: A Multilingual Dataset of Biomedical Texts for Detecting AI-Generated Content
Styll, P., Llanos, L. C., Fernández-García, J., & Segura-Bédmar, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_5
Learning or Cheating? Assessing Data Contamination in Large Vision-Language Models
Masry, A., Ahmed, M. F., & Mahbub, R. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204323
DAD-PDD: Dual-Aspect Discrepancies-Based Pre-Training Data Detection for LLMs
Fan, X., Mo, F., Miyamoto, R., Aung, E. S., Chen, C., Matsumoto, T., Kido, F., & Yamana, H. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4091-4_7
Theories of Inclusion and Social Justice
Malahlela, M. K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-3236-9.ch001
Evaluate LLMs on Code Review Understanding: A Multi-step Reasoning Benchmark
Wei, X., & Xin, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4445-5_34
Talebi, S., Tong, E., & Mofrad, M. R. K. (2026) · Discover Computing · doi:10.1007/s10791-026-09940-y
Interactive Visual Assessment for Text-to-Image Generation Models
Mi, X., Tang, F., Cao, J., Sheng, Q., Huang, Z., Li, P., Liu, Y., & Lee, T. (2026) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2026.3658714
Analysis of Semantic Benchmark Data Contamination Attack for LLM-Driven Fake News Detection
Xu, C. G., & Kechadi, T. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402039
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
Tan, D., Chen, P., van Genabith, J., & Chowdhury, K. D. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-short.26
Large Language Model Evaluation
坂井, 優. (2024) · Institutional Repositories DataBase (IRDB) · doi:10.34413/dr.02100
Meta-Evaluation Methodology and Benchmark for Automatic Story Generation
Chhun, C. (2024) · theses.fr (ABES) · doi:10.70675/ec296eb8z87a1z4fdbzbbbdz12bb27449b8f
Peuplement semi-automatisé de bases de connaissances à partir de documents non structurés.
Prieur, M. (2025) · theses.fr (ABES) · doi:10.70675/6c995326zb3b9z422dz9e5azbbabf8c75862
MISP-Bench: Decomposing User-Provided False Priors into Answer, Rationale, and Guard EffectsPreprint
Jeong, I., Kim, Y., Park, J.-H., & Lee, H. (2026) · medRxiv · doi:10.64898/2026.05.07.26352627
- Authors
- Magar, I., & Schwartz, R.
- ORCID
- Schwartz, Roy
- Venue
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2022
- Pages
- 157-165
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4221145126
- Cited by
- 88
- References
- 0
- Citations by year
- 2026: 21 · 2025: 22 · 2024: 21 · 2023: 19 · 2022: 5
- FWCI
- 9.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Allen Institute for Artificial Intelligence
- Israel Science Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- contaminated data, memorization, exploitation, pre-trained language models, BERT, Wikipedia, data duplication, model size, language comprehension
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
