Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Open accessAbstract
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, Ondrej Dusek. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Chain-of-Thought Hub benchmark used final answer accuracy as a proxy for reasoning capability without considering the correctness of intermediate steps, potentially failing to differentiate genuine reasoning from technical optimization.
“Chain-of-Thought Hub [40]: The study used final answer accuracy as a proxy for reasoning capability without considering the correctness of intermediate steps, potentially failing to differentiate genuine reasoning from technical optimization.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [40]
Chain-of-Thought Hub included bilingual reasoning in English and Simplified Chinese (erroneously labeling them as 'Chinese' without differentiation with Traditional Chinese), but did not address linguistic differences and embedded logics across different languages beyond this.
“The benchmark study did not acknowledge this inadequacy or make attempts to address it.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [40]
Chain-of-Thought Hub integrated diverse benchmarks for evaluating LLM reasoning, but did not address implementation inconsistencies, focusing instead on model performance comparison.
“The study integrated diverse benchmarks (e.g., GSM8k, MATH, MMLU) for evaluating LLM reasoning, but did not address implementation inconsistencies, focusing instead on model performance comparison.There was no mention of efforts to standardize benchmark execution or address potential inconsistencies across different research teams.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [40]
Chain-of-Thought Hub aimed to evaluate LLMs' reasoning capabilities but did not specifically address slow test iteration time; it required evaluating new models across multiple complex reasoning tasks, likely extending the evaluation period significantly.
“The study introduced the Chain-of-Thought Hub, which aimed to evaluate LLMs' reasoning capabilities, but did not specifically address the slow test iteration time.This benchmark required evaluating new models across multiple complex reasoning tasks, which likely extended the evaluation period significantly, similar to the slow iteration time observed in benchmarks like BIG-bench and HELM.However,”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [40]
Chain-of-Thought Hub introduced a suite of reasoning benchmarks to measure LLM reasoning capabilities without addressing the intricacy of crafting prompts that accurately assess these capabilities without bias, and did not specifically mention attempts to address prompt engineering.
“The study introduced the Chain-of-Thought Hub to measure reasoning capabilities of LLMs using a suite of reasoning benchmarks without addressing the intricacy of crafting prompts that accurately assess these capabilities without bias, which was crucial for fair evaluation.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [40]
Chain-of-Thought Hub curated benchmarks without addressing diversity in human curators and evaluators, leading to potential biases, and did not account for variability in cultural, religious, political, and academic backgrounds of humans.
“The study curated benchmarks without addressing the diversity in human curators and evaluators, leading to potential biases in benchmark development and evaluation.The benchmarks did not account for the variability in cultural, religious, political, and academic backgrounds of humans, which could intro-duce inconsistencies and subjectivity in LLM evaluation.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 30
- Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christo- pher Choquette Choo, and Nicholas Carlini. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Lan- guage Generation Conference, pages 28-53, Prague, Czechia. Association for Computational Linguistics.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
- Zhengxaing Shi and Aldo Lipani. 2023. Don't stop pretraining? make prompt-based fine-tuning power- ful learner. In Thirty-seventh Conference on Neural Information Processing Systems.
- Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2022. MultiWOZ 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 351-360, Edinburgh, UK. Association for Computational Lin- guistics.
- Fares Antaki, Samir Touma, Daniel Milad, Jonathan El-Khoury, and Renaud Duval. 2023. Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings. Ophthal- mology Science, 3(4):100324.
- Zhenguang G Cai, David A Haslett, Xufeng Duan, Shuqi Wang, and Martin J Pickering. 2023. Does ChatGPT resemble humans in language use? arXiv preprint arXiv:2303.08014. · arXiv
- Sarah E. Finch, Ellie S. Paek, and Jinho D. Choi. 2023. Leveraging large language models for automated dia- logue analysis. In Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dia- logue, pages 202-215, Prague, Czechia. Association for Computational Linguistics.
- Shahriar Golchin and Mihai Surdeanu. 2023. Time travel in LLMs: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08493. · arXiv
- Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics. · DOI · Paper page
- Tahsina Hashem, Weiqing Wang, Derry Tanti Wijaya, Mohammed Eunus Ali, and Yuan-Fang Li. 2023. Generating faithful text from a knowledge graph with noisy reference text. In Proceedings of the 16th Inter- national Natural Language Generation Conference, pages 106-122, Prague, Czechia. Association for Computational Linguistics.
- Takanobu Hirosawa, Yukinori Harada, Masashi Yokose, Tetsu Sakamoto, Ren Kawamura, and Taro Shimizu. 2023. Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: A pilot study. International Journal of Environmental Research and Public Health, 20(4).
- Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christo- pher Choquette Choo, and Nicholas Carlini. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Lan- guage Generation Conference, pages 28-53, Prague, Czechia. Association for Computational Linguistics.
- Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wik- toria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radliński, Konrad Wojtasik, Stanisław Woźniak, and Prze- mysław Kazienko. 2023. ChatGPT: Jack of all trades, master of none. Information Fusion, 99:101861. · DOI · Paper page
- Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz- Candido, James Maningo, and Victor Tseng. 2023. Performance of ChatGPT on usmle: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2):1-12. · DOI · Paper page
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christo- pher D Manning, Christopher Re, Diana Acosta- Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan An- drew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Ko- reeda. 2023a. Holistic evaluation of language models. Transactions on Machine Learning Research. Fea- tured Certification, Expert Certification.
- Yancheng Liang, Jiajie Zhang, Hui Li, Xiaochen Liu, Yi Hu, Yong Wu, Jiaoyao Zhang, Yongyan Liu, and Yi Wu. 2023b. Breaking the bank with ChatGPT: Few-shot text classification for finance. In Proceed- ings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecasting, pages 74- 80, Macao.
- Xin Liu, Yuan Tan, Zhenghang Xiao, Jianwei Zhuge, and Rui Zhou. 2023f. Not the end of story: An eval- uation of ChatGPT-driven vulnerability description mappings. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 3724-3731, Toronto, Canada. Association for Computational Lin- guistics. · DOI
- Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023. Exploring effectiveness of GPT-3 in grammatical error correction: A study on performance and controllability in prompt-based methods. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 205-219, Toronto, Canada. Association for Computational Linguistics. · DOI
- Duan Nan. 2023. Frontier review of multimodal AI. In Proceedings of the 22nd Chinese National Confer- ence on Computational Linguistics (Volume 2: Fron- tier Forum), pages 110-118, Harbin, China. Chinese Information Processing Society of China.
- Andrew Nedilko. 2023. Generative pretrained trans- formers for emotion detection in a code-switching setting. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 616-620, Toronto, Canada. Association for Computational Linguistics. · DOI
- Amin Omidvar and Aijun An. 2023. Empowering con- versational agents using semantic in-context learning. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 766-771, Toronto, Canada. Asso- ciation for Computational Linguistics. · DOI
- Lidiia Ostyakova, Veronika Smilga, Kseniia Petukhova, Maria Molchanova, and Daniel Kornev. 2023. Chat- GPT vs. crowdsourcing vs. experts: Annotating open- domain conversations with speech functions. In Pro- ceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, pages 242-254, Prague, Czechia. Association for Computational Lin- guistics.
- Andrei Popescu-Belis, Àlex R. Atrio, Bastien Bernath, Etienne Boisson, Teo Ferrari, Xavier Theimer- lienhard, and Giorgos Vernikos. 2023. GPoeT: a language model trained for rhyme generation on syn- thetic data. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Litera- ture, pages 10-20, Dubrovnik, Croatia. Association for Computational Linguistics. · DOI
- Jakob Schuster and Katja Markert. 2023. Nut- cracking sledgehammers: Prioritizing target language data over bigger language models for cross-lingual metaphor detection. In Proceedings of the 2023 CLASP Conference on Learning with Small Data (LSD), pages 98-106, Gothenburg, Sweden. Associa- tion for Computational Linguistics.
- Zhengxaing Shi and Aldo Lipani. 2023. Don't stop pretraining? make prompt-based fine-tuning power- ful learner. In Thirty-seventh Conference on Neural Information Processing Systems.
- David Stap and Ali Araabi. 2023. ChatGPT is not a good indigenous translator. In Proceedings of the Workshop on Natural Language Processing for In- digenous Languages of the Americas (AmericasNLP), pages 163-167, Toronto, Canada. Association for Computational Linguistics. · DOI
- Bhaskara Hanuma Vedula, Prashant Kodali, Manish Shrivastava, and Ponnurangam Kumaraguru. 2023. PrecogIIITH@WASSA2023: Emotion detection for Urdu-English code-mixed text. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 601-605, Toronto, Canada. Association for Computational Linguistics. · DOI
- Zihang Xu, Ziqing Yang, Yiming Cui, and Shijin Wang. 2023c. IDOL: Indicator-oriented logic pre-training for logical reasoning. In Findings of the Associa- tion for Computational Linguistics: ACL 2023, pages 8099-8111, Toronto, Canada. Association for Com- putational Linguistics. · DOI
- Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. How would stance detection techniques evolve after the launch of ChatGPT? arXiv preprint arXiv:2212.14548. · arXiv
- Yi Zheng, Björn Ross, and Walid Magdy. 2023c. What makes good counterspeech? a comparison of genera- tion approaches and evaluation metrics. In Proceed- ings of the 1st Workshop on CounterSpeech for On- line Abuse (CS4OA), pages 62-71, Prague, Czechia. Association for Computational Linguistics.
Cited by
Sorted by citations
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
Manias, D. M., Chouman, A., & Shami, A. A. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901065 · Cited by 21 · see citing works
An Empirical Study on the Code Refactoring Capability of Large Language Models
Cordeiro, J., Noei, S., & Zou, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3801158 · Cited by 17 · see citing works
A comparison of chain-of-thought reasoning strategies across datasets and models
Hebenstreit, K., Praas, R., Kiesewetter, L. P., & Samwald, M. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1999 · Cited by 15 · see citing works
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.20 · Cited by 37 · see citing works
Towards privacy-aware mental health AI models
Mandal, A., Chakraborty, T., & Gurevych, I. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00875-w · Cited by 25 · see citing works
Panzeri, D., Laohawetwanit, T., Akpınar, R., Carlo, C. D., Belsito, V., Terracciano, L., Aghemo, A., Pugliese, N., Chirico, G. G., Inverso, D., et al. (2025) · Hepatology Communications · doi:10.1097/hc9.0000000000000695 · Cited by 14 · see citing works
Lessons from complex systems science for AI governance
Kolt, N., Shur‐Ofry, M., & Reuven Cohen, C. (2025) · Patterns · doi:10.1016/j.patter.2025.101341 · Cited by 22 · see citing works
Applying Artificial Intelligence and machine learning in precision nutrition
Massara, P., Kirkland, J., Pagani, I., Huey, S. L., Hirsh, H., McDonald, D. T., Patel, L., Finkelstein, J. L., Gantz, M. G., Wang, F., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-75004-w · Cited by 14 · see citing works
LLM-Enhanced Learning Environments for CS: Exploring Data Structures and Algorithms with Gurukul
Rachha, A., & Seyam, M. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893211 · Cited by 13 · see citing works
The Relevance of Large Language Models for Project Management
Karnouskos, S. (2024) · IEEE Open Journal of the Industrial Electronics Society · doi:10.1109/ojies.2024.3412222 · Cited by 15 · see citing works
Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation
Zhang, R., Ouni, J., & Eger, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00519 · Cited by 10 · see citing works
Abdullah, A., Ather, M. A., Kolesnikova, O., & Sidorov, G. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9070190 · Cited by 8 · see citing works
A systematic survey of natural language processing for the Greek language
Bakagianni, J., Pouli, K., Gavriilidou, M., & Pavlopoulos, J. (2025) · Patterns · doi:10.1016/j.patter.2025.101313 · Cited by 14 · see citing works
The future of large language models in fighting emerging outbreaks: lights and shadows
Rizzo, A., Mensa, E., & Giacomelli, A. (2024) · The Lancet Microbe · doi:10.1016/j.lanmic.2024.07.017 · Cited by 8 · see citing works
Challenges and opportunities of automated essay scoring for low-proficient L2 English writers
Wilde, V. D., & Clercq, O. D. (2025) · Assessing Writing · doi:10.1016/j.asw.2025.100982 · Cited by 7 · see citing works
VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination
Wang, Z., Shao, M., Bhandari, J., Mankali, L., Karri, R., Sinanoglu, O., Shafique, M., & Knechtel, J. (2025) · IEEE International Conference on LLM-Aided Design (ICLAD) · doi:10.1109/iclad65226.2025.00017 · Cited by 6 · see citing works
Data Leakage in Visual Datasets
Ramos, P., Ramos, R., & García, N. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00661 · Cited by 7 · see citing works
General scales unlock AI evaluation with explanatory and predictive power
Zhou, L., Pacchiardi, L., Martínez‐Plumed, F., Collins, K. M., Moros-Daval, Y., Zhang, S., Zhao, Q., Huang, Y., Sun, L., Prunty, J. E., et al. (2026) · Nature · doi:10.1038/s41586-026-10303-2 · Cited by 8 · see citing works
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Waldis, A., Perlitz, Y., Choshen, L., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 9 · see citing works
Representations of smells: The next frontier for language models?
Kurfalı, M., Herman, P. A., Pierzchajlo, S., Olofsson, J., & Hörberg, T. (2025) · Cognition · doi:10.1016/j.cognition.2025.106243 · Cited by 5 · see citing works
Debelak, R., Koch, T. K., Aßenmacher, M., & Stachl, C. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251351285 · Cited by 5 · see citing works
Liu, C., Wei, Z., Liao, H., & Dong, W. (2026) · Transactions in GIS · doi:10.1111/tgis.70189 · Cited by 4 · see citing works
ConvNLP: Image-based AI Text Detection
Jambunathan, S. P., Shankarnarayan, A., & Dube, P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825691 · Cited by 3 · see citing works
Kalimah norms: Ratings for 2,467 modern standard Arabic words on two scales
Alzahrani, A. A., Aljuaythin, W., Alshumrani, H. A., Saleh, A. M., & Mostafa, M. M. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02692-5 · Cited by 3 · see citing works
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanovic, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 3 · see citing works
Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?
Razumovskaia, E., Vulić, I., & Korhonen, A. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.33 · Cited by 9 · see citing works
Revisiting Unnaturalness for Automated Program Repair in the Era of Large Language Models
Yang, A. Z. H., Kolak, S., Hellendoorn, V. J., Martins, R., & Goues, C. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00089 · Cited by 3 · see citing works
A Reproducibility Study of Graph-Based Legal Case Retrieval
Donabauer, G., & Kruschwitz, U. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730282 · Cited by 2 · see citing works
Kumar, V., Rajawat, P. S., & Ntoutsi, E. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228771 · Cited by 2 · see citing works
Meisenbacher, S., Klymenko, A., Bodea, A.-E., & Matthes, F. (2025) · Workshop on Privacy in the Electronic Society · doi:10.1145/3733802.3764058 · Cited by 2 · see citing works
Parschan, P., & Jakob, C. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02500-4 · Cited by 2 · see citing works
Should We Evaluate LLM Based Security Analysis Approaches on Open Source Systems?
Dozono, K., Engesser, J., Hummel, B., Roehm, T., & Pretschner, A. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00265 · Cited by 2 · see citing works
Adaptive, Privacy-Preserving Small Language Models for Multi-Task Clinical Assistance
Zheng, G., Kamel, P. I., Pillai, J. J., Akhbardeh, A., Braverman, V., Jacobs, M. A., & Parekh, V. S. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01912-4 · Cited by 2 · see citing works
Gibson, B. N., Chodapuneedi, S., Koh, H. Y., Gupta, S., Gašević, D., Wang, A. Y., Marzolini, C., Khoo, S., Romero, L., Dalli, L. L., et al. (2026) · British Journal of Pharmacology · doi:10.1111/bph.70515 · Cited by 2 · see citing works
Benchmark of linguistic variation in LLM‑generated texts
Milička, J., Marklová, A., & Cvrček, V. (2026) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.25151.mil · Cited by 2 · see citing works
Albert Gatt, A. (2025) · Oxford University Press eBooks · doi:10.1093/acrefore/9780199384655.013.896 · Cited by 2 · see citing works
Salim, R. A., Delvin, D., Harefa, J., & Jingga, K. (2024) · International Conference on Informatics, Multimedia, Cyber and Information System (ICIMCIS) · doi:10.1109/icimcis63449.2024.10957638 · Cited by 1 · see citing works
Unmasking the Type Inference Capabilities of LLMs for Java Code Snippets
Dong, Y., Xu, Z., Tian, Y., & Sun, C. P. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3790099 · Cited by 1 · see citing works
Tseng, Y., Parcollet, T., van Dalen, R. C., Zhang, S., & Bhattacharya, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434665 · Cited by 3 · see citing works
Silent Failures in Federated Personalization of Foundation Models
Oh, Y., & Bui, A. A. T. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818661 · Cited by 1 · see citing works
AI cannot replace human participants in behavioral research
Hoover, J., Cooke, A. D. J., Tomaino, G., & Lee, S. H. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2611363123 · Cited by 1 · see citing works
What can AI learn from the history of statistics?
Láinez-Moreno, D., Lorenzo-Arribas, A., & Puig, P. (2024) · Significance · doi:10.1093/jrssig/qmae077 · Cited by 1 · see citing works
Gete, H., Etchegoyhen, T., Labaka, G., Corral, A., Saralegi, X., Aranberri, N., Ponce, D., Santos, I. E., & Martín, M. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09854-z
MedAID-ML: A Multilingual Dataset of Biomedical Texts for Detecting AI-Generated Content
Styll, P., Llanos, L. C., Fernández-García, J., & Segura-Bédmar, I. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_5
Is Word2Vec Dead for Topic Modeling? A Case Study on Hospitality Opinion Mining
Azaïs, M.-A., Guillaume, J., & Coustaty, M. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-06136-2_11
Augmented human intelligence goes beyond the AGI mirage
Lissack, M. R., & Meagher, B. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00683-1
Song, Y., & Cohen, E. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-09156-7_2
Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions
Finch, J. D., Josyula, Y., & Choi, J. D. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.66
Is Machine-Translation Enough? Understanding Impacts in LLM Benchmarking
Pedrosa, T., Costa, E. D. B., & dos Santos, R. J. R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15984-7_8
Sun, M., Reiter, E., Murchie, P., Kiltie, A. E., Ramsay, G., Duncan, L., & Adam, R. (2026) · medRxiv · doi:10.64898/2026.02.02.26345346
Janoudi, G., Uzun, M., Yasinov, E., & Richter, T. A. (2026) · medRxiv · doi:10.64898/2026.03.02.26347434
Hsu, T.-Y., Hsu, Y.-L., Rohatgi, S., Huang, C.-Y., Ng, H. Y. S., Rossi, R., Kim, S., Yu, T., Ku, L., Giles, C. L., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.653 · Cited by 2 · see citing works
Security, Privacy, and Ethical Challenges in Large Language Models: A Review
Yogi, B., Ghosh, N., & Sahid, S. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10680-x
Pyeon, G., Heo, I., Jung, J., Hwang, T., Namgoong, H., Seo, H., Han, Y., Kim, E., Kang, H., & Jung, S. (2026) · Applied Artificial Intelligence · doi:10.1080/08839514.2026.2684121
Local language models for Spanish L2 writing assessment: human baselines, RAG, and fine-tuning
Zimotti, G. (2026) · Journal of Computers in Education · doi:10.1007/s40692-026-00399-w
Is It Novel and Why? Fine-Grained Patent Novelty Prediction Based on Passage Retrieval
Knappich, V., Hätty, A., Razniewski, S., & Friedrich, A. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809576
BenchGuard: A Two-Layer Framework for Contamination-Aware Evaluation of LLM Vulnerability Detection
Mahmud, A., Rawajfih, Y., Arnold, R. D., & Shahriar, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33260-8_22
Joaquín, R. A. R. (2026) · Espila · doi:10.61454/yhetdk61
The Impact and Potential of Large Language Models in Personalized Learning Environments
Aderghal, Y., Ghazi, A. E., & Hafidi, H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-23844-3_30
Azaïs, M.-A., Coustaty, M., & Guillaume, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31438-3_34
A survey of reward hacking in agentic large language model systems
Morampudi, A., Irrink, U., Grandhi, R., Pagadala, V., & Maddula, M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01980-z
Sadiq, B. H., & Ahmed, S. H. (2026) · Dasinya Journal for Engineering and Informatics · doi:10.65542/djei.v2i3.82
Rafi, M. N., Kim, D. J., Chen, T.-H., & Wang, S. (2024) · arXiv · doi:10.48550/arxiv.2412.18750
Deininger, P., Rappl, S., Lindner, H., & Gebeshuber, K. (2026) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp6050163
A Survey of Evaluation Methods for the Security and Trustworthiness of Large Language Models
Zhao, L., Xu, Z., Zhao, R., Li, C., Yang, B., & Feng, D. (2026) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt69916.2026.11689272
Language Models in Video Anomaly Detection: A Dynamic Survey
Mumcu, F., Bekit, L., Jones, M., Cherian, A., & Yılmaz, Y. (2026) · IEEE Access · doi:10.1109/access.2026.3737989
Beyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models
Góral, G., Ziarko, A., Miłoś, P., Nauman, M., Wołczyk, M., & Kosinski, M. (2026) · doi:10.1109/cvprf71909.2026.00170
Gu, F., & Boyd-Graber, J. L. (2026) · IntechOpen eBooks · doi:10.5772/intechopen.1018041
- Authors
- Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O.
- Affiliations
- Charles University (CZ)
- Venue
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 67-93
- OpenAlex
- W4411630291
- Cited by
- 70
- References
- 0
- Citations by year
- 2026: 32 · 2025: 28 · 2024: 10
- FWCI
- 20.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- Funders
- Univerzita Karlova v Praze
- HORIZON EUROPE European Research Council (101039303)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- contaminated data, data leakage, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
