Task Contamination: Language Models May Not Be Few-Shot Anymore
Open accessAbstract
Large language models (LLMs) offer impressive performance in various zero-shot and few-shot tasks. However, their success in zero-shot or few-shot settings may be affected by task contamination, a potential limitation that has not been thoroughly examined. This paper investigates how zero-shot and few-shot performance of LLMs has changed chronologically over datasets released over time, and over LLMs released over time. Utilizing GPT-3 series models and several other recent open-sourced LLMs, and controlling for dataset difficulty, we find that datasets released prior to the LLM training data creation date perform surprisingly better than datasets released post the LLM training data creation date. This strongly indicates that, for many LLMs, there exists task contamination on zero-shot and few-shot evaluation for datasets prior to the LLMs' training data creation date. Additionally, we utilize training data inspection, training data extraction, and a membership inference attack, which reveal further evidence of task contamination. Importantly, we find that for tasks with no possibility of task contamination, LLMs rarely demonstrate statistically significant improvements over simple majority baselines, in both zero and few-shot settings.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
We reserve a random sample of 50% of the dataset to guard against potential model memorization and ensure that a proper test set remains available for future evaluation.
“Consistent with emerging understanding of AI benchmarking and leaderboards, we reserve a random sample of 50% of the dataset to guard against potential model memorization and ensure that a proper test set remains available for future evaluation (Haimes et al. 2024; Li and Flanigan 2024)”
From Hallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools (2025) · cites this paper as Li and Flanigan 2024
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305 · see citing works
Toolformer: language models can teach themselves to use tools
Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2997 · Cited by 1,150 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · see citing works
Membership Inference Attacks on Machine Learning: A Survey
Hu, H., Salcic, Z. A., Sun, L., Dobbie, G., Yu, P. S., & Zhang, X. (2022) · ACM Computing Surveys · doi:10.1145/3523273 · Cited by 586 · see citing works
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · see citing works
Dixon, W. J., & Mood, A. M. (1946) · Journal of the American Statistical Association · doi:10.1080/01621459.1946.10501898 · Cited by 418 · see citing works
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
Qin, G., & Eisner, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.410 · Cited by 342 · see citing works
Efficient Large Scale Language Modeling with Mixtures of ExpertsPreprint
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J.-F., Iyer, S., Pasunuru, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2112.10684 · Cited by 262 · see citing works
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.08887 · Cited by 193 · see citing works
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129 · see citing works
Efficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S. V., Pasunuru, R., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.804 · Cited by 122 · see citing works
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Chang, K. K., Cramer, M. H., Soni, S., & Bamman, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.453 · Cited by 110 · see citing works
Few-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107 · see citing works
Iteratively Prompt Pre-trained Language Models for Chain of Thought
Wang, B., Deng, X. B., & Sun, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.174 · Cited by 89 · see citing works
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
In-Context Learning for Few-Shot Dialogue State Tracking
Hu, Y., Lee, C.-H., Xie, T., Yu, T., Smith, N. A., & Mari Ostendorf, E. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.193 · Cited by 74 · see citing works
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72 · see citing works
Synchromesh: Reliable code generation from pre-trained language models
Poesia, G., Polozov, O., Le, V., Tiwari, A., Soares, G., Meek, C., & Gulwani, S. (2022) · arXiv · doi:10.48550/arxiv.2201.11227 · Cited by 72 · see citing works
Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model EvaluationPreprint
Li, Y. (2023) · arXiv.org · doi:10.48550/arxiv.2309.10677 · Cited by 59 · see citing works
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Yang, R., Song, L., Li, Y., Zhao, S., Ge, Y., Li, X., & Shan, Y. (2023) · neural information processing systems · doi:10.52202/075280-3149 · Cited by 46 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.308 · Cited by 36 · see citing works
Jacovi, A., Caciularu, A., Goldman, O., & Goldberg, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.10160 · Cited by 36 · see citing works
Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2023) · arXiv · doi:10.48550/arxiv.2308.08493 · Cited by 29 · see citing works
Prompting Language Models for Linguistic Structure
Blevins, T., Gonen, H., & Zettlemoyer, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.367 · Cited by 25 · see citing works
Don't Make Your LLM an Evaluation Benchmark Cheater
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., & Han, J. (2023) · arXiv · doi:10.48550/arxiv.2311.01964 · Cited by 25 · see citing works
Briakou, E., Cherry, C., & Foster, G. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.524 · Cited by 23 · see citing works
Proving Test Set Contamination in Black Box Language Models
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2310.17623 · Cited by 11 · see citing works
Investigating Data Contamination in Modern Benchmarks for Large Language Models
Deng, C., Zhao, Y., Tang, X., Gerstein, M., & Cohan, A. (2023) · arXiv · doi:10.48550/arxiv.2311.09783 · Cited by 8 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Flagged first, then by citations
Cited by
Sorted by citations
Hallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182 · see citing works
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 130 · see citing works
A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108189 · Cited by 116 · see citing works
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Harnessing domain insights: A prompt knowledge tuning method for aspect-based sentiment analysis
Sun, X., Zhang, K., Liu, Q., Bao, M., & Chen, Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111975 · Cited by 63 · see citing works
Large Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 47 · see citing works
The sociolinguistic foundations of language modeling
Grieve, J., Bartl, S., Fuoli, M., Grafmiller, J., Huang, W., Jawerbaum, A., Murakami, A., Perlman, M., Rœmling, D., & Winter, B. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1472411 · Cited by 45 · see citing works
ASR Error Correction Using Large Language Models
Ma, R., Qian, M., Gales, M., & Knill, K. M. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3551083 · Cited by 24 · see citing works
Benchmark Data Contamination of Large Language Models: A Survey
Xu, C., Guan, S., Greene, D., & Kechadi, M.-T. (2024) · arXiv · doi:10.48550/arxiv.2406.04244 · Cited by 22 · see citing works
Aligning generalization between humans and machines
Ilievski, F., Hammer, B., van Harmelen, F., Paaßen, B., Saralajew, S., Schmid, U., Biehl, M. L., Bolognesi, M., Dong, X., Gashteovski, K., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01109-4 · Cited by 23 · see citing works
Overloaded minds and machines: a cognitive load framework for human-AI symbiosis
Wang, P., Liu, H., Zou, L., & PAAS, F. G. W. C. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11510-z · Cited by 17 · see citing works
Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
Golchin, S., & Surdeanu, M. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.20 · Cited by 37 · see citing works
Large language models show both individual and collective creativity comparable to humans
Sun, L., Yuan, Y., Yao, Y., Li, Y., Zhang, H., Xie, X., Wang, X., Luo, F., & Stillwell, D. J. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101870 · Cited by 18 · see citing works
Vision of the future: large language models in ophthalmology
Tailor, P. D., D’Souza, H. S., Li, H., & Starr, M. R. (2024) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000001062 · Cited by 10 · see citing works
UNK-VQA: A Dataset and a Probe Into the Abstention Ability of Multi-Modal Large Models
Guo, Y., Jiao, F., Shen, Z., Nie, L., & Kankanhalli, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3437288 · Cited by 10 · see citing works
Thelwall, M., & Mohammadi, E. (2026) · Scientometrics · doi:10.1007/s11192-026-05585-2 · Cited by 7 · see citing works
SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)
Meeus, M., Shilov, I., Jain, S., Faysse, M., Rei, M., & de Montjoye, Y.-A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00028 · Cited by 20 · see citing works
The future of large language models in fighting emerging outbreaks: lights and shadows
Rizzo, A., Mensa, E., & Giacomelli, A. (2024) · The Lancet Microbe · doi:10.1016/j.lanmic.2024.07.017 · Cited by 8 · see citing works
Addressing Data Leakage in HumanEval Using Combinatorial Test Design
Bradbury, J. S., & More, R. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989022 · Cited by 6 · see citing works
Had Enough of Experts? Quantitative Knowledge Retrieval From Large Language Models
Selby, D., Iwashita, Y., Spriestersbach, K., Saad, M., Bappert, D., Warrier, A., Mukherjee, S., Kise, K., & Vollmer, S. J. (2025) · Stat · doi:10.1002/sta4.70054 · Cited by 5 · see citing works
Deng, Y., Qiu, X., Tan, X., Qu, C., Pan, J., Cheng, Y., Xu, Y., & Chu, W. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887899 · Cited by 4 · see citing works
Survey of Computerized Adaptive Testing: A Machine Learning Perspective
Zhuang, Y., Liu, Q., Bi, H., Huang, Z., Huang, W., Li, J., Yu, J., Liu, Z., Hu, Z., Hong, Y., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3672850 · Cited by 9 · see citing works
Benchmarking AI: Toward Inclusive Evaluation of Language Models
Jovanovic, M., & Campbell, M. (2025) · Computer · doi:10.1109/mc.2025.3575880 · Cited by 3 · see citing works
Yakura, H. (2024) · Journal of Intelligence · doi:10.3390/jintelligence12070070 · Cited by 6 · see citing works
CodeCleaner: Mitigating Data Contamination for LLM Benchmarking
Cao, J., Chen, S., Zhang, W., Lo, H. C., Li, Y., & Cheung, S.-C. (2025) · Proceedings of the 16th International Conference on Internetware · doi:10.1145/3755881.3755901 · Cited by 2 · see citing works
The mosaic memory of large language models
Shilov, I., Meeus, M., & de Montjoye, Y.-A. (2026) · Nature Communications · doi:10.1038/s41467-026-68603-0 · Cited by 11 · see citing works
Bao, Q., Gendron, G., Peng, A. Y., Zhong, W., Tan, N. Ö., Chen, Y., Witbrock, M., & Liu, J. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6603-4_22 · Cited by 2 · see citing works
Wang, L., & Law, K. L. E. (2025) · Computer · doi:10.1109/mc.2024.3423378 · Cited by 1 · see citing works
Tong, Z., Sun, F., & Nguyen, L.-M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97144-0_14 · Cited by 1 · see citing works
Wahidi, M., Rizk, A., & Imad, R. (2025) · IEEE Access · doi:10.1109/access.2025.3589514 · Cited by 1 · see citing works
MASA: Multi-view adaptive subspace alignment for enhanced few-shot learning
Wang, X., Liu, J., & Li, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114148 · Cited by 1 · see citing works
Automated and LLM-Assisted Conceptual Modeling from Semi-structured Data
Guimarães, P., Vieira, A., Sousa, V., Moreira, G., & Santos, M. Y. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-12063-2_8 · Cited by 1 · see citing works
Roxas, D. Q., & Emekci, H. (2026) · Journal of Polytechnic · doi:10.2339/politeknik.1719005 · Cited by 2 · see citing works
Aiyappa, R., Senthilmani, S., An, J., Kwak, H., & Ahn, Y. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3540 · Cited by 7 · see citing works
Scoring Structured Academic Documents with Large Language Models: Impact Case Studies
Thelwall, M. A., Kousha, K., & He, G. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2025-0465 · Cited by 1 · see citing works
Silent Failures in Federated Personalization of Foundation Models
Oh, Y., & Bui, A. A. T. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818661 · Cited by 1 · see citing works
Musical heritage historical entity linking
Graciotti, A., Lazzari, N., Presutti, V., & Tripodi, R. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11102-9 · Cited by 1 · see citing works
IDE: A Multi-Agent-Driven Iterative Framework for Dynamic Evaluation of LLMs
Tong, X., Jin, B., Wang, J., Wu, X., Xia, T., & Han, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890123 · Cited by 1 · see citing works
Fung, S.-H., Zhang, Z., Wang, R., Miao, C., Wong, B. S., Li, K. Y., Hong, C., Zhou, J., Yip, K. Y.-L., Tsui, S. K., et al. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.02.11.637772 · Cited by 1 · see citing works
A Comparative Analysis of Retrieval-Augmented Generation and Crowdsourcing for Fact-Checking
Bona, F. B. D., Barbera, D. L., Mizzaro, S., & Roitero, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_44
Parameter Drift as a Signal for Membership Inference in Overfit-Tuned LLMs
Kitamura, T., & Suzuki, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02215-8_23
Task Contamination: Language Models May Not Be Few-Shot Anymore
Association for Artificial Intelligence 2024, Flanigan, J., & Li, C. (2024) · Underline Science Inc. · doi:10.48448/rzmg-6991
Self-citationVolDoGER: LLM-Assisted Datasets for Domain Generalization in Vision-Language Tasks
Choi, J., Kwon, J., Yun, J., Yu, S., & Kim, Y. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00712 · Cited by 1 · see citing works
Self-Refining Topology Optimization via an LLM-Based Multi-Agent FrameworkPreprint
Park, H., & Chung, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7162224
Kirtac, K. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7213792
From Gameplay Traces to Game Mechanics: Causal Induction with Large Language Models
Jiwatode, M., Dockhorn, A., & Rosenhahn, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31927-2_1
Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs
Chen, H., Ho, Y.-H., Huang, T.-Y., Hsu, P.-H., Chen, L.-Y., Lee, C., & Sun, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37041-9_23
Evaluating Gemma 3 27B for local undergraduate physics tutoring
Völschow, M., Buczek, P., Riefer, P., Jasko, A., & Steinbach, P. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02296-8
- Authors
- Li, C., & Flanigan, J.
- Affiliations
- University of California, Santa Cruz (US)
- ORCID
- Flanigan, Jeffrey
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 16
- Pages
- 18471-18480
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393160461
- Cited by
- 47
- References
- 35
- Citations by year
- 2026: 14 · 2025: 24 · 2024: 9
- FWCI
- 13.87
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- other-oa
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1730158, 1540112, 1826967, 2100237, 2112167, 1541349, 2120019)
- University of California, San Diego
- Office of the President, University of California
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, zero-shot learning, few-shot learning, training data contamination, membership inference attacks, training data extraction, GPT-3
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
