How Is ChatGPT’s Behavior Changing Over Time?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4’s amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5’s performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4’s ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our finding that most LLMs consistently produced algorithmic overgeneralizations across multiple occasions adds nuance to prior research that found that, in some domains, LLMs produced unstable or inconsistent responses over time [48–50].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our findings thus offer novel contributions to research on overgeneralization in science communication [17,18,54] and the shortcomings of LLMs in text summarization [48,52,55], revealing a subtle form of ‘hallucination’ that has not yet been documented in the literature.”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Cited by
Items 101–200 of 517
Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction
Kang, S., Yoon, J., Askarbekkyzy, N., & Yoo, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3450837 · Cited by 55
On the challenges of using black-box APIs for toxicity evaluation in research
Pozzobon, L., Ermis, B., Lewis, P., & Hooker, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.472 · Cited by 29
Leveraging GPT-like LLMs to Automate Issue Labeling
Colavito, G., Lanubile, F., Novielli, N., & Quaranta, L. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644903 · Cited by 58
Metadata integrity in bioinformatics: Bridging the gap between data and knowledge
Caliskan, A., Dangwal, S., & Dandekar, T. (2023) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2023.10.006 · Cited by 32
Yudovich, M., Makarova, E., Hague, C. M., & Raman, J. D. (2024) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2024.21.17 · Cited by 32
Conjuring algorithms: Understanding the tech industry as stage magicians
Nagy, P., & Neff, G. (2024) · New Media & Society · doi:10.1177/14614448241251789 · Cited by 26
Emerging trends: a gentle introduction to RAG
Church, K., Sun, J., Yue, R., Vickers, P., Saba, W., & Chandrasekar, R. (2024) · Natural Language Engineering · doi:10.1017/s1351324924000044 · Cited by 45
ChatGPT as a Stable and Fair Tool for Automated Essay Scoring
García-Varela, F., Nussbaum, M., Mendoza, M., Martínez, C., & Bekerman, Z. (2025) · Education Sciences · doi:10.3390/educsci15080946 · Cited by 48
Evaluating AI and human authorship quality in academic writing through physics essays
Yeadon, W., Agra, E., Inyang, O.-O., Mackay, P., & Mizouri, A. (2024) · European Journal of Physics · doi:10.1088/1361-6404/ad669d · Cited by 26
Danish university policies on generative AI: Problems, assumptions and sustainability blind spots
Driessens, O., & Pischetola, M. (2024) · MedieKultur: Journal of media and communication research · doi:10.7146/mk.v40i76.143595 · Cited by 25
Embracing Generative Artificial Intelligence in Clinical Research and Beyond
Foote, H. P., Hong, C., Anwar, M. M., Borentain, M., Bugin, K., Dreyer, N. A., Fessel, J. P., Goyal, N., Hanger, M., Hernandez, A. F., et al. (2025) · JACC Advances · doi:10.1016/j.jacadv.2025.101593 · Cited by 42
Herold, S., Heller, J., Rozemeijer, F. A., & Mahr, D. (2025) · Journal of Purchasing and Supply Management · doi:10.1016/j.pursup.2025.101012 · Cited by 44
van Duijn, M., van Dijk, B., Kouwenhoven, T., de Valk, W., Spruit, M., & vanderPutten, P. (2023) · Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2023.conll-1.25 · Cited by 24
Andersson, M., & McIntyre, D. (2025) · Journal of Pragmatics · doi:10.1016/j.pragma.2025.02.001 · Cited by 45
Haslberger, M., Gingrich, J. R., & Bhatia, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4594466 · Cited by 49
Human innovation and the creative agency of the world in the age of generative AI
Peschl, M. F. (2024) · Possibility Studies & Society · doi:10.1177/27538699241238049 · Cited by 30
Automated electrosynthesis reaction mining with multimodal large language models (MLLMs)
Leong, S. X., Pablo‐García, S., Zhang, Z., & Aspuru‐Guzik, A. (2024) · Chemical Science · doi:10.1039/d4sc04630g · Cited by 40
Mitigating bias in clinical machine learning models
Perez-Downes, J. C., Tseng, A. S., McConn, K. A., Elattar, S. M., Sokumbi, O., Sebro, R. A., Allyse, M. A., Dangott, B. J., Carter, R. E., & Adedinsewo, D. (2024) · Current Treatment Options in Cardiovascular Medicine · doi:10.1007/s11936-023-01032-0 · Cited by 23
Computational reproducibility in computational social science
Schoch, D., Chan, C.-H., Wagner, C., & Bleier, A. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00514-w · Cited by 40
Moorhouse, B. L., Consoli, S., & Curle, S. M. (2025) · Applied Linguistics Review · doi:10.1515/applirev-2025-0021 · Cited by 30
Collective intelligence for AI-assisted chemical synthesis
Li, H., Sarkar, S., Lu, W., Loftus, P. O., Qiu, T., Shee, Y., Cuomo, A. E., Webster, J.-P., Kelly, H. R., Manee, V., et al. (2026) · Nature · doi:10.1038/s41586-026-10131-4 · Cited by 41
Gao, Z., Li, L., Ma, S., Wang, Q., Hemphill, L., & Xu, R. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03385-w · Cited by 33
(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs
Ma, W., Yang, C., & Kästner, C. (2024) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3644815.3644950 · Cited by 61
Frenkel, M., & Emara, H. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22781 · Cited by 40
Eisele‐Metzger, A., Lieberum, J.-L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Böhringer, D., & Meerpohl, J. J. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.12 · Cited by 32
Teaching accounting in the era of ChatGPT – The student perspective
Sundkvist, C. H., & Kulset, E. M. (2024) · Journal of Accounting Education · doi:10.1016/j.jaccedu.2024.100932 · Cited by 77
Generative AI: Overview, Economic Impact, and Applications in Asset ManagementPreprint
Luk, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574814 · Cited by 26
Questionable practices in machine learning
Leech, G., Vazquez, J. J., Kupper, N., Yagudin, M., & Aitchison, L. (2024) · arXiv · doi:10.48550/arxiv.2407.12220 · Cited by 21
The rise of large language models: challenges for Critical Discourse Studies
Gillings, M., Kohn, T., & Mautner, G. (2024) · Critical Discourse Studies · doi:10.1080/17405904.2024.2373733 · Cited by 50
Büker, M., Sümbüllü, M., & Arslan, H. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.100920 · Cited by 27
Chen, Z., Shamsabadi, E. A., Jiang, S., Shen, L., & Dias‐da‐Costa, D. (2026) · Nature Communications · doi:10.1038/s41467-025-68216-z · Cited by 46
Cox, S. R., & Ooi, W. T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54975-5_10 · Cited by 57
Li, J., Li, L., Liu, J., Yu, X., Liu, X., & Keung, J. (2024) · Software Practice and Experience · doi:10.1002/spe.3360 · Cited by 27
Coding with AI in the Physical Chemistry Laboratory
Hare, P. M. (2024) · Journal of Chemical Education · doi:10.1021/acs.jchemed.4c00560 · Cited by 27
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0.2 · Cited by 24
Hayes, D. S., Foster, B. K., Makar, G. S., Manzar, S., Ozdag, Y., Shultz, M. J., Klena, J. C., & Grandizio, L. C. (2024) · Journal of surgical education · doi:10.1016/j.jsurg.2024.08.002 · Cited by 26
Stock portfolio selection based on risk appetite: Evidence from ChatGPT
Schneider, C. J., & Yilmaz, Y. (2025) · Finance research letters · doi:10.1016/j.frl.2025.107517 · Cited by 26
Towards Evaluation Guidelines for Empirical Studies Involving LLMs
Wagner, S., Barón, M. M., Falessi, D., & Baltes, S. (2025) · IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE) · doi:10.1109/wsese66602.2025.00011 · Cited by 54
NORMSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly
Fung, Y. R., Chakrabarty, T., Guo, H., Rambow, O., Muresan, S., & Ji, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.941 · Cited by 91
Wikipedia and AI: Access, representation, and advocacy in the age of large language models
McDowell, Z. J. (2024) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565241238924 · Cited by 29
Easy-read and large language models: on the ethical dimensions of LLM-based text simplification
Freyer, N., Kempt, H., & Klöser, L. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09792-4 · Cited by 38
Polverini, G., & Gregorcic, B. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1452414 · Cited by 29
Jiang, Y., Ko-Wong, L., & Gutierrez, I. V. (2025) · Educational Researcher · doi:10.3102/0013189x251314821 · Cited by 29
Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
Šakota, M., Peyrard, M., & West, R. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635825 · Cited by 142
van Dijk, B., Kouwenhoven, T., Spruit, M. R., & van Duijn, M. J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.779 · Cited by 57
How to Regulate Large Language Models for Responsible AI
Berengueres, J. O. L. (2024) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2024.3403681 · Cited by 30
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 245
Simplifying software compliance: AI technologies in drafting technical documentation for the AI Act
Sovrano, F., Hine, E., Anzolut, S., & Bacchelli, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10645-x · Cited by 36
A perspective for adapting generalist AI to specialized medical AI applications and their challenges
Wang, Z., Wang, H., Danek, B. P., Li, Y., Mack, C. D., Arbuckle, L., Biswal, D., Poon, H., Wang, Y., Rajpurkar, P., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01789-7 · Cited by 51
Generative AI as a lab partner: A case study
Kilde-Westberg, S., Johansson, A., & Enger, J. (2025) · Physical Review Physics Education Research · doi:10.1103/ggy1-3kjk · Cited by 31
A Prompting Framework to Enhance Language Model Output
Ratnayake, H., & Wang, C. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8391-9_6 · Cited by 20
Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments
Rigaki, M., Lukáš, O., Catania, C., & Garcia, S. (2024) · Proceedings of the 16th International Conference on Agents and Artificial Intelligence · doi:10.5220/0012391800003636 · Cited by 17
Young, R. A., Martin, C. M., Sturmberg, J. P., Hall, S., Bazemore, A., Kakadiaris, I. A., & Lin, S. (2024) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.2023.230219r1 · Cited by 17
Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam
Mendonça, N. C. (2024) · ACM Transactions on Computing Education · doi:10.1145/3674149 · Cited by 26
Stability Analysis of ChatGPT-Based Sentiment Analysis in AI Quality Assurance
Ouyang, T., MaungMaung, A., Konishi, K., Seo, Y., & Echizen, I. (2024) · Electronics · doi:10.3390/electronics13245043 · Cited by 27
ChatGPT’s performance in university admissions tests in mathematics
Udías, Á., Alonso‐Ayuso, A., Álfaro, C., Algar, M. J., Cuesta, M., Fernández-Isabel, A., Gómez, J., Lancho, C., Cano, E. L., de Diego, I. M., et al. (2024) · International Electronic Journal of Mathematics Education · doi:10.29333/iejme/15517 · Cited by 32
Ćirković, A., & Katz, T. R. (2023) · JMIR Formative Research · doi:10.2196/51798 · Cited by 22
ConnectGPT: Connect Large Language Models with Connected and Automated Vehicles
Tong, K., & Solmaz, S. (2024) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv55156.2024.10588835 · Cited by 31
Hsieh, C., Hsieh, H.-Y., & Lin, H. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e34851 · Cited by 20
Generative AI: Applications, Models, Challenges, Opportunities, and Future Directions
Madaan, G., Asthana, S. K., & Kaur, J. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-8557-9.ch004 · Cited by 25
Challenges for implementing generative artificial intelligence ( GenAI ) into clinical healthcare
Roberts, L. J., Jayasena, R., Khanna, S., Arnott, L., Lane, P., & Bain, C. (2025) · Internal Medicine Journal · doi:10.1111/imj.70035 · Cited by 22
A Review of Prompt Engineering Techniques for Large Language Models
Naser, M. Z. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2607553 · Cited by 25
Iterative Prompt Refinement for Mining Gene Relationships from ChatGPT
Chen, Y., Gao, J., Petruc, M., Hammer, R., Popescu, M., & Xu, D. H. (2024) · International Journal of Artificial Intelligence and Robotics Research · doi:10.1142/s2972335324500054 · Cited by 16
Towards Robust QA Evaluation via Open LLMs
Kamalloo, E., Upadhyay, S., & Lin, J. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657675 · Cited by 15
Leveraging Transfer Learning for Hate Speech Detection in Portuguese Social Media Posts
Ramos, G., Batista, F., Ribeiro, R., Fialho, P., Moro, S., Fonseca, A. J. F., Guerra, R., Carvalho, P. C. Q. D. F., Marques, C. M. V. A., & Silva, C. (2024) · IEEE Access · doi:10.1109/access.2024.3430848 · Cited by 19
Du, W., Jin, X., Harris, J. C., Brunetti, A., Johnson, E., Leung, O. J., Li, X., Walle, S., Yu, Q., Zhou, X. L., et al. (2024) · Annals of Diagnostic Pathology · doi:10.1016/j.anndiagpath.2024.152392 · Cited by 20
Kjell, O., Kjell, K., & Schwartz, H. A. (2023) · doi:10.31234/osf.io/yfd8g · Cited by 15
An ontology-based text mining dataset for extraction of process-structure-property entities
Durmaz, A. R., Thomas, A., Mishra, L., Murthy, R. N., & Straub, T. (2024) · Scientific Data · doi:10.1038/s41597-024-03926-5 · Cited by 27
Toward non-human-centered design: designing an academic article with ChatGPT
LİVBERBER, T. (2023) · El Profesional de la Informacion · doi:10.3145/epi.2023.sep.12 · Cited by 30
Gamble, J. L., Ferguson, D., Yuen, J., & Sheikh, A. M. (2023) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231218250 · Cited by 16
ChatGPT M.D.: Is there any room for generative AI in neurology?
Nógrádi, B., Polgár, T. F., Meszlényi, V., Kádár, Z., Hertelendy, P., Csáti, A., Szpisjak, L., Halmi, D., Erdélyi-Furka, B., Tóth, M., et al. (2024) · PLoS ONE · doi:10.1371/journal.pone.0310028 · Cited by 23
Mete, U., & Özmen, Ö. A. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-09039-4 · Cited by 17
Stock Market Forecasting: From Traditional Predictive Models to Large Language Models
Darwish, M., Hassanien, E. E., & Eissa, A. H. B. (2025) · Computational Economics · doi:10.1007/s10614-025-11024-w · Cited by 29
Hilbert, M., Thakur, A., Flores, P. M., Zhang, X., Bhan, J. Y., Bernhard, P., & Ji, F. (2023) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2023.102743 · Cited by 30
Supporting High-Level to Low-Level Requirements Coverage Reviewing with Large Language Models
Preda, A.-R., Mayr‐Dorn, C., Mashkoor, A., & Egyed, A. F. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644922 · Cited by 27
Tran, V.-T., Gartlehner, G., Yaacoub, S. G., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Aboulayeh, F., Afach, S., Meerpohl, J. J., et al. (2023) · medRxiv · doi:10.1101/2023.12.15.23300018 · Cited by 13
Klein, S. H., Papies, D., & Utz, S. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2351707 · Cited by 23
What is it like to be a bot? The world according to GPT-4
Lloyd, D. E. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1292675 · Cited by 18
Investigating the Efficacy of ChatGPT-3.5 for Tutoring in Chinese Elementary Education Settings
Bai, Y., Li, J., Shen, J., & Zhao, L. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3464560 · Cited by 18
NGUYEN, D. L., Kim, G. H. J., & Bedayat, A. (2025) · Clinical Imaging · doi:10.1016/j.clinimag.2025.110551 · Cited by 16
GPT as a Financial AdvisorPreprint
Niszczota, P., & Abbas, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4384861 · Cited by 12
Speer, A. B., Oswald, F. L., & Putka, D. J. (2025) · Organizational Research Methods · doi:10.1177/10944281251346404 · Cited by 18
Classifying Sentiments on Social Media Texts: A GPT-4 Preliminary Study
Maceda, L. L., Llovido, J. L., Artiaga, M. B., & Abisado, M. (2023) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3639233.3639353 · Cited by 22
Conceptualizing digital service: coconstitutive essence and value cocreation dynamics
Vial, G., & Grange, C. (2024) · Journal of service management · doi:10.1108/josm-12-2023-0520 · Cited by 13
An Adaptive Pricing Framework for Real-Time AI Model Service Exchange
Gao, J., Wang, Z., & Wei, X. (2024) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2024.3432917 · Cited by 14
Introducing ChatSQC: Enhancing statistical quality control with augmented AI
Megahed, F. M., Chen, Y., Zwetsloot, I. M., Knoth, S., Montgomery, D. C., & Jones‐Farmer, L. A. (2024) · Journal of Quality Technology · doi:10.1080/00224065.2024.2372328 · Cited by 29
Automated Text Scoring in the Age of Generative AI for the GPU-poor
Ormerod, C. M., & Kwako, A. (2024) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/okuu1904 · Cited by 24
On Sarcasm Detection with OpenAI GPT-Based Models
Gole, M., Nwadiugwu, W., & Miranskyy, A. V. (2024) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon62161.2024.10837875 · Cited by 37
Assessing the Evolution of LLM Capabilities for Knowledge Graph Engineering in 2023
Frey, J., Meyer, L., Brei, F., Gründer-Fahrer, S., & Martin, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-78952-6_5 · Cited by 20
Natural language processing in veterinary pathology: A review
Stimmer, L., Kuiper, R., Polledo, L., Ressel, L., Rodriguez, J. M. M., Veiga, I. B., Williams, J. M., & Herder, V. (2025) · Veterinary Pathology · doi:10.1177/03009858251347529 · Cited by 12
Temporal evolution of large language models (LLMs) in oncology
Qiu, Z., Jiang, A., Chang, Q., Gan, W., Zhu, L., Mou, W., Zeng, D., Xiao, M., Chu, G., Peng, S., et al. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-07227-2 · Cited by 17
Vision of the future: large language models in ophthalmology
Tailor, P. D., D’Souza, H. S., Li, H., & Starr, M. R. (2024) · Current Opinion in Ophthalmology · doi:10.1097/icu.0000000000001062 · Cited by 12
Mesquiti, S., Seraj, S., Weyland, A., Ashokkumar, A., Boyd, R. L., Mihalcea, R. F., & Pennebaker, J. W. (2025) · Scientific Reports · doi:10.1038/s41598-025-89165-z · Cited by 17
Lauderdale, S. A., Schmitt, R., Wuckovich, B., Dalal, N., Desai, H., & Tomlinson, S. (2025) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2025.1544951 · Cited by 14
Rejithkumar, G., & Anish, P. R. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00031 · Cited by 22
Language models for drug–drug interactions: current applications, pitfalls, and future directions
Meslamani, A. Z. A., & Hajal, A. A. (2025) · Expert Opinion on Drug Metabolism & Toxicology · doi:10.1080/17425255.2025.2551724 · Cited by 16
Artificial Intelligence in Childcare: Assessing the Performance and Acceptance of ChatGPT Responses
Kaneda, Y., Namba, M., Kaneda, U., & Tanimoto, T. E. (2023) · Cureus · doi:10.7759/cureus.44484 · Cited by 10
Polak, M. P., & Morgan, D. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05352 · Cited by 10
Yadav, P. D., Madhukar, V., Dhamija, A., & Manrai, R. (2025) · Cogent Social Sciences · doi:10.1080/23311886.2025.2476044 · Cited by 13
Metadata
- Authors
- Chen, L., Zaharia, M., & Zou, J.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- Harvard Data Science Review
- Publisher
- MIT Press
- Published
- 2024-03-12
- Volume
- 6
- Issue
- 2
- ISSN
- 2644-2353 (ISSN-L)
- OpenAlex
- W4392681182
- Cited by
- 507
- References
- 0
- Citations by year
- 2026: 128 · 2025: 139 · 2024: 163
- FWCI
- 63.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Harvard Data Science Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, GPT-3.5, GPT-4, model drift, model updating, chain-of-thought prompting, opinion surveys, multi-hop question answering, code generation, visual reasoning, medical licensing examinations, instruction following
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.