How Is ChatGPT’s Behavior Changing Over Time?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4’s amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5’s performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4’s ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our finding that most LLMs consistently produced algorithmic overgeneralizations across multiple occasions adds nuance to prior research that found that, in some domains, LLMs produced unstable or inconsistent responses over time [48–50].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our findings thus offer novel contributions to research on overgeneralization in science communication [17,18,54] and the shortcomings of LLMs in text summarization [48,52,55], revealing a subtle form of ‘hallucination’ that has not yet been documented in the literature.”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Cited by
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
Guidance for generative AI in education and research
(2023) · doi:10.54675/ewzm9535 · Cited by 808
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
Polak, M. P., & Morgan, D. (2024) · Nature Communications · doi:10.1038/s41467-024-45914-8 · Cited by 365
Testing theory of mind in large language models and humans
Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01882-z · Cited by 299
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Larger and more instructable language models become less reliable
Zhou, L., Schellaert, W., Martínez-Plumed, F., Moros-Daval, Y., Ferri, C., & Hernández-Orallo, J. (2024) · Nature · doi:10.1038/s41586-024-07930-y · Cited by 203
Sentiment Analysis in the Age of Generative AI
Krugmann, J. O., & Hartmann, J. (2024) · Customer Needs and Solutions · doi:10.1007/s40547-024-00143-4 · Cited by 189
Beware of botshit: How to manage the epistemic risks of generative chatbots
Hannigan, T. R., McCarthy, I. P., & Spicer, A. (2024) · Business Horizons · doi:10.1016/j.bushor.2024.03.001 · Cited by 224
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 103
Hallucination‐Free? Assessing the Reliability of Leading AI Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182
REFORMS: Consensus-based Recommendations for Machine-learning-based Science
Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C. A., Gundersen, O. E., Hofman, J. M., Hullman, J., Lones, M. A., Malik, M. M., et al. (2024) · Science Advances · doi:10.1126/sciadv.adk3452 · Cited by 129
Prompt engineering as a new 21st century skill
Federiakin, D. A., Molerov, D., Zlatkin‐Troitschanskaia, O., & Maur, A. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1366434 · Cited by 262
Multimodal generative AI for medical image interpretation
Rao, V. M., Hla, M., Moor, M., Adithan, S., Kwak, S., Topol, E. J., & Rajpurkar, P. (2025) · Nature · doi:10.1038/s41586-025-08675-y · Cited by 182
How large language models can reshape collective intelligence
Burton, J. W., Lopez-Lopez, E., Hechtlinger, S., Rahwan, Z., Aeschbach, S., Bakker, M. A., Becker, J. A., Berditchevskaia, A., Berger, J., Brinkmann, L. U. L., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01959-9 · Cited by 243
Generalization bias in large language model summarization of scientific research
Peters, U., & Chin-Yee, B. (2025) · Royal Society Open Science · doi:10.1098/rsos.241776 · Cited by 126
Using ChatGPT for human–computer interaction research: a primer
Tabone, W., & de Winter, J. (2023) · Royal Society Open Science · doi:10.1098/rsos.231053 · Cited by 116
Applications of large language models in cancer care: current evidence and future perspectives
Iannantuono, G. M., Bracken-Clarke, D., Floudas, C. S., Roselli, M., Gulley, J. L., & Karzai, F. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1268915 · Cited by 115
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Performance of large language models on a neurology board-style examination
Schubert, M. C., Wick, W., & Venkataramani, V. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.46721 · Cited by 115
Chatgpt: a meta-analysis after 2.5 months
Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., & Eger, S. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100541 · Cited by 115
Jain, A. (2024) · Current Opinion in Solid State and Materials Science · doi:10.1016/j.cossms.2024.101189 · Cited by 123
Responsible governance of generative AI: conceptualizing GenAI as complex adaptive systems
Janssen, M. (2025) · Policy and Society · doi:10.1093/polsoc/puae040 · Cited by 174
AI hype as a cyber security risk: the moral responsibility of implementing generative AI in business
Humphreys, D., Koay, A., Desmond, D., & Mealy, E. (2024) · AI and Ethics · doi:10.1007/s43681-024-00443-4 · Cited by 98
Deception abilities emerged in large language models
Hagendorff, T. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2317967121 · Cited by 100
Seven Pillars for the Future of Artificial Intelligence
Cambria, E., Mao, R., Chen, M., Wang, Z., & Ho, S.-B. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3329745 · Cited by 138
Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024) · Journal of Medical Internet Research · doi:10.2196/52113 · Cited by 96
Can ChatGPT assist in picking stocks?
Pelster, M., & Val, J. (2023) · Finance research letters · doi:10.1016/j.frl.2023.104786 · Cited by 111
Diversity and Standards in Writing for Publication in the Age of AI—Between a Rock and a Hard Place
Kuteeva, M., & Andersson, M. (2024) · Applied Linguistics · doi:10.1093/applin/amae025 · Cited by 91
Large Language Models for Telecom: Forthcoming Impact on the Industry
Maatouk, A., Piovesan, N., Ayed, F., De Domenico, A., & Debbah, M. (2025) · IEEE Communications Magazine · doi:10.1109/mcom.001.2300473 · Cited by 85
Is GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings
Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1272229 · Cited by 158
Antaki, F., Milad, D., Chia, M. A., Giguère, C.-É., Touma, S., El-Khoury, J., Keane, P. A., & Duval, R. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324438 · Cited by 83
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81
Niszczota, P., & Abbas, S. (2023) · Finance research letters · doi:10.1016/j.frl.2023.104333 · Cited by 126
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76
Artificial Intelligence in Cardiovascular Care—Part 2: Applications
Jain, S. S., Elias, P., Poterucha, T., Randazzo, M., Lopez Jimenez, F., Khera, R., Perez, M., Ouyang, D., Pirruccello, J., Salerno, M., et al. (2024) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2024.03.401 · Cited by 74
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
Dotan, R., Parker, L. S., & Radzilowicz, J. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3659023 · Cited by 71
Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability
Rodriguez, A. D., Dearstyne, K. R., & Cleland‐Huang, J. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00087 · Cited by 121
Tang, Y., Chen, L., Chen, Z., Chen, W., Cai, Y., Du, Y., Yang, F., & Sun, L. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642899 · Cited by 101
Self-citationTran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
Generative AIPreprint
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4443189 · Cited by 65
ChatGPT as a tool for medical education and clinical decision-making on the wards: Case study
Skryd, A., & Lawrence, K. (2024) · JMIR Formative Research · doi:10.2196/51346 · Cited by 63
The path forward for large language models in medicine is open
Riedemann, L., Labonne, M., & Gilbert, S. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01344-w · Cited by 63
Evaluating the Utilities of Foundation Models in Single‐Cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2026) · Advanced Science · doi:10.1002/advs.202514490 · Cited by 61
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62
Generative Artificial Intelligence in Qualitative Data Analysis: Analyzing—Or Just Chatting?
Nguyen, D. C., & Welch, C. (2025) · Organizational Research Methods · doi:10.1177/10944281251377154 · Cited by 124
Large language models in psychiatry: Opportunities and challenges
Volkmer, S., Meyer-Lindenberg, A., & Schwarz, E. (2024) · Psychiatry Research · doi:10.1016/j.psychres.2024.116026 · Cited by 59
Huppertz, M., Siepmann, R. M., Topp, D., Nikoubashman, O., Yüksel, C., Kuhl, C. K., Truhn, D., & Nebelung, S. (2024) · European Radiology · doi:10.1007/s00330-024-11115-6 · Cited by 78
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Khondaker, M. T. I., Waheed, A., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.16 · Cited by 130
Hwang, T., Aggarwal, N., Khan, P. Z., Roberts, T., Mahmood, A., Griffiths, M. M., Parsons, N., & Khan, S. (2024) · PLOS ONE · doi:10.1371/journal.pone.0297701 · Cited by 59
Cambria, E., Zhang, X., Mao, R., Chen, M., & Kwok, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76827-9_11 · Cited by 97
An, J., Huang, D., Lin, C., & Tai, M. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf089 · Cited by 58
Evolving code with a large language model
Hemberg, E., Moskal, S., & O’Reilly, U.-M. (2024) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-024-09494-2 · Cited by 129
Evaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness
Shafee, S., Bessani, A., & Ferreira, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125509 · Cited by 112
Systematic review: The use of large language models as medical chatbots in digestive diseases
Giuffrè, M., Kresevic, S., You, K., Dupont, J., Huebner, J., Grimshaw, A. A., & Shung, D. L. (2024) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.18058 · Cited by 68
Schulhoff, S., Pinto, J., Khan, A., Bouchard, L.-F., Si, C., Anati, S., Tagliabue, V., Kost, A., Carnahan, C., & Boyd-Graber, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.302 · Cited by 54
Hadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 53
Fu, X., Wang, R., & Li, C. (2023) · Journal of the American Planning Association · doi:10.1080/01944363.2023.2271893 · Cited by 64
Enough of the chit-chat: A comparative analysis of four AI chatbots for calculus and statistics
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.2.22 · Cited by 50
Alvero, A. J., Lee, J., Regla-Vargas, A., Kizilcec, R. F., Joachims, T., & Antonio, A. L. (2024) · Journal of Big Data · doi:10.1186/s40537-024-00986-7 · Cited by 48
Su, M.-C., Lin, L.-E., Lin, L.-H., & Chen, Y. (2024) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2024.104717 · Cited by 54
The Foundation Model Transparency Index
Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., Zhang, D., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2310.12941 · Cited by 46
Assessing the Performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination
Kaneda, Y., Takahashi, R., Kaneda, U., Akashima, S., Okita, H., Misaki, S., Yamashiro, A., Ozaki, A., & Tanimoto, T. (2023) · Cureus · doi:10.7759/cureus.42924 · Cited by 45
CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Tools
Ni, J., Bingler, J., Senni, C. C., Kraus, M., Gostlow, G., Schimanski, T., Stammbach, D., Vaghefi, S. A., Wang, Q., Webersinke, N., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.3 · Cited by 85
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 66
Leveraging AI to improve evidence synthesis in conservation
Berger‐Tal, O., Wong, B. B. M., Adams, C. A., Blumstein, D. T., Candolin, U., Gibson, M. J., Greggor, A. L., Lagisz, M., Macura, B., Price, C. J., et al. (2024) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2024.04.007 · Cited by 58
Malgieri, G., & Pasquale, F. (2023) · Computer law & security review · doi:10.1016/j.clsr.2023.105899 · Cited by 88
Heilmeyer, F., Böhringer, D., Reinhard, T., Arens, S., Lyssenko, L., & Haverkamp, C. (2024) · JMIR Medical Informatics · doi:10.2196/59617 · Cited by 49
Red teaming ChatGPT in medicine to yield real-world insights on model behavior
Chang, C. T., Farah, H., Gui, H., Rezaei, S. J., Bou-Khalil, C., Park, Y., Swaminathan, A., Omiye, J. A., Kolluri, A., Chaurasia, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01542-0 · Cited by 77
Benefit-Risk Reporting for FDA-Cleared Artificial Intelligence−Enabled Medical Devices
Lin, J. C., Jain, B., Iyer, J. M., Rola, I., Srinivasan, A. R., Kang, C., Patel, H., & Parikh, R. B. (2025) · JAMA Health Forum · doi:10.1001/jamahealthforum.2025.3351 · Cited by 42
Trustworthy and Synergistic Artificial Intelligence for Software Engineering: Vision and Roadmaps
Lo, D. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00010 · Cited by 40
Gandolfi, A. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00403-3 · Cited by 69
GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4
Kocmi, T., & Federmann, C. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.64 · Cited by 313
Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4
Pelrine, K., Imouza, A., Thibault, C., Reksoprodjo, M., Gupta, C., Christoph, J. N., Godbout, J., & Rabbany, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.395 · Cited by 131
Enhancing Postmarketing Surveillance of Medical Products With Large Language Models
Matheny, M. E., Yang, J., Smith, J. C., Walsh, C. G., Al-Garadi, M. A., Davis, S. E., Marsolo, K. A., Fabbri, D., Reeves, R. R., Johnson, K. B., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.28276 · Cited by 38
Large language models as oracles for instantiating ontologies with domain-specific knowledge
Ciatto, G., Agiollo, A., Magnini, M., & Omicini, A. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112940 · Cited by 44
Prompting or Fine-tuning? A Comparative Study of Large Language Models for Taxonomy Construction
Chen, B., Yi, F., & Varró, D. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1109/models-c59198.2023.00097 · Cited by 80
Gozali, I., Wijaya, A. R. T., Lie, A., Cahyono, B. Y., & Suryati, N. (2024) · The JALT CALL Journal · doi:10.29140/jaltcall.v20n1.1200 · Cited by 97
Griewing, S., Knitza, J., Boekhoff, J., Hillen, C., Lechner, F., Wagner, U., Wallwiener, M., & Kuhn, S. (2024) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-024-07565-4 · Cited by 38
Prompt architecture induces methodological artifacts in large language models
Brucks, M. S., & Toubia, O. (2025) · PLoS ONE · doi:10.1371/journal.pone.0319159 · Cited by 91
Siepmann, R. M., Huppertz, M., Rastkhiz, A., Reen, M., Corban, E., Schmidt, C., Wilke, S., Schad, P., Yüksel, C., Kuhl, C. K., et al. (2024) · European Radiology · doi:10.1007/s00330-024-10727-2 · Cited by 43
Uppal, K., & Hajian, S. (2024) · European Journal of Educational Research · doi:10.12973/eu-jer.14.1.199 · Cited by 36
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
Koga, S. (2023) · Pathology International · doi:10.1111/pin.13382 · Cited by 41
Saibene, A. M., Allevi, F., Calvo‐Henríquez, C., Maniaci, A., Mayo‐Yáñez, M., Paderno, A., Vaira, L. A., Felisati, G., & Craig, J. R. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08372-4 · Cited by 39
Demir, G. B., Süküt, Y., Duran, G. S., Topsakal, K. G., & Görgülü, S. (2024) · European Journal of Orthodontics · doi:10.1093/ejo/cjae011 · Cited by 33
Disinformation Capabilities of Large Language Models
Vykopal, I., Pikuliak, M., Srba, I., Moro, R., Macko, D., & Bielikova, M. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.793 · Cited by 34
ArcheType: A Novel Framework for Open-Source Column Type Annotation Using Large Language Models
Feuer, B., Liu, Y., Hegde, C., & Freire, J. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3665844.3665857 · Cited by 72
KnowNet:Guided Health Information Seeking from LLMs via Knowledge Graph Integration
Yan, Y., Hou, Y., Xiao, Y., Zhang, R., & Wang, Q. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456364 · Cited by 66
Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Research
Lin, H., & Zhang, Y. (2026) · Social Science Computer Review · doi:10.1177/08944393251366243 · Cited by 33
Student (Mis)Use of Generative AI Tools for University-Related Tasks
Reiter, L., Jörling, M., Fuchs, C., Working group ‘Artificial Intelligence in Higher Education’, & Böhm, R. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2462083 · Cited by 65
Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
Liang, J. T., Lin, M. M., Rao, N., & Myers, B. A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729342 · Cited by 73
Bioinformatics and biomedical informatics with ChatGPT: Year one review
Wang, J., Cheng, Z., Yao, Q., Liu, L., Xu, D. H., & Hu, G. (2024) · Quantitative Biology · doi:10.1002/qub2.67 · Cited by 41
SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents
Lin, F., Kim, D. J., & Chen, T.-H. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00140 · Cited by 32
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0 · Cited by 68
Peng, Z., Ma, R., Zhang, Y., Yan, M., Lu, J. L., Cheng, Q., Liao, J., Zhang, Y., Wang, J., Zhao, Y., et al. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1323924 · Cited by 32
The Application of Large Language Models for Radiologic Decision Making
Zaki, H. A., Aoun, A., Munshi, S., Abdel-Megid, H., Nazario-Johnson, L., & Ahn, S. H. (2024) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2024.01.007 · Cited by 38
Agostini, D., & Picasso, F. (2024) · Intelligenza Artificiale · doi:10.3233/ia-240033 · Cited by 46
Metadata
- Authors
- Chen, L., Zaharia, M., & Zou, J.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- Harvard Data Science Review
- Publisher
- MIT Press
- Published
- 2024-03-12
- Volume
- 6
- Issue
- 2
- ISSN
- 2644-2353 (ISSN-L)
- OpenAlex
- W4392681182
- Cited by
- 507
- References
- 0
- Citations by year
- 2026: 128 · 2025: 139 · 2024: 163
- FWCI
- 63.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Harvard Data Science Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, GPT-3.5, GPT-4, model drift, model updating, chain-of-thought prompting, opinion surveys, multi-hop question answering, code generation, visual reasoning, medical licensing examinations, instruction following
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.