How Is ChatGPT’s Behavior Changing Over Time?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4’s amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5’s performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4’s ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our finding that most LLMs consistently produced algorithmic overgeneralizations across multiple occasions adds nuance to prior research that found that, in some domains, LLMs produced unstable or inconsistent responses over time [48–50].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our findings thus offer novel contributions to research on overgeneralization in science communication [17,18,54] and the shortcomings of LLMs in text summarization [48,52,55], revealing a subtle form of ‘hallucination’ that has not yet been documented in the literature.”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Cited by
Items 201–300 of 517
Lauderdale, S. A., Griffin, S. A., Lahman, K. R., Mbaba, E., & Tomlinson, S. (2025) · Personality and Mental Health · doi:10.1002/pmh.70018 · Cited by 13
Nordmann, K., Sauter, S., Stein, M., Aigner, J., Redlich, M.-C., Schaller, M., & Fischer, F. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02532-2 · Cited by 12
Analyzing the dependability of Large Language Models for code clone generation
Eagal, A., Stolee, K. T., & Ore, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112548 · Cited by 14
Al-Jebrni, A. H., Jiang, Z., Qin, Y., Qiu, Y., Jiang, N., Li, G., Zheng, Y., Wong, T. Y., Kim, S., Jung, Y., et al. (2026) · Computational Visual Media · doi:10.26599/cvm.2026.9450530 · Cited by 11
Red Teaming Large Language Models in Medicine: Real-World Insights on Model BehaviorPreprint
Chang, C. T., Farah, H., Gui, H., Rezaei, S. J., Bou-Khalil, C., Park, Y., Swaminathan, A., Omiye, J. A., Kolluri, A., Chaurasia, A. R., et al. (2024) · medRxiv · doi:10.1101/2024.04.05.24305411 · Cited by 9
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
Humans vs. ChatGPT: Evaluating Annotation Methods for Financial Corpora
Kaikaus, J., Li, H., & Brunner, R. J. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386425 · Cited by 14
GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants
Hamid, A., Samidi, H. R., Pappachan, P., Finin, T., & Yus, R. (2024) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0081 · Cited by 27
What is in your LLM-based framework?
(2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00896-6 · Cited by 9
Covington, N. V., & Vruwink, O. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00427-9 · Cited by 12
Artificial Intelligence in Physics Courses to Support Active Learning
Robledo-Rella, V., & Toh, B.-Y. (2024) · International Conference e-Society, e-Learning and e-Technologies · doi:10.1145/3678610.3678631 · Cited by 21
Stepchenkova, S. O., Kirilenko, A., & Jing, Y. (2024) · Journal of Destination Marketing & Management · doi:10.1016/j.jdmm.2024.100957 · Cited by 14
Tinnes, C., Welter, A., & Apel, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00112 · Cited by 20
Bridging the Green Attitude–Behavior Gap
Fabio, R. A., Croce, A., & Calabrese, C. (2025) · Journal of Sustainability Research · doi:10.20900/jsr20250059 · Cited by 27
Lew, M. W., Pozhitkov, A. E., Rossi, L. A., Raytis, J. L., & Kidambi, T. D. (2023) · Cureus · doi:10.7759/cureus.47155 · Cited by 9
Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic Systems
Oltramari, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27701 · Cited by 9
Does ChatGPT Help Us Understand the Medical Literature?
Lemley, K. V. (2023) · Journal of the American Society of Nephrology · doi:10.1681/asn.0000000000000237 · Cited by 8
The Heterogeneous Productivity Effects of Generative AI *
Kreitmeir, D., & Raschky, P. (2024) · doi:10.2139/ssrn.4745624 · Cited by 5
Catching Silent Failures: A Machine Learning Model Monitoring and Explainability Survey
Karval, R., & Singh, K. N. (2023) · OITS International Conference on Information Technology (OCIT) · doi:10.1109/ocit59427.2023.10431343 · Cited by 12
AI ‐Powered Laryngoscopy: Exploring the Future With Google Gemini
Setzen, S. A., Andreadis, K., Elemento, O., & Rameau, A. (2025) · The Laryngoscope · doi:10.1002/lary.32089 · Cited by 11
Hierarchical Index Retrieval-Driven Wireless Network Intent Translation With LLM
Wang, J., Guo, L., Wu, J., Yan, C., Sun, H., Zhang, L., Zhuang, Z., Qi, Q., & Liao, J. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3564937 · Cited by 11
Zhang, Y., Chen, T. Y., Pike, M., Towey, D., Ying, Z., & Zhou, Z. Q. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107828 · Cited by 11
Stylometric analysis of AI-generated texts: a comparative study of ChatGPT and DeepSeek
Jaashan, H. M. S., & Bin-Hady, W. R. A. (2025) · Cogent Arts and Humanities · doi:10.1080/23311983.2025.2553162 · Cited by 16
Bornea, A.-L., Ayed, F., Domenico, A. D., Piovesan, N., Salem, T., & Maatouk, A. (2025) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2025.3641573 · Cited by 14
Large language models for porous materials: from text mining to autonomous laboratory
Han, S., Bae, T., Kim, J., Kim, Y.-H., & Kim, J. (2026) · Digital Discovery · doi:10.1039/d5dd00578g · Cited by 10
Impacto de la inteligencia artificial generativa en la creatividad de los estudiantes universitarios
Mendoza, G. E. V., Merino, R. D. L. B., Trujillo, S. C. L., & Peña, P. M. (2024) · Technology rain journa · doi:10.55204/trj.v3i1.e33 · Cited by 33
Meter, A. V., Wheaton, M. G., Cosgrove, V. E., Andreadis, K., & Robertson, R. E. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000711 · Cited by 8
ChatGPT and academic work: new psychological phenomena
de Winter, J., Hancock, P. A., & Eisma, Y. B. (2025) · AI & Society · doi:10.1007/s00146-025-02241-w · Cited by 9
Evaluating the Utilities of Large Language Models in Single-cell Data AnalysisPreprint
Zhao, H., Liu, T., Li, K., Wang, Y., & Li, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3376641/v1 · Cited by 16
Generative artificial intelligence in finance
OECD (2023) · OECD artificial intelligence papers · doi:10.1787/ac7149cc-en · Cited by 22
Cassell, K., Ologunowa, A., Rastegar-Mojarad, M., Chun, B., Huang, Y.-L., Wang, D., & Cossrow, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662202 · Cited by 15
Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 11
Pandita, A., Keniston, A., & Madhuripan, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01658-3 · Cited by 14
Chen, H., Zhao, J., Zheng, S., Zhang, X., Duan, H., & Lü, X. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02674-3 · Cited by 8
Huynh, L., & McNamara, D. S. (2025) · Applied Sciences · doi:10.3390/app152212128 · Cited by 13
Digital prejudices: an analysis of gender, racial and religious biases in generative AI chatbots
Chua, A. Y. K., Chen, M., Kan, M., & Seoh, W. (2025) · Internet Research · doi:10.1108/intr-10-2024-1536 · Cited by 12
Software refactoring research with large language models: A systematic literature review
Martinez, S., Xu, L., Elnaggar, M., & AlOmar, E. A. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112762 · Cited by 20
When employees meet gen AI: The double-edged effects of AI attitude on creativity
Ma, L., Ge, L., Li, Y., & Wang, Y. (2026) · Technovation · doi:10.1016/j.technovation.2026.103495 · Cited by 7
Basaran, A. E., Güresir, Á., Knoch, H., Vychopen, M., Güresir, E., & Wach, J. (2025) · Neurosurgical Review · doi:10.1007/s10143-025-03194-w · Cited by 11
LLMs are not ready for editorial work
Lindsay, G. W. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01730-6 · Cited by 8
Generative AI: Impactful Considerations to Responsible Data Practices in Business Execution
Osmëni, T., & Ali, M. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384863 · Cited by 12
Yosifova, V. (2024) · Preprints.org · doi:10.20944/preprints202405.1169.v1 · Cited by 8
Political Biases of ChatGPT in Different LanguagesPreprint
Retzlaff, N. (2024) · Preprints.org · doi:10.20944/preprints202406.1224.v1 · Cited by 13
Sørensen, C. B., Gram‐Hanssen, A., Rosenberg, J., & Baker, J. J. (2025) · Cureus · doi:10.7759/cureus.82525 · Cited by 6
Schrödinger's Update: User Perceptions of Uncertainties in Proprietary Large Language Model Updates
Ma, Z., Mei, Y., Gajos, K. Z., & Arawjo, I. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3651100 · Cited by 16
Generation of Robot Manipulation Plans Using Generative Large Language Models
Töberg, J.-P., Frese, P., & Cimiano, P. (2025) · International Journal of Semantic Computing · doi:10.1142/s1793351x25410041 · Cited by 5
Comparative Evaluation of Artificial Intelligence Models for Contraceptive Counseling
Patel, A. V., Jasani, S., AlAshqar, A., Doshi, R. H., Amin, K., Panakam, A., Patil, A., & Sheth, S. S. (2025) · Digital · doi:10.3390/digital5020010 · Cited by 5
Safrai, M., & Azaria, A. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302217 · Cited by 9
Towards a Conversational Ethics of Large Language Models
Kempt, H., Lavie, A., & Nagel, S. K. (2024) · American Philosophical Quarterly · doi:10.5406/21521123.61.4.04 · Cited by 9
Lim, G., & Perrault, S. T. (2024) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3678884.3681867 · Cited by 20
HMP-LLM: Human Mobility Prediction Based on Pre-trained Large Language Models
Zhong, X., Xiang, Y., Fang, Y. X., Li, C., & Yang, Q. (2024) · IEEE International Conference on Digital Twins and Parallel Intelligence (DTPI) · doi:10.1109/dtpi61353.2024.10778764 · Cited by 14
PedMedQA: Comparing Large Language Model Accuracy in Pediatric and Adult Medicine
Jaiswal, N., Ma, Y., Lebouché, B., Poenaru, D., & Osmanlliu, E. (2025) · Pediatrics Open Science · doi:10.1542/pedsos.2025-000485 · Cited by 4
Exploring the impact of ChatGPT on Wikipedia engagement
Reeves, N., Yin, W., & Simperl, E. P. B. (2025) · Collective Intelligence · doi:10.1177/26339137251372599 · Cited by 20
Andrew, K., & Montalbano, M. J. (2025) · Medical Science Educator · doi:10.1007/s40670-025-02332-9 · Cited by 10
Nielsen, J. P. S., Mikkelsen, A. K., Künzel, J., Sebelik, M., Madani, G., Yang, T., & Todsen, T. (2025) · Diagnostics · doi:10.3390/diagnostics15151848 · Cited by 5
Kring, T., Prasad, S., Dadi, S., Sokhn, E., & Franzmann, E. J. (2025) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2025.104710 · Cited by 6
Mitigating Bias in Large Language Models Through Culturally-Relevant LLMs
Narayan, M., Pasmore, J., Sampaio, E., Vijay, R., Maity, S., Waters, G., & Howard, A. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098204 · Cited by 9
Generative Artificial Intelligence Methodology Reporting in Otolaryngology: A Scoping Review
Alter, I. L., Chan, K., Andreadis, K., & Rameau, A. (2025) · The Laryngoscope · doi:10.1002/lary.70165 · Cited by 4
Auditing the Shadows: A Review of Methods to Detect Shared Training Data in Large Language Models
Naser, M. Z. (2025) · ACM Computing Surveys · doi:10.1145/3774897 · Cited by 9
Bin‐Nashwan, S. A., Alyafai, N., Ismaiel, A. E. A., & Ma’aji, M. M. (2026) · Journal of Information Communication and Ethics in Society · doi:10.1108/jices-09-2025-0235 · Cited by 6
Becker, M., Jong, D. D., Briker, R., Mennens, K., Heller, J., Mahr, D., & Grewal, D. (2026) · The Leadership Quarterly · doi:10.1016/j.leaqua.2026.101952 · Cited by 7
Probing with Precision: Probing Question Generation for Architectural Information Elicitation
Rejithkumar, G., Anish, P. R., Shukla, J., & Ghaisas, S. S. (2024) · doi:10.1145/3643666.3648577 · Cited by 5
Warren, B. E., Alkhalifah, F., Ahrari, A., Min, A., Fawzy, A., Annamalai, G., Jaberi, A., Beecroft, R., Kachura, J. R., & Mafeld, S. C. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371241269436 · Cited by 4
„Creating God“: Religiöse Metaphorik in KI- und Technikdiskursen
Rothenhäusler, A. (2024) · De Gruyter eBooks · doi:10.1515/9783111351490-013 · Cited by 4
Human intelligence: justifying debate in the Age of AI
Bricker, B. J., & Justice, J. W. (2024) · Argumentation and Advocacy · doi:10.1080/10511431.2024.2386801 · Cited by 8
Following the Compass: LLM-Empowered Intent Translation with Manual Guidance
Guo, L., Wang, J., Wu, J., Yan, C., Sun, H., Zhuang, Z., Qi, Q., Dong, Y., Ren, H., & Liao, J. (2024) · IEEE International Conference on Network Protocols (ICNP) · doi:10.1109/icnp61940.2024.10858507 · Cited by 7
Self supervised learning and the poverty of the stimulus
Veres, C., & Sampson, J. (2023) · Data & Knowledge Engineering · doi:10.1016/j.datak.2023.102208 · Cited by 5
Mitigating Data Sparsity in Integrated Data through Text Conceptualization
Rahman, M. A., Nadal, S., Romero, O. E., & Sacharidis, D. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00269 · Cited by 6
Lawyers Should Not Trust AI: A call for an Open-source Legal Language ModelPreprint
Dahan, S., Bhambhoria, R. V., Liang, D., & Zhu, X. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4587092 · Cited by 4
Fanfiction in the Age of AI: Community Perspectives on Creativity, Authenticity and Adoption
Alfassi, R., Cooper, A., Mitchell, Z., Calabro, M., Shaer, O., & Mokryn, O. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2531272 · Cited by 4
Nikitin, N. A. (2024) · Finance: Theory and Practice · doi:10.26794/2587-5671-2024-28-3-206-217 · Cited by 3
Prompt. Exploit. Repeat: Automating Network Security Testing with LLMs
Rigaki, M., Lukáš, O., Catania, C. A., & García, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-87330-0_2 · Cited by 4
Wu, K. Y., Qian, S. Y., & Marchand, M. (2025) · Diagnostics · doi:10.3390/diagnostics15141820 · Cited by 4
Bahmani, A. H., Cha, K., Alavi, A., Dixit, A., Ross, A., Park, R. M., Goncalves, F., Ma, S., Saxman, P., Nair, R. V., et al. (2025) · Communications Medicine · doi:10.1038/s43856-025-01034-y · Cited by 13
The application and challenges of ChatGPT in laboratory medicine
Niu, Z., Kuang, X., Chen, J., Cai, X., & Zhang, P. (2025) · Advances in Laboratory Medicine / Avances en Medicina de Laboratorio · doi:10.1515/almed-2025-0080 · Cited by 4
A brief review of some artificial intelligence methods in nephrology
Lemley, K. V. (2025) · Pediatric Nephrology · doi:10.1007/s00467-025-06995-9 · Cited by 4
Tan, C., Xu, X., & Ma, L. (2026) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2026.103924 · Cited by 7
Johnson, J. T., Duin, J. J., Kuiper, T., Drewes, Y. M., Gussekloo, J., van den Bos, F., Lefebvre, A., Spruit, M. R., van Dijk, B., & Mooijaart, S. P. (2026) · European Geriatric Medicine · doi:10.1007/s41999-026-01454-6 · Cited by 4
Huynh, L., & McNamara, D. S. (2026) · Electronics · doi:10.3390/electronics15061262 · Cited by 7
Semantic stability protocol: intercoder reliability for zero-shot classification via AI coders
Hsu, H.-Y., & Hsu, M. (2026) · Quality & Quantity · doi:10.1007/s11135-026-02832-9 · Cited by 4
Building Software by Rolling the Dice: A Qualitative Study of Vibe Coding
Chou, Y.-H., Jiang, B., Chen, Y. W., Weng, M., Jackson, V., Zimmermann, T., & Jones, J. A. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797105 · Cited by 24
Granat, M. M., Paź, A., & Mirowska-Guzel, D. M. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2954 · Cited by 4
HyLLM-IDS: A Conceptual Hybrid LLM-Assisted Intrusion Detection Framework for Cyber-Physical Systems
Muhammad, M., Shaaban, A. M., German, R., & Sardy, L. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02018-5_10 · Cited by 7
Pio‐Lopez, L., Hartl, B., & Levin, M. G. (2025) · Advanced Science · doi:10.1002/advs.202509872 · Cited by 11
Evaluating AI for Law: Bridging the Gap with Open-Source Solutions
Bhambhoria, R. V., Dahan, S., Li, J., & Zhu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-12795-2_5 · Cited by 23
Expert review of AI-generated responses to the top ten patient complaints in primary care
Gillie, M., & Kent, G. (2024) · Academia Medicine · doi:10.20935/acadmed7388 · Cited by 5
Preprinting in AI Ethics: Toward a Set of Community Guidelines
Sætra, H. S. (2025) · Communications of the ACM · doi:10.1145/3699951 · Cited by 3
Road Rage Against the Machine: Humans and LLMs Share a Blame Bias Against Driverless Cars
Chu, Y., Liu, P., Savulescu, J., & Earp, B. D. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2526593 · Cited by 3
Valz, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4468899 · Cited by 9
How Generative AI Impacts Content Engagement: Evidence from an Online Learning PlatformPreprint
Narang, U., Ahlbom, C.-P., & Banerjee, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.4558719 · Cited by 4
Evaluating Language Model Agency through Negotiations
Davidson, T. R., Veselovsky, V., Josifoski, M., Peyrard, M., Bosselut, A., Kosiński, M., & West, R. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2401.04536 · Cited by 70
Leveraging AI to improve evidence synthesis in conservationPreprint
Berger‐Tal, O., Wong, B. B. M., Adams, C. A., Blumstein, D. T., Candolin, U., Gibson, M., Greggor, A. L., Lagisz, M., Macura, B., Price, C. J., et al. (2024) · doi:10.32942/x21s64 · Cited by 3
Algorithm Aversion, Appreciation, and Investor Return BeliefsPreprint
Stradi, F., & Verdickt, G. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5207430 · Cited by 8
Management of lower extremity traumas: Comparing appropriate use criteria ChatGPT recommendations
Lu, S., Nietsch, K. S., Duey, A. H., Zaidat, B., Ndjonko, L. C. M., Shrestha, N., Kim, J., & Cho, S. K.-W. (2025) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2025.116229 · Cited by 2
Eminovic, S., Levita, B., Dell’Orco, A., Leppig, J. A., Nawabi, J., & Penzkofer, T. (2025) · Journal of Personalized Medicine · doi:10.3390/jpm15060235 · Cited by 2
Automating Plan Evaluation Using Agentic Large Language Models
Fu, X.-Y., & Li, C.-S. (2026) · Journal of Planning Education and Research · doi:10.1177/0739456x251372082 · Cited by 1
Are answers obtained from artificial intelligence models for information purposes repeatable?
Tunca, Y., Kaplan, V., & Tunca, M. (2025) · International Orthodontics · doi:10.1016/j.ortho.2025.101071 · Cited by 5
Framing the Role of AI Agents Through Metaphors
Beshlei, O. (2025) · Dialogica · doi:10.5709/dialogica-01.01.2025-02 · Cited by 2
ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts
AlOmar, E. A., Xu, L., Martinez, S., Peruma, A., Mkaouer, M. W., Newman, C. D., & Ouni, A. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00068 · Cited by 8
Akhund, S. A., Qazi, S., Mazhar, M. A., Shaikh, A. A., Atif, E., Shaikh, A., Shaibah, H., Musa, M. K., N.Qazi, S., Khan, M. A., et al. (2026) · Medical Education Online · doi:10.1080/10872981.2026.2684837 · Cited by 4
Metadata
- Authors
- Chen, L., Zaharia, M., & Zou, J.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- Harvard Data Science Review
- Publisher
- MIT Press
- Published
- 2024-03-12
- Volume
- 6
- Issue
- 2
- ISSN
- 2644-2353 (ISSN-L)
- OpenAlex
- W4392681182
- Cited by
- 507
- References
- 0
- Citations by year
- 2026: 128 · 2025: 139 · 2024: 163
- FWCI
- 63.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Harvard Data Science Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, GPT-3.5, GPT-4, model drift, model updating, chain-of-thought prompting, opinion surveys, multi-hop question answering, code generation, visual reasoning, medical licensing examinations, instruction following
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.