How Is ChatGPT’s Behavior Changing Over Time?
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4’s amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5’s performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4’s ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the “same” LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Our finding that most LLMs consistently produced algorithmic overgeneralizations across multiple occasions adds nuance to prior research that found that, in some domains, LLMs produced unstable or inconsistent responses over time [48–50].”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our findings thus offer novel contributions to research on overgeneralization in science communication [17,18,54] and the shortcomings of LLMs in text summarization [48,52,55], revealing a subtle form of ‘hallucination’ that has not yet been documented in the literature.”
From Generalization bias in large language model summarization of scientific research (2025) · cites this paper as [48]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Part of this paper’s reference list could not be split into single entries.
Cited by
Items 401–500 of 517
Interval evaluation of temporal (in)stability for neural machine translation
Egorova, A., Кружков, М. Г., Nuriev, V. A., & Zatsman, I. M. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00222-y
Advancing sentiment understanding and safety in NLP systems
Deng, Y. (2025) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/183610 · Cited by 1
Thinking with Machines: How AI Solves Math Puzzles and What Educators Can Learn
Applebaum, M. (2025) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-26055 · Cited by 1
How Risky Are Open Frontier Models?
Krier, S., Law, H., & Klyman, K. (2025) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780198940272.013.0023
A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows
Alismail, A., & Lanquillon, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93418-6_9 · Cited by 13
Rattananan, W. (2025) · medRxiv · doi:10.1101/2025.06.12.25329503
Navigating Large Language Models: A Comprehensive Overview
Shukla, S., & Tiwari, K. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-4883-2_14
The impact of task similarity on performance drift of LLMs: a theoretical and empirical analysis
Ni, X., Zhao, Q., Huang, S., & Yu, L. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4442-8
Zhijie, Z., Deng, Y., Qi, H., & Wang, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351223
Ruskov, M., Ognibene, D., Hernández‐Leo, D., Taibi, D., Dimitriadis, Y. A., Fulantelli, G., Handmann, U., Zarifis, G. K., Wu, Y., & Ferri, P. M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99267-4_28
Towards the Versioning of LLM-Agent-Based Software
Liu, C., Zhang, L., Xu, X., Guo, W., & Liu, Y. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728714 · Cited by 4
The Hitchhikers Guide to Production-ready Trustworthy Foundation Model Powered Software (FMware)
Vasilevski, K., Rajbahadur, G. K., Oliva, G. A., Rombaut, B., Gallaba, K., Cogo, F. R., Lin, J., Lin, D., Zhang, H., Chen, B., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736572 · Cited by 3
Dejean-Bouyer, E., Kanlagna, A., Thuau, F., Perrot, P., & Lancien, U. (2025) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2025.22.27
Rakityanskaya, A. (2025) · Open Library of Humanities · doi:10.16995/olh.24212
LLM Integration in Game Writing: An Investigation of Visions and Tools
Sezen, D., & Sezen, T. İ. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-06982-5_5
Ortner, V. K., Lent, H., Karmisholt, K. E., Lei, U., Gyldenløve, M., Omland, S. H., Kaur‐Knudsen, D., Kainu, K., Togsverd‐Bo, K., Wiegell, S. R., et al. (2025) · Experimental Dermatology · doi:10.1111/exd.70175
Punnen, T. G., Shan, K., Patel, M. A., McCreary, M., Tran, D. H., Santoyo, J. R., Burgess, K. W., Moog, T. M., Smith, A. D., & Okuda, D. T. (2025) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2025.100314
DT4LM: Differential Testing for Reliable Language Model Updates in Classification Tasks
Zuo, X., Xiao, Y., Cao, X., Wang, W., & Dong, J. S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3622251 · Cited by 1
Aplicaciones y retos de ChatGPT en la medicina de laboratorio
Niu, Z., Kuang, X., Chen, J., Cai, X., & Zhang, P. (2025) · Advances in Laboratory Medicine / Avances en Medicina de Laboratorio · doi:10.1515/almed-2025-0158
Social Impact of Generative and Multimodal AI on Human Social Action
Sudoh, O. (2025) · Springerbriefs for data scientists and innovators · doi:10.1007/978-981-95-2613-0_3
Machine Learning in Materials Science: Current Challenges and Future Outlook
Bhaskar, A., Yadav, P., Singh, B., Singh, S. K., & Rana, S. S. (2026) · Springer eBooks · doi:10.1007/978-3-032-09540-4_11
Exploring the Impact of Large Language Models on Opinion Analysis in Arabic Social Media
Alzyadneh, M., Fraj, F. B., & Zribi, C. B. O. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315188
Minding the gap between artificial and biological computing paradigms for biologically loyal AI
Kirkpatrick, K. L. (2026) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2025.1695493
Shakespeare did not know our vocabulary: Measuring the Historical Adequacy of LLMs
Cuscito, M., Ferrara, A., & Ruskov, M. (2025) · IEEE International Conference on Cyber Humanities (IEEE-CH) · doi:10.1109/ieee-ch65308.2025.11279542 · Cited by 2
How Marketers Can Utilize Generative AI in Qualitative Research
Fares, O. H. (2026) · Springer eBooks · doi:10.1007/978-3-031-75316-9_55-1
A framework for verb annotation of Spanish L2 written corpora using local LLMs
Zimotti, G. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100308
Furfaro, E. (2026) · Journal of Data Science · doi:10.6339/25-jds1208b
Stelling, H., Kraus, A., Grieb, G., & Güler, I. (2026) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe16020023
Developing Problem-Solving Competency in Data Science: Exploring A Case-Based Approach
Chen, L., Alomair, M., Yousuf, M. A., & Pan, S. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772656
Self-citationA Survey on Federated Parameter-Efficient Fine-Tuning for Large Language Models
Wen, Q., Zhang, X., Xiang, N., Chen, J., Wang, X., & Zhang, J. (2025) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia68682.2025.11383022
FLAME: Enhancing Functional Coverage in Processor Verification via Large Language Models
Li, X., Chen, J., Yan, M. X., Wang, D., Fan, X., Tang, Z., Kai, S., Hao, J., Yuan, M., & Wang, Z. (2026) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2026.3666802
Madzhumder, M. S., & Begunova, D. D. (2025) · Scientific and Technical Information Processing · doi:10.3103/s0147688225700479 · Cited by 1
Dixon, J. C., & Buchmann, H. (2026) · Teaching Sociology · doi:10.1177/0092055x261422474
Uma, S., Ghosh, S., & Mustaqim, D. D. A. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789130
Ong, A. Y., Nguyen, Q., Barai, I., Engelmann, J., Antaki, F., Sevgi, M., Merle, D. A., Ju, L., Dow, E. R., Zhou, Y., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-03136-w · Cited by 1
Abedi, M., Sanusi, I. T., & Tukiainen, M. (2026) · AI Brain and Child · doi:10.1007/s44436-026-00029-3
Interpretable Music Recommender Systems
Afchar, D. (2023) · theses.fr (ABES) · doi:10.70675/acfb6676z8613z4e08zad14z98d4a9c2ee74 · Cited by 3
ChatGPT as a Semiotic Technology: Evolving Gains and Losses for English Language LearningPreprint
Sindoni, M. G. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6592482
Artificial Intelligence in Surgical Education: A Pilot Study Using ASCRS Guideline-Derived Questions
Pandya, S., Wilson, T. W., Meyer, R. J., Bresler, T. E., & Fujita, M. (2026) · The American Surgeon · doi:10.1177/00031348261443342
Model Equality Testing of Black-Box LLM APIs Via Prefix Tree Statistics
Shi, K.-Y., Li, F.-Q., Zhang, Z.-M., & Wang, S.-L. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461352
Liu, F., Hu, R., Kim, D., Lo, H., Kharonov, Y., Johri, A., Gerber, B. S., Rundensteiner, E., Westafer, L. M., Allison, J., et al. (2026) · JMIR Medical Informatics · doi:10.2196/79972
Xiang, Y., & Zhang, C. (2026) · Scientometrics · doi:10.1007/s11192-026-05646-6
Large Language Models and Gender Bias: An Empirical AnalysisPreprint
ÇAKIR, P., & TEMİR, E. (2026) · Research Square · doi:10.21203/rs.3.rs-9577604/v1
Chen, T., Su, L. Y.-F., Ng, Y. M. M., Stasiewicz, M. J., & Wang, Y.-C. (2026) · Food Quality and Preference · doi:10.1016/j.foodqual.2026.105968 · Cited by 1
The Human Factor and Artificial Intelligence: A Balance of Collaboration
Баторшина, Г., & Никитина, Н. (2026) · Media Theory and Practice · doi:10.65324/mtp030
Kang, W. J., Sim, J., Loh, E. E.-M., Lim, A. C. Y., & Foong, K. W. C. (2026) · medRxiv · doi:10.64898/2026.05.17.26353409
Yağcı, B., Palaz, S., & Senirli, R. T. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02008-9 · Cited by 2
Cross-sectional analysis of large language models in current natural language generation challenges
Maestre, M. M., Martínez-Murillo, I., Martínez, A. M., & Lloret, E. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104922 · Cited by 1
Behavioral Fingerprints for LLM Endpoint Stability and Identity
Leshin, J., Shah, M., Timmis, I., & Kang, D. D. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813194 · Cited by 6
Villarrubia, C., Villarrubia-Martin, E. A., Vara, J. M., & Granada, D. (2026) · Requirements Engineering · doi:10.1007/s00766-026-00463-y
A Multi-Criteria Decision Framework for Enterprise LLM Routing
Nowak, M. (2026) · Information · doi:10.3390/info17060539 · Cited by 1
AI-Enabled Security Operations Centers (SOCs) Accelerating Incident Response
Srinivasan, A., Dineshnath, G., & Iniyan, S. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0990-3.ch008
Enhancing academic assessment: Leveraging AI for grading scholarly student papers
Keith, A. C., Ruegg, T. A., Roy, P., & Greene, C. (2026) · Teaching and learning in nursing · doi:10.1016/j.teln.2026.05.019
Tracing moral value drift across large language model generations and their societal implications
Naser, M. (2026) · Technology and Society · doi:10.1016/j.techsoc.2026.103431 · Cited by 1
Agentic AI in Enterprise Middleware: Toward Autonomous Integration Orchestration
Jaidi, S. R. (2026) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.5339
Toward Reliable LLM-Integrated Web Architectures for Teacher-Aligned Automatic Student Grading
Gwozdz, J., & Both, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29372-5_8 · Cited by 1
Yost, C. D., Callas, B., Halligan, P., Palumbo, P., Rawal, S. C., Leyfman, Y., & Nguyen, R. H.-T. (2026) · Cancers · doi:10.3390/cancers18121981 · Cited by 2
Kim, R. S. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01573-w
Evaluating LLMs Reliability for Coding Adversarial Machine Learning Research Papers
Khanal, M., Jasser, J., & Basirat, M. (2026) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw72489.2026.00035
Cox, M., Lin, E., Oleck, N. C., Jones, C., Li, N. Y., Mithani, S. K., & Allori, A. C. (2026) · JMIR Formative Research · doi:10.2196/87276
Nazarovets, S., & da Silva, J. A. T. (2026) · Journal of Ethics in Entrepreneurship and Technology · doi:10.1108/jeet-09-2025-0059 · Cited by 2
Correia, L., & Feng, Y. (2026) · Journal of Food Protection · doi:10.1016/j.jfp.2026.100852
Karakoyunlu, E., Mıdoglu, G., Gürler, K., Öztürk, Y. E. G., & Yılmaz, K. (2026) · BMC Oral Health · doi:10.1186/s12903-026-09298-z
LLM-RAG Intent Translation Meets Dual Time Scale Control for QoS-Guaranteed Video Services
Zhang, M., Guo, L., Chen, D., Yan, Y., Zhuang, Z., Qi, Q., Sun, H., Li, M., Zhou, C., Lu, L., et al. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3557-5_16
Choi, I., Kim, J., Kim, J.-H., Lee, H.-Y., Park, W., Kim, C.-E., & Lim, D. (2026) · Applied Sciences · doi:10.3390/app16147223
Abdurrazaq, A. O., Okebukola, P. A., Enikuomehin, O. A., Agbanimu, D. O., Ugwuoke, U., Abokunwa, V., & Ayinde, B. O. (2026) · Discover Education · doi:10.1007/s44217-026-01846-7
Qian, C. (2026) · IEEE International Conference on Electronic Technology, Communication and Information (ICETCI) · doi:10.1109/icetci68937.2026.11610760
Citation Intent Classification via Exponential Borda Fusion and SciBERT
Barchane, M., Belefqih, S., Benlahmar, E. H., Zahour, O., & Zellou, A. (2026) · Algorithms · doi:10.3390/a19080612
Performance of GPT-4o and Claude in Medical Ethics Scenarios: Comparative Study
Desai, K., Gorsky, A. L., & Zelenski, N. A. (2026) · JMIR Medical Education · doi:10.2196/70199 · Cited by 1
Randomness In Large Language Models: What Researchers Need to Know (And Report)Preprint
Coqueret, G., Llull, J., Oswald, F., Pérignon, C., Scheuch, C., & Vilhuber, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7191580 · Cited by 2
Beyond the Conventional Pipeline: Large Language Models for Social Science Measurement
Bảo, L. T. (2026) · Chinese Political Science Review · doi:10.1007/s41111-026-00351-4
Redefining instructor roles and responsibilities in the era of generative AI
Vaghjee, H., & Vaghjee, G. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-36775-5.00006-0
Optimizing sample selection for large language model-based entity matching using AssistEM
Mugeni, J. B., Lynden, S. J., AMAGASA, T., & Matono, A. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01223-5 · Cited by 2
Tieszen, J. (2026) · Journal of Map & Geography Libraries · doi:10.1080/15420353.2026.2707079
Recipes for Agents: Understanding Skills and Their Open Questions
Xing, H., Zhuang, H., Zhao, X., Huang, Y., Tang, Z., & Zhang, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818652 · Cited by 2
Lu, C., Zhang, R., Jiang, H., Yu, Y., Zhang, S., Lin, Q., Wu, P., & Li, H. (2026) · Journal of Medical Internet Research · doi:10.2196/100442 · Cited by 1
Leveraging Artificial Intelligence to Reduce Opioid Overprescription
Borbiev, S. T., Morgan, C. D., Watt, M. J., Burris, B. J., McCain, J. P., Peacock, Z. S., & Wang, T. T. (2026) · Current Surgery Reports · doi:10.1007/s40137-026-00528-5
ZHENG, X., Chen, B., Yuqing, C. H. E. N., & Yang, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-10367743/v1
Bhatt, S., & Das, D. (2026) · Management Review Quarterly · doi:10.1007/s11301-026-00624-x
Global Trends in Research Ethics in the Context of Artificial Intelligence
Resulbegoviq, H. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1232-1.ch012
Proof Schemes Used and Owned by Artificial Intelligence Chatbots
Cihan, F. (2026) · Disiplinlerarası Eğitim Araştırmaları Dergisi · doi:10.57135/jier.1894348
Lin, W., Li, H., Zhuang, J., Li, Y., Yao, X., Chen, M., & Yang, Y. (2026) · International Conference on Innovation Management and Information Systems (ICIMIS) · doi:10.1145/3821966.3821996
Güler, I., Kraus, A., Grieb, G., & Stelling, H. (2026) · Biomedicines · doi:10.3390/biomedicines14091996
Declaring the machine: A disclosure policy on artificial intelligence for authors and reviewers
Iliescu, D., & Greiff, S. (2026) · Intelligence · doi:10.1016/j.intell.2026.102060
From Regulation to Requirements: An Automated Requirement Derivation and Explanation Pipeline
Nair, P. P., & Anish, P. R. (2026) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re68928.2026.00038
What Do We Architect When We Architect LLM Agent Systems?
Li, K., Xie, M., Ge, J., & Liu, Y. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3795870
A Framework for Assessing LLM Consistency in Knowledge Engineering
Saeedizade, M. J., Alharbi, R., Giglou, H. B., Lippolis, A. S., Blomqvist, E., Tamma, V., Grasso, F., Payne, T. R., D'Souza, J., Auer, S., et al. (2026) · Semantic Web: – Interoperability, Usability, Applicability · doi:10.1177/22104968261479512 · Cited by 1
Do Students Learn from Writing Feedback from an AI Teaching Assistant?
Liu, X., Wei, Z., Baker, R. S., Pankiewicz, M., Dai, Y., & Vanacore, K. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_20
Alkhanishvili, Z., Ketevan, G., Gvasalia, N., Samkharadze, S., Lursmanashvili, L., NACHKEBIA, L., Chkhikvishvili, A., Tahir, A. Q., & Tsitskishvili, N. (2026) · Cureus · doi:10.7759/cureus.117183
Human–AI Collaboration is an Underspecified Intervention: A Reporting Guide for Cumulative Research
Rafner, J. F. (2026) · Creativity Research Journal · doi:10.1080/10400419.2026.2726243 · Cited by 1
Aligning Editorial Review With the Pace of Language Models: Five Proposals for Oncology Publishing
Cantón, E. D. (2026) · JCO Clinical Cancer Informatics · doi:10.1200/cci-26-00245
New Asymmetries: How to Cheat on a Humanities Paper with AI and Still Learn Something
Lupowitz, J. L. (2026) · Tiempo devorado · doi:10.5565/rev/tdevorado.279
Mihăluţe, E., & Mititelu, N.-R. (2026) · Buletinul Institutului Politehnic din Iaşi. Secţia Construcţii de maşini/Buletinul Institutului Politehnic din Iaşi, Secţia Construcţii de maşini · doi:10.2478/bipcm-2026-0028
Duojie, J., Devdas, S., Stieger, M., de Spindler, A., & Lu, G. (2026) · Electronics · doi:10.3390/electronics15184241
Coverage-Guided GenAI Test Generation: A Comparative Study of One-Shot vs. Iterative Methods
Maood, A. S., Ali, M. H., Amair, M. W., Mohammed, I. K., Al-Sharif, Z. A., & Omari, S. (2026) · Advances in intelligent systems and computing · doi:10.1007/978-3-032-27214-0_37
Yang, M., Park, C., Won, H., Kim, T., Song, H., & Han, J. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816903
Chai, B. S., Yim, M. M., Ranjbar, D., & Tang, D. T. (2026) · Global Surgical Education - Journal of the Association for Surgical Education · doi:10.21203/rs.3.rs-10198678/v1
Cipriano, B. P., & Ferreira, L. S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-39106-3_17
When the Machine Chooses: Brand Management Under Selection Scarcity
Rach, M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9589-0.ch002
Metadata
- Authors
- Chen, L., Zaharia, M., & Zou, J.
- Affiliations
- Stanford University (US)
- University of California, Berkeley (US)
- Venue
- Harvard Data Science Review
- Publisher
- MIT Press
- Published
- 2024-03-12
- Volume
- 6
- Issue
- 2
- ISSN
- 2644-2353 (ISSN-L)
- OpenAlex
- W4392681182
- Cited by
- 507
- References
- 0
- Citations by year
- 2026: 128 · 2025: 139 · 2024: 163
- FWCI
- 63.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Harvard Data Science Review (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, GPT-3.5, GPT-4, model drift, model updating, chain-of-thought prompting, opinion surveys, multi-hop question answering, code generation, visual reasoning, medical licensing examinations, instruction following
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.