Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions
Open accessAbstract
GPT-4 demonstrates high accuracy in medical QA tasks, leading with an accuracy of 86.70%, followed by Med-PaLM 2 at 86.50%. However, around 14% of errors remain. Additionally, current works use GPT-4 to only predict the correct option without providing any explanation and thus do not provide any insight into the thinking process and reasoning used by GPT-4 or other LLMs. Therefore, we introduce a new domain-specific error taxonomy derived from collaboration with medical students. Our GPT-4 USMLE Error (G4UE) dataset comprises 4153 GPT-4 correct responses and 919 incorrect responses to the United States Medical Licensing Examination (USMLE) respectively. These responses are quite long (258 words on average), containing detailed explanations from GPT-4 justifying the selected option. We then launch a large-scale annotation study using the Potato annotation platform and recruit 44 medical experts through Prolific, a well-known crowdsourcing platform. We annotated 300 out of these 919 incorrect data points at a granular level for different classes and created a multi-label span to identify the reasons behind the error. In our annotated dataset, a substantial portion of GPT-4's incorrect responses is categorized as a "Reasonable response by GPT-4," by annotators. This sheds light on the challenge of discerning explanations that may lead to incorrect options, even among trained medical professionals. We also provide medical concepts and medical semantic predications extracted using the SemRep tool for every data point. We believe that it will aid in evaluating the ability of LLMs to answer complex medical questions. We make the resources available at https://github.com/roysoumya/usmle-gpt4-error-taxonomy.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
An analysis of GPT-4's errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions.
“Another analysis of GPT-4’s errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions40.”
From Large Language Models lack essential metacognition for reliable medical reasoning (2025) · cites this paper as [40]
The Introduction (p. 2) identifies “sticking with the wrong diagnosis” and “incorrect or vague conclusion” as the most frequent reasoning-error subclasses; the former resembles anchoring. But §5.2 (p. 8) reports them at 26.9% and 22.21%, respectively, and does not establish that most errors fall into either category. The paper also describes the pattern as observed in its annotated subset, not all GPT-4 USMLE errors. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605 · see citing works
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556 · see citing works
Rindflesch, T. C., & Fiszman, M. (2003) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2003.11.003 · Cited by 519 · see citing works
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509 · see citing works
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · see citing works
Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge
Kanjee, Z., Crowe, B., & Rodman, A. (2023) · JAMA · doi:10.1001/jama.2023.8288 · Cited by 449 · see citing works
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408 · see citing works
Fact or Fiction: Verifying Scientific Claims
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., & Hajishirzi, H. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.609 · Cited by 382 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 7× · see citing works
“Liévin et al. [21] investigated various chain-of-thought prompting techniques on GPT-3.”
BackgroundResultInfluentialTowards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Towards Accurate Differential Diagnosis with Large Language Models
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00164 · Cited by 311 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160 · see citing works
Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
Jin, Q., Chen, F., Zhou, Y., Xu, Z., Cheung, J. M., Chen, R., Summers, R. M., Rousseau, J. F., Ni, P., Landsman, M. J., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01185-7 · Cited by 146 · see citing works
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training DataPreprint
Han, T., Adams, L. C., Papaioannou, J.-M., Grundmann, P., Oberhauser, T., Figueroa, A., Löser, A., Truhn, D., & Bressem, K. K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08247 · Cited by 117 · see citing works
Broad-coverage biomedical relation extraction with SemRep
Halil Kilicoglu, B., Rosemblat, G., Marcelo Fiszman, B., & Shin, D. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-3517-7 · Cited by 106 · see citing works
Pre-trained language models with domain knowledge for biomedical extractive summarization
Xie, Q., Bishop, J. A., Tiwari, P., & Ananiadou, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109460 · Cited by 92 · see citing works
DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task
Xiong, H., Wang, S., Zhu, Y., Zhao, Z., Liu, Y., Huang, L., Wang, Q., & Shen, D. (2023) · arXiv · doi:10.48550/arxiv.2304.01097 · Cited by 81 · see citing works
Question Answering with Long Multiple-Span Answers
Zhu, M., Ahuja, A., Juan, D.-C., Wei, W., & Reddy, C. K. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.342 · Cited by 62 · see citing works
PMC-LLaMA: Towards Building Open-source Language Models for MedicinePreprint
Wu, C., Lin, W., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14454 · Cited by 48 · see citing works
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Moramarco, F., Papadopoulos Korfiatis, A., Perera, M., Juric, D., Flann, J., Reiter, E., Belz, A., & Savkov, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.394 · Cited by 40 · see citing works
Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliveryPreprint
Dash, D., Thapa, R., Banda, J. M., Swaminathan, A., Cheatham, M., Kashyap, M., Kotecha, N., Chen, J. H., Gombar, S., Downing, N. L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.13714 · Cited by 33 · see citing works
Chern, I.-C., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2307.13528 · Cited by 29 · see citing works
POTATO: The Portable Text Annotation Tool
Pei, J., Ananthasubramaniam, A., Wang, X., Zhou, N., Dedeloudis, A., Sargent, J., & Jurgens, D. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.33 · Cited by 27 · see citing works
Characterization of Stigmatizing Language in Medical Records
Harrigian, K., Zirikly, A., Chee, B. W., Ahmad, A., Links, A. R., Saha, S., Beach, M. C., & Dredze, M. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.28 · Cited by 25 · see citing works
Benchmarking the symptom-checking capabilities of ChatGPT for a broad range of diseases
Chen, A., Chen, D. O., & Tian, L. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad245 · Cited by 25 · see citing works
Teaching Algorithmic Reasoning via In-context LearningPreprint
Zhou, H., Nova, A., Larochelle, H., Courville, A. C., Neyshabur, B., & Sedghi, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09066 · Cited by 24 · see citing works
Everyone’s Voice Matters: Quantifying Annotation Disagreement Using Demographic Information
Wan, R., Kim, J., & Kang, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26698 · Cited by 23 · see citing works
Discovering novel drug-supplement interactions using SuppKG generated from the biomedical literature
Schutte, D., Vasilakes, J., Bompelli, A., Zhou, Y., Marcelo Fiszman, B., Xu, H., Halil Kilicoglu, B., Bishop, J. R., Adam, T. J., & Zhang, R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104120 · Cited by 21 · see citing works
Poddar, S., Samad, A. M., Mukherjee, R., Ganguly, N., & Ghosh, S. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531745 · Cited by 21 · see citing works
Self-citationWhen the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks
Fleisig, E., Abebe, R., & Klein, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.415 · Cited by 20 · see citing works
A Survey for Biomedical Text Summarization: From Pre-trained to Large Language ModelsPreprint
Xie, Q., Luo, Z., Wang, B., & Ananiadou, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08763 · Cited by 10 · see citing works
Human Evaluation and Correlation with Automatic Metrics in Consultation Note GenerationPreprint
Moramarco, F., Korfiatis, A. P., Perera, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.00447 · Cited by 3 · see citing works
Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicinePreprint
Jin, Q., Chen, F., Zhou, Y., et al. (2024) · PubMed · doi:10.48550/arxiv.2401.08396 · Cited by 2 · see citing works
Pei, J., & Jurgens, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.06826 · Cited by 1 · see citing works
Schutte, D., Vasilakes, J., Bompelli, A., Zhou, Y., Marcelo Fiszman, B., Xu, H., Halil Kilicoglu, B., Bishop, J. R., Adam, T. J., & Zhang, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12741
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2023) · arXiv · doi:10.48550/arxiv.2308.02312
Flagged first, then by citations
Research works printed but not linked: 5
- [1][1] Griffin Adams, Jason Zuckerg, and Noémie Elhadad. 2023. A meta-evaluation of faithfulness metrics for long-form hospital-course summarization. In Machine Learning for Healthcare Conference . PMLR, 2-30. -
- [11][11] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations . https://openreview.net/forum?id=d7KBjmI3GmQ -
- [26][26] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In Proceedings of the Conference on Health, Inference, and Learning (Proceedings of Machine Learning Research, Vol. 174) . 248-260. https://proceedings.mlr.press/v174/pal22a.html -
- [43][43] Caroline Wellbery. 2011. Flaws in clinical reasoning: a common cause of diagnostic error. American family physician 84, 9 (2011), 1042-1048. -
- [47][47] Qianqian Xie, Edward J Schenck, He S Yang, et al. 2023. Faithful ai in medicine: A systematic review with large language models and beyond. medRxiv (2023). -
Cited by
Sorted by citations
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
“Another analysis of GPT-4 ’ s errors on the USMLE demonstrated that most errors are either caused by an anchoring bias or incorrect conclusions 40 .”
Maximising Large Language Model Utility in Cardiovascular Care: A Practical Guide
Nolin-Lapalme, A., Thériault-Lauzier, P., Corbin, D., Tastet, O., Sharma, A., Hussin, J., Kadoury, S., Jiang, R., Krahn, A. D., Gallo, R., et al. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.05.024 · Cited by 28 · see citing works
Strasser, L. M., Anschuetz, W., Dennstädt, F., & Hastings, J. (2026) · JMIR Medical Education · doi:10.2196/81399 · Cited by 20 · see citing works
Teng, D., Tan, L., Cao, Q., Xia, Y., Zhang, N., Li, J., & Zhao, D. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02547-z · Cited by 18 · see citing works
Reis, Z. S. N., Pagano, A. S., de Oliveira, I. J. R., Dias, C. D. S., Lage, E. M., Mineiro, É. F., Pereira, G. M. V., Gomes, I. D. C., Basilio, V. A., Cruz-Correia, R. J., et al. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.09.006 · Cited by 11 · see citing works
Multi-Agentic LLMs for Personalizing STEM Texts
Vaccaro, M., Friday, M., & Zaghi, A. E. (2025) · Applied Sciences · doi:10.3390/app15137579 · Cited by 12 · see citing works
Altermatt, F. R., Neyem, A., Sumonte, N. I., Villagrán, I., Mendoza, M., Lacassie, H. J., & Delfino, A. E. (2025) · BMC Medical Education · doi:10.1186/s12909-025-08084-9 · Cited by 10 · see citing works
Performance evaluation of large language models for the national nursing examination in Japan
Kuribara, T., Hirayama, K., & Hirata, K. (2025) · Digital Health · doi:10.1177/20552076251346571 · Cited by 5 · see citing works
Li, T. W., Song, Y., Sundaram, H., & Karahalios, K. G. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729550 · Cited by 5 · see citing works
Ren, B., Cheon, E., & Li, J. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757641 · Cited by 6 · see citing works
Altermatt, F. R., Neyem, A., Sumonte, N. I., Villagrán, I., Mendoza, M., & Lacassie, H. J. (2025) · Applied Sciences · doi:10.3390/app15116245 · Cited by 3 · see citing works
Instruções de Uso de Medicamentos Suportadas por RAG em Grandes Modelos de Linguagem
dos Reis, D. N., Reis, Z. S. N., & da Rocha, L. C. D. (2025) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2025.6039 · Cited by 1 · see citing works
Reis, Z. S. N., de Albergaria, E. T., Pagano, A. S., Lage, E. M., de Oliveira, F. R., Dias, C. D. S., Oliveira, J. A., Pereira, G. M. V., de Oliveira, I. J. R., Mineiro, É. F., et al. (2025) · doi:10.12688/verixiv.1359.1 · Cited by 1 · see citing works
Building Trustworthy AI Models for Medicine: From Theory to Applications
Roy, S., Sundaram, S. S., Wolff, D., & Ganguly, N. (2025) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3701551.3703477
Self-citationVaxEval: A Benchmark Dataset for Evaluating Large Language Models on Vaccine Knowledge (Preprint)
Chen, S., Wu, Z., Wass, L. M., Garay, L. G., Vizoso, J., Leung, K. S. M., & Lin, L. K. (2025) · JMIR AI · doi:10.2196/71216
V-RAG: Competitive Tree Reranking and Static Distillation for Answer-Source Alignment
Zhao, H., Lin, W., Cai, S., Duo, F., & Yang, J. (2025) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/ic-nidc67200.2025.11390266
Stalp, J. L., Schneider, J. A., STEINKASSERER, L., Hachenberg, J., Jentschke, M., Hillemanns, P., Wolff, D., & Denecke, A. (2026) · Archives of Gynecology and Obstetrics · doi:10.1007/s00404-026-08498-w
- Authors
- Roy, S., Khatua, A., Ghoochani, F., Hadler, U., Nejdl, W., & Ganguly, N.
- Affiliations
- Indian Institute of Technology Kharagpur (IN)
- University of Michigan (US)
- L3S Research Center (DE)
- Venue
- Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2024-07-10
- Pages
- 1073-1082
- OpenAlex
- W4395064833
- Cited by
- 17
- References
- 25
- Citations by year
- 2026: 3 · 2025: 12 · 2024: 2
- FWCI
- 4.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Indian Institute of Technology Kharagpur
- Federal Ministry of Education and Research (BMBF), Germany (01DD20003)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, USMLE, medical licensing examinations, error taxonomy, large language models, medical question answering, incorrect responses, explanation quality, annotation, SemRep, medical students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
