The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study
Open accessAbstract
The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical vignettes with common chief complaints. General internal medicine physicians created clinical cases, correct diagnoses, and five differential diagnoses for ten common chief complaints. The rate of correct diagnosis by ChatGPT-3 within the ten differential-diagnosis lists was 28/30 (93.3%). The rate of correct diagnosis by physicians was still superior to that by ChatGPT-3 within the five differential-diagnosis lists (98.3% vs. 83.3%, p = 0.03). The rate of correct diagnosis by physicians was also superior to that by ChatGPT-3 in the top diagnosis (53.3% vs. 93.3%, p < 0.001). The rate of consistent differential diagnoses among physicians within the ten differential-diagnosis lists generated by ChatGPT-3 was 62/88 (70.5%). In summary, this study demonstrates the high diagnostic accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical cases with common chief complaints. This suggests that AI chatbots such as ChatGPT-3 can generate a well-differentiated diagnosis list for common chief complaints. However, the order of these lists can be improved in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · cited 1×
“For example, gender [37,38], racial [17], and religious [17] biases have been reported.”
BackgroundAI bot ChatGPT writes smart essays — should professors worry?
Stokel-Walker, C. (2022) · Nature · doi:10.1038/d41586-022-04397-7 · Cited by 543
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · cited 2×
“Some of these data sources, including the Common Crawl [18] and English-language Wikipedia [19], contain not only general information but also science, medicine, and health information.”
BackgroundThe Future of AI in Medicine: A Perspective from a Chatbot
King, M. R. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-022-03121-w · Cited by 299
Gender and Representation Bias in GPT-3 Generated Stories
Lucy, L., & Bamman, D. (2021) · Proceedings of the Third Workshop on Narrative Understanding · doi:10.18653/v1/2021.nuse-1.5 · Cited by 287
Cited by
ChatGPT and Open-AI Models: A Preliminary Review
Roumeliotis, K. I., & Tselikas, N. D. (2023) · Future Internet · doi:10.3390/fi15060192 · Cited by 706
Haleem, A., Javaid, M., & Singh, R. P. (2022) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100089 · Cited by 694
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283
Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023) · JMIR Medical Education · doi:10.2196/48002 · Cited by 271
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192
“(Kitamura, 2023; Elali and Rachid, 2023; Liebrenz et al., 2023; Biswas, 2023a; Arif et al., 2023a; Marchandot et al., 2023; Arif et al., 2023b; Ufuk, 2023; Cascella et al., 2023b; Lubowitz, 2023) Mental (Subramani et al., 2023; Sedaghat, 2023; Hisan and Amri, 2023; Sallam et al., 2023; Hisan and Amri, 2023; Fatani, 20…”
Assessing the research landscape and clinical utility of large language models: a scoping review
Park, Y.-J., Pillai, A., Deng, J., Guo, E., Gupta, M., Paget, M., & Naugler, C. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02459-6 · Cited by 187
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203
Hirosawa, T., Kawamura, R., Harada, Y., Mizuta, K., Tokumasu, K., Kaji, Y., Suzuki, T., & Shimizu, T. (2023) · JMIR Medical Informatics · doi:10.2196/48808 · Cited by 130
“Previous studies have reported that the diagnostic accuracy of the differential diagnosis lists generated by ChatGPT for clinical vignettes falls between 64% and 83% [21,22].”
BackgroundResultSelf-citationChatGPT in radiology: A systematic review of performance, pitfalls, and future perspectives
Keshavarz, P., Bagherieh, S., Nabipoorashrafi, S. A., Chalian, H., Rahsepar, A. A., Kim, G. H. J., Hassani, C., Raman, S. S., & Bedayat, A. (2024) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2024.04.003 · Cited by 132
Wagner, M. W., & Ertl-Wagner, B. B. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231171125 · Cited by 125
Moshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124
Kernberg, A., Gold, J. A., & Mohan, V. (2024) · Journal of Medical Internet Research · doi:10.2196/54419 · Cited by 120
“Health care providers envision numerous applications for ChatGPT-4, such as answering patient questions, automated insurance prior authorizations, and creating differential diagnoses [9,10].”
Eppler, M., Ganjavi, C., Ramacciotti, L. S., Piazza, P., Rodler, S., Checcucci, E., Gomez Rivas, J., Kowalewski, K. F., Belenchón, I. R., Puliatti, S., et al. (2024) · European Urology · doi:10.1016/j.eururo.2023.10.014 · Cited by 117
Artificial Intelligence in Clinical Diagnosis: Opportunities, Challenges, and Hype
Kulkarni, P. A., & Singh, H. (2023) · JAMA · doi:10.1001/jama.2023.11440 · Cited by 112
Whiles, B. B., Bird, V. G., Canales, B. K., DiBianco, J. M., & Terry, R. S. (2023) · Urology · doi:10.1016/j.urology.2023.07.010 · Cited by 99
ChatGPT and Generating a Differential Diagnosis Early in an Emergency Department Presentation
Berg, H. T., van Bakel, B., van de Wouw, L., Jie, K. E., Schipper, A., Jansen, H., O’Connor, R. D., van Ginneken, B., & Kurstjens, S. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2023.08.003 · Cited by 97
Pardos, Z. A., & Bhandari, S. (2024) · PLOS ONE · doi:10.1371/journal.pone.0304013 · Cited by 97
“Among its many healthcare applications, ChatGPT has shown effectiveness in generating suggestions for clinical decision support and creating differential-diagnosis lists for cases involving common chief complaints [38, 39].”
BackgroundChatGPT as a Complementary Mental Health Resource: A Boon or a Bane
Farhat, F. (2023) · doi:10.20944/preprints202307.1479.v1 · Cited by 76
Shan, G., Chen, X., Wang, C., Liu, L., Gu, Y., Jiang, H., & Shi, T. (2025) · JMIR Medical Informatics · doi:10.2196/64963 · Cited by 71
Evaluation of large language models as a diagnostic aid for complex medical cases
Ríos-Hoyo, A., Shan, N. L., Li, A., Pearson, A. T., Pusztai, L., & Howard, F. M. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1380148 · Cited by 66
“Whereas high accuracy was seen when evaluating GPT-3 in diagnosing common presentations such as upper respiratory tract infections (14), we found that in approximately one third of cases the best model failed to identify the correct diagnosis in complex cases.”
BackgroundWatari, T., Takagi, S., Sakaguchi, K., Nishizaki, Y., Shimizu, T., Yamamoto, Y., & Tokuda, Y. (2023) · JMIR Medical Education · doi:10.2196/52202 · Cited by 54
Self-citationAnalyzing evaluation methods for large language models in the medical field: a scoping review
Lee, J., Park, S., Shin, J., & Cho, B. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02709-7 · Cited by 52
Sahu, P. K., Benjamin, L. A., Singh Aswal, G., & Williams-Persad, A. (2023) · Postgraduate Medical Journal · doi:10.1093/postmj/qgad090 · Cited by 52
Wang, L., Li, J., Zhuang, B., Huang, S., Fang, M., Wang, C., Li, W., Zhang, M., & Gong, S. (2025) · Journal of Medical Internet Research · doi:10.2196/64486 · Cited by 51
Shao, C.-Y., Li, H., Liu, X.-L., Li, C., Yang, L.-Q., Zhang, Y.-J., Luo, J., & Zhao, J. (2023) · Interactive Journal of Medical Research · doi:10.2196/46900 · Cited by 49
Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz
Ueda, D., Walston, S. L., Matsumoto, T., Deguchi, R., Tatekawa, H., & Miki, Y. (2023) · doi:10.1101/2023.05.04.23289493 · Cited by 45
The AI Future of Emergency Medicine
Petrella, R. J. (2024) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2024.01.031 · Cited by 43
Ji, Y., Zhan, Z., Li, T., Zou, X., & Lyu, S. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3554584 · Cited by 42
A bibliometric analysis of the advance of artificial intelligence in medicine
Lin, M., Lin, L., Lin, L., Lin, Z., & Yan, X. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1504428 · Cited by 36
ChatGPT-Based Learning: Generative Artificial Intelligence in Medical Education
Stretton, B., Kovoor, J., Arnold, M., & Bacchi, S. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01934-5 · Cited by 34
Galido, P. V., Butala, S., Chakerian, M., & Agustines, D. (2023) · Cureus · doi:10.7759/cureus.38166 · Cited by 29
Yazaki, M., Maki, S., Furuya, T., Inoue, K., Nagai, K., Nagashima, Y., Maruyama, J., Toki, Y., Kitagawa, K., Iwata, S., et al. (2025) · Prehospital Emergency Care · doi:10.1080/10903127.2024.2374400 · Cited by 29
Wada, A., Akashi, T., Shih, G., Hagiwara, A., Nishizawa, M., Hayakawa, Y., Kikuta, J., Shimoji, K., Sano, K., Kamagata, K., et al. (2024) · Diagnostics · doi:10.3390/diagnostics14141541 · Cited by 25
Exploring the role of ChatGPT in oncology: providing information and support for cancer patients
Cè, M., Chiarpenello, V., Bubba, A., Felisaz, P. F., Oliva, G., Irmici, G., & Cellina, M. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4020049 · Cited by 24
Mixed methods assessment of the influence of demographics on medical advice of ChatGPT
Andreadis, K., Newman, D. R., Twan, C., Shunk, A., Mann, D. M., & Stevens, E. R. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae086 · Cited by 20
Can We Trust AI Chatbots’ Answers about Disease Diagnosis and Patient Care?
Huh, S. (2023) · Journal of the Korean Medical Association · doi:10.5124/jkma.2023.66.4.218 · Cited by 10
de Arriba-Pérez, F., & García-Méndez, S. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-024-09508-2 · Cited by 8
Metadata
- Authors
- Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T.
- Venue
- International Journal of Environmental Research and Public Health
- Publisher
- MDPI AG
- Published
- 2023-02-15
- Volume
- 20
- Issue
- 4
- Pages
- 3378
- ISSN
- 1660-4601 (ISSN-L)
- OpenAlex
- W4320920036
- Cited by
- 343
- References
- 35
- FWCI
- 11.80
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.