The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study
Open accessAbstract
The diagnostic accuracy of differential diagnoses generated by artificial intelligence (AI) chatbots, including the generative pretrained transformer 3 (GPT-3) chatbot (ChatGPT-3) is unknown. This study evaluated the accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical vignettes with common chief complaints. General internal medicine physicians created clinical cases, correct diagnoses, and five differential diagnoses for ten common chief complaints. The rate of correct diagnosis by ChatGPT-3 within the ten differential-diagnosis lists was 28/30 (93.3%). The rate of correct diagnosis by physicians was still superior to that by ChatGPT-3 within the five differential-diagnosis lists (98.3% vs. 83.3%, p = 0.03). The rate of correct diagnosis by physicians was also superior to that by ChatGPT-3 in the top diagnosis (53.3% vs. 93.3%, p < 0.001). The rate of consistent differential diagnoses among physicians within the ten differential-diagnosis lists generated by ChatGPT-3 was 62/88 (70.5%). In summary, this study demonstrates the high diagnostic accuracy of differential-diagnosis lists generated by ChatGPT-3 for clinical cases with common chief complaints. This suggests that AI chatbots such as ChatGPT-3 can generate a well-differentiated diagnosis list for common chief complaints. However, the order of these lists can be improved in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553 · cited 1×
“For example, gender [37,38], racial [17], and religious [17] biases have been reported.”
BackgroundAI bot ChatGPT writes smart essays — should professors worry?
Stokel-Walker, C. (2022) · Nature · doi:10.1038/d41586-022-04397-7 · Cited by 543
Evaluation of symptom checkers for self diagnosis and triage: audit study
Semigran, H. L., Linder, J. A., Gidengil, C., & Mehrotra, A. (2015) · BMJ · doi:10.1136/bmj.h3480 · Cited by 489
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · cited 2×
“Some of these data sources, including the Common Crawl [18] and English-language Wikipedia [19], contain not only general information but also science, medicine, and health information.”
BackgroundThe Future of AI in Medicine: A Perspective from a Chatbot
King, M. R. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-022-03121-w · Cited by 299
Gender and Representation Bias in GPT-3 Generated Stories
Lucy, L., & Bamman, D. (2021) · Proceedings of the Third Workshop on Narrative Understanding · doi:10.18653/v1/2021.nuse-1.5 · Cited by 287
Korngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259
Natural Language Processing for Smart Healthcare
Zhou, B., Yang, G., Shi, Z., & Ma, S. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3210270 · Cited by 230
Jackson, R. G., Patel, R., Jayatilleke, N., Kolliakou, A., Ball, M., Gorrell, G., Roberts, A., Dobson, R. J., & Stewart, R. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012012 · Cited by 221
Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization
Chintagunta, B., Katariya, N., Amatriain, X., & Kannan, A. (2021) · doi:10.18653/v1/2021.nlpmc-1.9 · Cited by 200
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00097-9 · Cited by 199
Barnett, M. L., Boddupalli, D., Nundy, S., & Bates, D. W. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.0096 · Cited by 154 · cited 1×
“Additionally, consistent differential diagnoses made by two independent physicians were suggested to be important differential diagnoses, which were collective intelligence to reduce misdiagnoses [32].”
BackgroundSezgın, E., Sirrianni, J., & Linwood, S. L. (2022) · JMIR Medical Informatics · doi:10.2196/32875 · Cited by 152
Predicting dementia from spontaneous speech using large language models
Agbavor, F., & Liang, H. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000168 · Cited by 147
Riches, N., Panagioti, M., Alam, R., Cheraghi‐Sohi, S., Campbell, S., Esmail, A., & Bower, P. (2016) · PLoS ONE · doi:10.1371/journal.pone.0148991 · Cited by 144
Technical Aspects of Developing Chatbots for Medical Applications: Scoping Review
Safi, Z., Abd-Alrazaq, A. A., Khalifa, M., & Househ, M. S. (2020) · Journal of Medical Internet Research · doi:10.2196/19127 · Cited by 140
Differential Diagnosis Generators: an Evaluation of Currently Available Computer Programs
Bond, W. F., Schwartz, L. M., Weaver, K., Levick, D., Giuliano, M., & Graber, M. L. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-011-1804-8 · Cited by 137
Wikipedia and Medicine: Quantifying Readership, Editors, and the Significance of Natural Language
Heilman, J., & West, A. G. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4069 · Cited by 135
Accuracy of online symptom checkers and the potential impact on service utilisation
Ceney, A., Tolond, S., Glowinski, A., Marks, B., Swift, S., & Palser, T. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254088 · Cited by 125 · cited 2×
“Furthermore, compared with previous research on symptom checkers [27], the rate of correct diagnosis within the five differential-diagnosis lists generated by ChatGPT-3 was more than 30% (83.”
BackgroundResultNath, S., Marie, A., Ellershaw, S., Korot, E., & Keane, P. A. (2022) · British Journal of Ophthalmology · doi:10.1136/bjophthalmol-2022-321141 · Cited by 118
The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model
Levine, D. M., Tuwani, R., Kompa, B., Varma, A., Finlayson, S. G., Mehrotra, A., & Beam, A. (2023) · doi:10.1101/2023.01.30.23285067 · Cited by 106
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation
Schmieding, M. L., Kopka, M., Schmidt, K. F. R., Schulz-Niethammer, S., Balzer, F., & Feufel, M. A. (2022) · Journal of Medical Internet Research · doi:10.2196/31810 · Cited by 99
Medical Question Answering for Clinical Decision Support
Goodwin, T. R., & Harabagiu, S. M. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983819 · Cited by 80
Uncovering interpretable potential confounders in electronic medical records
Zeng, J., Gensheimer, M. F., Rubin, D. L., Athey, S. C., & Shachter, R. D. (2022) · Nature Communications · doi:10.1038/s41467-022-28546-8 · Cited by 52
A Review of AI Based Medical Assistant Chatbot
Bulla, C. M., Parushetti, C., Teli, A., Aski, S., & Koppad, S. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3902214 · Cited by 43
Effectiveness of an Ophthalmic Hospital-Based Virtual Service during the COVID-19 Pandemic
Wu, X., Chen, J., Yun, D., Yuan, M., Liu, Z., Yan, P., Sim, D. A., Zhu, Y. M., Chen, C., Hu, W., et al. (2020) · Ophthalmology · doi:10.1016/j.ophtha.2020.10.012 · Cited by 33
Chen, J. H., Dhaliwal, G., & Yang, D. (2022) · JAMA · doi:10.1001/jama.2022.13735 · Cited by 32
External features enriched model for biomedical question answering
Xu, G., Rong, W., Wang, Y., Ouyang, Y., & Xiong, Z. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04176-7 · Cited by 30
Evidence-based retrieval in evidence-based medicine.
Patrick, T. B., Demiris, G., Folk, L. C., Moxley, D., Mitchell, J. A., & Tao, D. (2004) · PubMed Central · Cited by 29
Early Prediction of Organ Failures in Patients with Acute Pancreatitis Using Text Mining
Luo, J., Lan, L., Yang, D., Huang, S., Li, M., Yin, J. Y., Xiao, J., & Zhou, X. (2021) · Scientific Programming · doi:10.1155/2021/6683942 · Cited by 3
The Power and the Pitfalls of Large Language Models: A Fireside Chat with Ricardo Baeza-YatesPreprint
Urs, S. R. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4280575 · Cited by 3
Cited by
ChatGPT and Open-AI Models: A Preliminary Review
Roumeliotis, K. I., & Tselikas, N. D. (2023) · Future Internet · doi:10.3390/fi15060192 · Cited by 706
Haleem, A., Javaid, M., & Singh, R. P. (2022) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100089 · Cited by 694
Utility of ChatGPT in Clinical Practice
Liu, J., Wang, C., & Liu, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48568 · Cited by 505
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
A study of generative large language model for medical research and healthcare
Peng, C., Yang, X., Chen, A., Smith, K. E., PourNejatian, N., Costa, A. B., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00958-w · Cited by 408
Metadata
- Authors
- Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T.
- Venue
- International Journal of Environmental Research and Public Health
- Publisher
- MDPI AG
- Published
- 2023-02-15
- Volume
- 20
- Issue
- 4
- Pages
- 3378
- ISSN
- 1660-4601 (ISSN-L)
- OpenAlex
- W4320920036
- Cited by
- 343
- References
- 35
- FWCI
- 11.80
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.