A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods: We utilized PubMed, ACM, and IEEE digital libraries as primary sources for this review. We followed the guidance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRIMSA) to screen and select peer-reviewed research articles that (1) were related to both healthcare applications and conversational LLMs and (2) were published before September 1 st , 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · cited 1×
“…the multi-modal LLMs that can process medical images, such as GPT-4, Large Language and Vision Assistant (LLaVA) [85] based on LLaMA, and LLaVA-Med [73] based on LLaVA, were just released before September 2023 and are still being examined by researchers regarding their capabilities in healthcare…”
BackgroundEthical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541
Lyu, Q., Tan, J., Zapadka, M. E., Ponnatapura, J., Niu, C., Myers, K. J., Wang, G., & Whitlow, C. T. (2023) · Visual Computing for Industry, Biomedicine, and Art · doi:10.1186/s42492-023-00136-5 · Cited by 367 · cited 2×
“Many authors expressed concerns about the inconsistency of the responses from LLMs [21,25,30], where different answers result from various prompts of the same question [17,32,39,58,59,64,67,72].”
BackgroundWalker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model
Huang, H., Zheng, O., Wang, D., Yin, J., Wang, Z., Ding, S., Yin, H., Xu, C., Yang, R., Zheng, Q., et al. (2023) · International Journal of Oral Science · doi:10.1038/s41368-023-00239-y · Cited by 324
High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content
Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023) · Cureus · doi:10.7759/cureus.39238 · Cited by 294
Johnson, S. B., King, A. J., Warner, E. L., Aneja, S., Kann, B. H., & Bylund, C. L. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad015 · Cited by 289 · cited 2×
“…biased output; Lee et al. [74] concerned health disparity may result from low readability made by the sophistication of LLM wording; and Johnson et al. [56] noted that LLM algorithms tend to reinforce the health disparity and to prevent LLM algorithms from exacerbating current disparity in health.”
BackgroundCan ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis
Macdonald, C., Adeloye, D., Sheikh, A., & Rudan, I. (2023) · Journal of Global Health · doi:10.7189/jogh.13.01003 · Cited by 243
Research works printed but not linked: 68
- 10.1038/s41591-023-02448-8 · DOI · Paper page
- 10.2196/47564 · DOI · Paper page
- 10.3390/ijerph20043378 · DOI · Paper page
- 10.3390/biomedicines11061623 · DOI
- 10.1016/j.sapharm.2023.03.006 · DOI
- 10.1038/s41746-023-00873-0 · DOI · Paper page
- 10.1371/journal.pmed.1000097 · DOI · Paper page
- 10.1097/MD.0000000000034068 · DOI · Paper page
- 10.7759/cureus.36830 · DOI
- 10.2196/48568 · DOI · Paper page
- 10.7759/cureus.38784 · DOI · Paper page
- 10.1007/s10916-023-01925-4 · DOI · Paper page
- 10.7759/cureus.40546 · DOI
- 10.1067/j.cpradiol.2023.04.001 · DOI · Paper page
- 10.1038/s41586-023-06160-y · DOI · Paper page
- 10.7759/cureus.36408 · DOI
- 10.1136/bmjhci-2023-100775 · DOI · Paper page
- 10.7759/cureus.37368 · doi:10.7759/cureus.37368
- 10.7759/cureus.37589 · DOI · Paper page
- 10.7759/cureus.40135 · DOI · Paper page
- 10.7759/cureus.41916 · DOI
- 10.7759/cureus.38249 · DOI · Paper page
- 10.1038/s41598-023-31412-2 · DOI · Paper page
- 10.7189/jogh.13.03037 · DOI
- 10.7759/cureus.42214 · DOI
- 10.1097/IOP.0000000000002418 · DOI · Paper page
- 10.1253/circj.CJ-23-0308 · DOI · Paper page
- 10.7759/cureus.40351 · DOI
- 10.3390/diagnostics13111950 · DOI · Paper page
- 10.3350/cmh.2023.0089 · DOI · Paper page
- 10.1177/08465371231171125 · DOI · Paper page
- 10.7759/cureus.40822 · DOI · Paper page
- 10.7759/cureus.36590 · doi:10.7759/cureus.36590
- 10.1007/s00405-023-08051-4 · DOI · Paper page
- 10.7759/cureus.35237 · DOI · Paper page
- 10.1177/20552076231184091 · DOI
- 10.1371/journal.pdig.0000198 · DOI · Paper page
- 10.3389/fonc.2023.1219326 · DOI · Paper page
- 10.1016/j.xops.2023.100324 · DOI · Paper page
- 10.2196/46599 · DOI · Paper page
- 10.7759/cureus.40895 · DOI · Paper page
- 10.1016/j.jaip.2023.05.042 · DOI
- 10.2196/48966 · DOI · Paper page
- 10.1093/eurjcn/zvad038 · DOI · Paper page
- 10.1145/3594806.3596520 · DOI
- 10.7759/cureus.38166 · DOI · Paper page
- 10.1093/jamia/ocad072 · DOI · Paper page
- 10.1515/cclm-2023-0355 · DOI · Paper page
- 10.1371/journal.pdig.0000168 · DOI · Paper page
- 10.18653/v1/2023.clinicalnlp-1.10 · DOI
- 10.1186/s12859-023-05411-z · DOI
- 10.4103/tjem.tjem_79_23 · DOI · Paper page
- 10.7759/cureus.37938 · DOI · Paper page
- 10.1038/s41523-023-00557-8 · DOI · Paper page
- 10.7759/cureus.38683 · doi:10.7759/cureus.38683
- 10.1148/radiol.230424 · DOI · Paper page
- 10.7759/cureus.36272 · DOI · Paper page
- 10.1145/3582515.3609536 · DOI · Paper page
- 10.3390/biom13020387 · DOI
- 10.1109/ACCESS.2023.3300381 · DOI · Paper page
- 10.1056/NEJMsr1809937 · DOI · Paper page
- 10.1136/bmj.h1139 · DOI · Paper page
- 10.1038/s41551-021-00751-8 · DOI · Paper page
- 10.48550/arXiv.1911.07135 · DOI · Paper page
- 10.1371/journal.pone.0028071 · DOI · Paper page
- 10.1080/15265161.2023.2233357 · DOI · Paper page
- 10.1016/S2589-7500(23)00225-X · DOI · Paper page
- 10.1101/2023.08.28.23294730 · DOI · Paper page
Cited by
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
“Bedi et al [15], Wang et al [17], Park et al [14], and Pressman et al [16] all reported concerns regarding the strategies used to evaluate these models.”
Gurbuz, S., Bahar, H., Yavuz, U., Keskin, A., Karslioglu, B., & Solak, Y. (2025) · Arthroplasty Today · doi:10.1016/j.artd.2025.101730 · Cited by 9
“These sophisticated AI systems have demonstrated remarkable capabilities in understanding and responding to complex queries, making them potentially valuable assets in addressing patient concerns and providing information [5].”
Goldie, J., Dennis, S., Hipgrave, L., & Coleman, A. (2025) · JMIR Human Factors · doi:10.2196/71065 · Cited by 7
“Likewise, a systematic review preprint on the use of generative AI in health care found that the technology is “heavily applied” for documentation, research, and administration among medical practitioners [30].”
Trialing DeepSeek, the new kid on the artificial intelligence block
Desmedt, C. P., & Forsyth, F. (2025) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvaf087 · Cited by 2
Metadata
- Authors
- Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B. A., & Yin, Z.
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-04-27
- OpenAlex
- W4395688696
- Cited by
- 41
- References
- 16
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare. https://app.cochat.ai/papers/10.1101/2024.04.26.24306390
- The paper
- Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B. A., & Yin, Z. (2024). A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare. medRxiv. https://doi.org/10.1101/2024.04.26.24306390
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.