The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review
Open accessPublished version of a preprint
Abstract
Background: The launch of ChatGPT (OpenAI) in November 2022 attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including health care. Numerous studies have since been conducted regarding how to use state-of-the-art LLMs in health-related scenarios. Objective: This review aims to summarize applications of and concerns regarding conversational LLMs in health care and provide an agenda for future research in this field. Methods: We used PubMed, ACM, and the IEEE digital libraries as primary sources for this review. We followed the guidance of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) to screen and select peer-reviewed research articles that (1) were related to health care applications and conversational LLMs and (2) were published before September 1, 2023, the date when we started paper collection. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 (7.9%) papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT (60/65, 92% of papers), followed by Bard (Google LLC; 1/65, 2% of papers), LLaMA (Meta; 1/65, 2% of papers), and other LLMs (6/65, 9% papers). These papers were classified into four categories of applications: (1) summarization, (2) medical knowledge inquiry, (3) prediction (eg, diagnosis, treatment recommendation, and drug synergy), and (4) administration (eg, documentation and information collection), and four categories of concerns: (1) reliability (eg, training data quality, accuracy, interpretability, and consistency in responses), (2) bias, (3) privacy, and (4) public acceptability. There were 49 (75%) papers using LLMs for either summarization or medical knowledge inquiry, or both, and there are 58 (89%) papers expressing concerns about either reliability or bias, or both. We found that conversational LLMs exhibited promising results in summarization and providing general medical knowledge to patients with a relatively high accuracy. However, conversational LLMs such as ChatGPT are not always able to provide reliable answers to complex health-related tasks (eg, diagnosis) that require specialized domain expertise. While bias or privacy issues are often noted as concerns, no experiments in our reviewed papers thoughtfully examined how conversational LLMs lead to these issues in health care research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications bring bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in health care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · cited 3×
“In addition, Sallam [8] focused on educational and research applications of ChatGPT only.”
The “All of Us” Research Program
The All of Us Research Program Investigators (2019) · New England Journal of Medicine · doi:10.1056/nejmsr1809937 · Cited by 2,440
How to properly use the PRISMA Statement
Sarkis-Onofre, R., Catalá-López, F., Aromataris, E., & Lockwood, C. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01671-z · Cited by 1,104
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P.-S., Trivedi, H., Vipani, A., Ayoub, W., Yang, J. D., Liran, O., Spiegel, B., et al. (2023) · doi:10.1101/2023.02.06.23285449 · Cited by 666
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512
Self-citationZack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., Abdulnour, R.-E. E., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00225-x · Cited by 510
Research works printed but not linked: 68
- 10.1038/s41591-023-02448-8 · DOI · Paper page
- 10.3390/ijerph20043378 · DOI · Paper page
- 10.3390/biomedicines11061623 · DOI
- 10.2196/51580 · DOI · Paper page
- 10.1016/j.ijom.2023.09.005 · DOI
- 10.1093/jamia/ocae035 · DOI
- 10.1038/s41746-023-00873-0 · DOI · Paper page
- 10.1371/journal.pmed.1000097 · DOI · Paper page
- 10.7759/cureus.38249 · DOI
- 10.1067/j.cpradiol.2023.04.001 · DOI
- 10.7189/jogh.13.03037 · DOI
- 10.7759/cureus.36408 · DOI
- 10.7759/cureus.40546 · DOI
- 10.7759/cureus.37589 · DOI
- 10.7759/cureus.37368 · doi:10.7759/cureus.37368
- 10.1136/bmjhci-2023-100775 · DOI
- 10.7759/cureus.36590 · doi:10.7759/cureus.36590
- 10.7759/cureus.42214 · DOI
- 10.7759/cureus.41916 · DOI
- 10.7759/cureus.40135 · DOI
- 10.7759/cureus.38683 · doi:10.7759/cureus.38683
- 10.1515/cclm-2023-0355 · DOI
- 10.1038/s41586-023-06160-y · DOI · Paper page
- 10.18653/v1/2023.clinicalnlp-1.10 · DOI
- 10.2196/48568 · DOI · Paper page
- 10.7759/cureus.38784 · DOI
- 10.7759/cureus.36830 · DOI
- 10.1007/s10916-023-01925-4 · DOI · Paper page
- 10.1097/IOP.0000000000002418 · DOI
- 10.1016/j.xops.2023.100324 · DOI · Paper page
- 10.1145/3594806.3596520 · DOI
- 10.3389/fonc.2023.1219326 · DOI · Paper page
- 10.2196/48966 · DOI
- 10.1253/circj.cj-23-0308 · DOI
- 10.3390/diagnostics13111950 · DOI
- 10.7759/cureus.40895 · DOI · Paper page
- 10.7759/cureus.40822 · DOI · Paper page
- 10.1093/eurjcn/zvad038 · DOI
- 10.1177/08465371231171125 · DOI · Paper page
- 10.3350/cmh.2023.0089 · DOI · Paper page
- 10.1177/20552076231184091 · DOI
- 10.7759/cureus.40351 · DOI
- 10.1016/j.jaip.2023.05.042 · DOI
- 10.1186/s12859-023-05411-z · DOI
- 10.4103/tjem.tjem_79_23 · DOI
- 10.1038/s41523-023-00557-8 · DOI · Paper page
- 10.7759/cureus.36272 · DOI
- 10.1148/radiol.230424 · DOI · Paper page
- 10.1093/jamia/ocad072 · DOI · Paper page
- 10.7759/cureus.37938 · DOI
- 10.3390/biom13020387 · DOI
- 10.1109/access.2023.3300381 · DOI · Paper page
- 10.1136/bmj.h1139 · DOI · Paper page
- 10.1038/s41551-021-00751-8 · DOI · Paper page
- 10.1109/cvpr42600.2020.00033 · DOI · Paper page
- 10.1080/15265161.2023.2233357 · DOI
- 10.1038/s41746-023-00873-0 · DOI · Paper page
- 10.1371/journal.pone.0120592 · DOI
- 10.1016/j.ajhg.2016.12.002 · DOI
- 10.1186/s12920-017-0282-1 · DOI
- 10.1126/sciadv.abe9986 · DOI
- 10.1038/s41576-022-00455-y · DOI
- 10.1038/s41467-022-35295-1 · DOI
- 10.1101/gr.277674.123 · DOI
- 10.1007/978-3-031-69651-0_5 · DOI
- 10.1109/tnnls.2021.3129592 · DOI
- Liu, H, arXiv
- Li, C, arXiv
Cited by
Scientific evidence for clinical text summarization using large language models: scoping review
Bednarczyk, L., Reichenpfader, D., Gaudet-Blavignac, C., Ette, A. K., Zaghir, J., Zheng, Y., Bensahla, A., Bjelogrlic, M., & Lovis, C. (2025) · Journal of Medical Internet Research · doi:10.2196/68998 · Cited by 83
ChatGPT in Research and Education: A SWOT Analysis of Its Academic Impact
Miah, A. S. M., Tusher, M. M. R., Hossain, M. M., Hossain, M. M., Rahim, M. A., Hamid, M. E., Islam, M. S., & Shin, J. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.064168 · Cited by 11
Mondal, H., Tiu, D. N., Mondal, S., Dutta, R., Naskar, A., & Podder, I. (2025) · Journal of Mid-life Health · doi:10.4103/jmh.jmh_182_24 · Cited by 6
Trialing DeepSeek, the new kid on the artificial intelligence block
Desmedt, C. P., & Forsyth, F. (2025) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvaf087 · Cited by 2
Metadata
- Authors
- Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E., Malin, B., & Yin, Z.
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-11-07
- Volume
- 26
- Pages
- e22769
- ISSN
- 1438-8871 (ISSN-L)
- OpenAlex
- W4403087472
- Cited by
- 165
- References
- 35
- FWCI
- 4.81
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review. https://app.cochat.ai/papers/10.2196/22769
- The paper
- Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E., Malin, B., & Yin, Z. (2024). Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review. Journal of Medical Internet Research. https://doi.org/10.2196/22769
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.