Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Open accessAbstract
Importance: There is much interest in the clinical integration of large language models (LLMs) in health care. Many studies have assessed the ability of LLMs to provide health advice, but the quality of their reporting is uncertain. Objective: To perform a systematic review to examine the reporting variability among peer-reviewed studies evaluating the performance of generative artificial intelligence (AI)–driven chatbots for summarizing evidence and providing health advice to inform the development of the Chatbot Assessment Reporting Tool (CHART). Evidence Review: A search of MEDLINE via Ovid, Embase via Elsevier, and Web of Science from inception to October 27, 2023, was conducted with the help of a health sciences librarian to yield 7752 articles. Two reviewers screened articles by title and abstract followed by full-text review to identify primary studies evaluating the clinical accuracy of generative AI-driven chatbots in providing health advice (chatbot health advice studies). Two reviewers then performed data extraction for 137 eligible studies. Findings: A total of 137 studies were included. Studies examined topics in surgery (55 [40.1%]), medicine (51 [37.2%]), and primary care (13 [9.5%]). Many studies focused on treatment (91 [66.4%]), diagnosis (60 [43.8%]), or disease prevention (29 [21.2%]). Most studies (136 [99.3%]) evaluated inaccessible, closed-source LLMs and did not provide enough information to identify the version of the LLM under evaluation. All studies lacked a sufficient description of LLM characteristics, including temperature, token length, fine-tuning availability, layers, and other details. Most studies (136 [99.3%]) did not describe a prompt engineering phase in their study. The date of LLM querying was reported in 54 (39.4%) studies. Most studies (89 [65.0%]) used subjective means to define the successful performance of the chatbot, while less than one-third addressed the ethical, regulatory, and patient safety implications of the clinical integration of LLMs. Conclusions and Relevance: In this systematic review of 137 chatbot health advice studies, the reporting quality was heterogeneous and may inform the development of the CHART reporting standards. Ethical, regulatory, and patient safety considerations are crucial as interest grows in the clinical integration of LLMs.
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic review of 137 chatbot health advice studies found substantial heterogeneity and incomplete reporting of LLM characteristics, prompt engineering, query strategy, and performance evaluation. Most studies evaluated closed-source ChatGPT models and used investigator opinion rather than evidence-based reference standards, with less than one-third addressing ethics, patient safety, or regulation.
Key findings
- Of 137 CHAS, 135 (98.5%) assessed ChatGPT and 136 (99.3%) evaluated inaccessible/closed-source models; only 1 (0.7%) fully described the LLM version, and 36 (26.3%) described none of the LLM characteristics.
- Performance evaluation was often subjective: 89 studies (65.0%) used investigators’ opinion without a reference standard, while 21 (15.3%) used clinical practice guideline recommendations; 18 (13.1%) described a standardized evaluation process and 16 (11.7%) reported blinding.
- Reporting of methods was incomplete: 121 (88.3%) did not describe chatbot selection, 40 (29.2%) did not describe query strategy, 37 (27.0%) did not describe prompt source, 136 (99.3%) did not describe prompt testing/engineering, and 20 (14.6%) used standardized prompts.
- Ethical, safety, and regulatory implications were addressed by 45 (32.8%) for ethics, 44 (32.1%) for patient safety, and 22 (16.1%) for regulation of LLMs.
Methods
- Systematic review following PRISMA; protocol prospectively registered on Open Science Framework; searched MEDLINE via Ovid, Embase via Elsevier, and Web of Science on October 27, 2023.
- Two independent researchers screened titles/abstracts and full texts, with a third resolving disagreements; 7,752 articles reviewed yielded 137 eligible CHAS.
- Data extracted in Covidence; descriptive statistics and narrative synthesis were used, with no meta-analysis planned due anticipated heterogeneity; quality of evidence was not evaluated due lack of validated risk-of-bias tools for CHAS.
Limitations
- Stated: reporting in CHAS may change beyond review time due the dynamic nature of the field; much included literature used hypothetical patient cases; there is a paucity of literature detailing ethical and regulatory considerations; future patient-centered prospective research is needed.
- Stated: quality of evidence was not evaluated because no high-quality validated risk-of-bias tools were available for CHAS.
- Evident from the text: heterogeneity in study subjects and designs precluded meta-analysis; the review describes reporting practices and cannot establish causal effects of reporting on clinical outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 65 cited sources checked · 0 supported · 0 flagged
- Findingown data
65 of the included articles (47.4%) included the full transcript of chatbot responses.
“Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.”
Quote in the paperNot a substantial claim - Findingown data
136 of the included articles (99.3%) did not describe a prompt testing/engineering phase.
“Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.”
Quote in the paperNot a substantial claim - Findingown data
60 (43.8%) examined advice regarding diagnosis.
“Articles most commonly examined advice regarding treatment (91 [66.4%]), diagnosis (60 [43.8%]), and differential diagnosis (23 [16.8%]).”
Quote in the paperNot a substantial claim - Findingown data
93 of the included articles (67.9%) shared the full transcript of their prompts.
“Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.”
Quote in the paperNot a substantial claim - Findingown data
Less than a third of studies addressed the ethical, regulatory, and patient safety implications of clinically integrating LLMs.
“Most (64.5%) studies used subjective means as the ground truth to define the successful performance of the LLM, while less than a third addressed the ethical, regulatory, and patient safety implications of clinically integrating LLMs.”
Quote in the paperSubstantial claim
- Findingown data
16 studies (11.7%) explained why the specific LLMs were selected to be studied.
“Overall, 16 studies (11.7%) explained why the specific LLMs were selected to be studied.”
Quote in the paperNot a substantial claim - Findingown data
Most (64.5%) studies used subjective means as the ground truth to define the successful performance of the LLM.
“Most (64.5%) studies used subjective means as the ground truth to define the successful performance of the LLM, while less than a third addressed the ethical, regulatory, and patient safety implications of clinically integrating LLMs.”
Quote in the paperSubstantial claim - Findingown data
39 of the included studies (28.5%) did not describe a structured approach to performance evaluation.
“With respect to performance evaluation of chatbots, few studies described a standardized evaluation process (18 [13.1%]) or blinding (16 [11.7%]), and many studies did not describe a structured approach to performance evaluation (39 [28.5%]).”
Quote in the paperNot a substantial claim - Findingown data
32 of the included studies (23.4%) noted the number of chat windows.
“Study performance was suboptimal in noting the number of chat windows (32 [23.4%]) and the number of queries (48 [35.0%]).”
Quote in the paperNot a substantial claim - Findingown data
5 of the included studies (3.6%) noted the location of their query.
“Few studies noted the date (54 [39.4%]) and location (5 [3.6%]) of their query.”
Quote in the paperNot a substantial claim - Findingown data
No studies described the characteristics of the LLM in comprehensive detail including temperature, token length, fine-tuning availability, penalties, add-on availability, language, and layers.
“and no studies described the characteristics of the LLM in comprehensive detail including temperature (a parameter to control the randomness of LLM output), token length, fine-tuning availability, penalties, add-on availability, language, and layers related to the LLM being studied.”
Quote in the paperNot a substantial claim - Findingown data
Of the 137 articles, 51 (37.2%) were on medical topics.
“Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1).”
Quote in the paperNot a substantial claim - Findingown data
55 (40.1%) were on surgical topics.
“Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1).”
Quote in the paperNot a substantial claim - Findingown data
91 (66.4%) examined advice regarding treatment.
“Articles most commonly examined advice regarding treatment (91 [66.4%]), diagnosis (60 [43.8%]), and differential diagnosis (23 [16.8%]).”
Quote in the paperNot a substantial claim - Findingown data
36 studies (26.3%) described none of the characteristics of the LLM being examined.
“More than one-quarter (36 [26.3%]) of the studies described none of the characteristics of the LLM being examined (Table 2).”
Quote in the paperNot a substantial claim - Findingown data
45 of the included studies (32.8%) addressed the ethical implications of the clinical integration of LLMs.
“Less than one-third of the studies addressed the ethical (45 [32.8%]) and patient safety (44 [32.1%]) implications of the clinical integration of LLMs, while 22 (16.1%) addressed the lack of regulation of LLMs (Table 2).”
Quote in the paperNot a substantial claim - Findingown data
7 (5.1%) were on psychiatry topics.
“Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1).”
Quote in the paperNot a substantial claim - Findingown data
In this systematic review of 137 articles, 99.3% of the studies assessed closed-source models.
“In this systematic review of 137 articles, 99.3% of the studies assessed closed-source models and did not provide enough information to identify the LLM.”
- Also stated in: § Study Characteristics · plain
“136 of the LLMs studied (99.3%) were inaccessible (proprietary/closed-source) models.”
Quote in the paperSubstantial claim - Also stated in: § Study Characteristics · plain
- Findingown data
54 of the included studies (39.4%) noted the date of their query.
“Few studies noted the date (54 [39.4%]) and location (5 [3.6%]) of their query.”
Quote in the paperNot a substantial claim - Findingown data
18 of the included studies (13.1%) described a standardized evaluation process.
“With respect to performance evaluation of chatbots, few studies described a standardized evaluation process (18 [13.1%]) or blinding (16 [11.7%]), and many studies did not describe a structured approach to performance evaluation (39 [28.5%]).”
Quote in the paperNot a substantial claim - Findingown data
A single study (0.7%) fully described the version of the LLM under evaluation.
“A single study (0.7%) fully described the version of the LLM under evaluation”
Quote in the paperNot a substantial claim - Findingown data
37 of the included studies (27.0%) did not explain the source of their prompts.
“More than one-quarter (37 [27.0%]) of the studies did not explain the source of their prompts or describe at least one aspect of their query strategy (40 [29.2%]).”
Quote in the paperNot a substantial claim - Findingown data
16 of the included studies (11.7%) described blinding.
“With respect to performance evaluation of chatbots, few studies described a standardized evaluation process (18 [13.1%]) or blinding (16 [11.7%]), and many studies did not describe a structured approach to performance evaluation (39 [28.5%]).”
Quote in the paperNot a substantial claim - Findingown data
13 (9.5%) were on primary care topics.
“Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1).”
Quote in the paperNot a substantial claim - Findingown data
40 of the included studies (29.2%) did not describe at least one aspect of their query strategy.
“More than one-quarter (37 [27.0%]) of the studies did not explain the source of their prompts or describe at least one aspect of their query strategy (40 [29.2%]).”
Quote in the paperNot a substantial claim - Findingown data
20 of the included studies (14.6%) used standardized prompts to evaluate their LLMs.
“Studies performed poorly (20 [14.6%]) in using standardized prompts to evaluate their LLMs.”
Quote in the paperNot a substantial claim - Findingown data
21 of the included studies (15.3%) used clinical practice guideline recommendations to evaluate LLMs.
“Rather than using clinical practice guideline recommendations as in 21 studies (15.3%), most investigators evaluated LLMs based on their opinion without a reference standard (89 [65.0%]).”
Quote in the paperNot a substantial claim - Findingown data
136 of the LLMs studied (99.3%) were inaccessible (proprietary/closed-source) models.
“and 136 of the LLMs studied (99.3%) were inaccessible (proprietary/closed-source) models.”
Quote in the paperSubstantial claim - Findingown data
99.3% of the studies did not provide enough information to identify the LLM.
“In this systematic review of 137 articles, 99.3% of the studies assessed closed-source models and did not provide enough information to identify the LLM.”
Quote in the paperSubstantial claim - Findingown data
23 (16.8%) examined advice regarding differential diagnosis.
“Articles most commonly examined advice regarding treatment (91 [66.4%]), diagnosis (60 [43.8%]), and differential diagnosis (23 [16.8%]).”
Quote in the paperNot a substantial claim - Findingown data
135 studies (99.3%) assessed ChatGPT to provide health advice.
“Almost all studies (135 [99.3%]) assessed the ability of the OpenAI Chat Generative Pretrained Transformer (ChatGPT) to provide health advice”
Quote in the paperNot a substantial claim - Findingown data
The study team reviewed 7752 articles that yielded 137 eligible CHAS.
“The study team reviewed 7752 articles that yielded 137 eligible CHAS (Figure).”
Quote in the paperSubstantial claim - Findingown data
48 of the included studies (35.0%) noted the number of queries.
“Study performance was suboptimal in noting the number of chat windows (32 [23.4%]) and the number of queries (48 [35.0%]).”
Quote in the paperNot a substantial claim - Findingown data
89 of the included studies (65.0%) evaluated LLMs based on investigator opinion without a reference standard.
“Rather than using clinical practice guideline recommendations as in 21 studies (15.3%), most investigators evaluated LLMs based on their opinion without a reference standard (89 [65.0%]).”
Quote in the paperNot a substantial claim - Findingown data
44 of the included studies (32.1%) addressed the patient safety implications of the clinical integration of LLMs.
“Less than one-third of the studies addressed the ethical (45 [32.8%]) and patient safety (44 [32.1%]) implications of the clinical integration of LLMs, while 22 (16.1%) addressed the lack of regulation of LLMs (Table 2).”
Quote in the paperNot a substantial claim - Findingown data
This systematic review identified large heterogeneity in the methods of CHAS.
“This systematic review identified large heterogeneity in the methods of CHAS and a need for improved reporting standards.”
Quote in the paperSubstantial claim - Findingown data
3 of the included studies (2.2%) reported a power calculation.
“| Yes | 3 (2.2)6,55,68,76,127 |”
Quote in the paperNot a substantial claim - Findingown data
Many studies failed to describe any aspect of the LLM characteristics.
“No study described the characteristics of the LLM under evaluation in sufficient detail to reproduce experiments, and many failed to describe any aspect of the LLM characteristics.”
Quote in the paperSubstantial claim - Findingown data
Most reports did not include reasons for chatbot selection.
“Most reports did not include reasons for chatbot selection, and they seldom described a prompt engineering phase.”
Quote in the paperNot a substantial claim - Findingreasoning
There is a lack of critical information about the characteristics of generative AI-driven chatbots in these studies.
“Generative AI-driven chatbots constitute the intervention in these studies, yet there is a lack of critical information about their characteristics.”
Quote in the paperSubstantial claim - Findingown data
22 of the included studies (16.1%) addressed the lack of regulation of LLMs.
“Less than one-third of the studies addressed the ethical (45 [32.8%]) and patient safety (44 [32.1%]) implications of the clinical integration of LLMs, while 22 (16.1%) addressed the lack of regulation of LLMs (Table 2).”
Quote in the paperNot a substantial claim - Findingnone given
Most studies that described an approach to evaluation relied on anecdotal opinion rather than systematic summaries of available evidence or official guidelines.
“Moreover, most studies that described an approach to evaluation relied on anecdotal opinion rather than systematic summaries of available evidence or official guidelines.”
Quote in the paperSubstantial claim - Findingnone given
Authors frequently apply subjective measures of chatbot performance, such as expert opinion, in CHAS.
“Authors frequently apply subjective measures of chatbot performance, such as expert opinion, in CHAS, limiting the generalizability of their methods and conclusions.”
Quote in the paperSubstantial claim - Findingnone given
Authors infrequently shared prompt and chatbot response transcripts.
“Authors infrequently shared prompt and chatbot response transcripts.”
Quote in the paperNot a substantial claim - Findingown data
No study described the characteristics of the LLM under evaluation in sufficient detail to reproduce experiments.
“No study described the characteristics of the LLM under evaluation in sufficient detail to reproduce experiments, and many failed to describe any aspect of the LLM characteristics.”
Quote in the paperSubstantial claim - Findingown data
134 of the included studies (97.8%) did not report a power calculation.
“| No | 134 (97.8)”
Quote in the paperNot a substantial claim - Findingown data
Many studies failed to adequately report the methods of chatbot performance evaluation.
“Many studies failed to adequately report the methods of chatbot performance evaluation.”
Quote in the paperSubstantial claim - Findingown data
Most included studies did not report a prompt engineering phase.
“Most reports did not include reasons for chatbot selection, and they seldom described a prompt engineering phase.”
- Also stated in: § Reporting of LLM Characteristics and Study Method · plain
“Most included studies did not report a prompt engineering phase.”
Quote in the paperSubstantial claim - Also stated in: § Reporting of LLM Characteristics and Study Method · plain
- Findingown data
In this systematic review of 137 CHAS, detailed and transparent reporting of key aspects often was not included in the studies, such as LLM characteristics, prompt engineering, query strategy, and performance evaluation.
“In this systematic review of 137 CHAS, we noted that detailed and transparent reporting of key aspects often was not included in the studies, such as LLM characteristics, prompt engineering, query strategy, and performance evaluation.”
Quote in the paperSubstantial claim - Findingnone given
Much of the literature evaluated in this review used hypothetical patient cases.
“Second, much of the literature evaluated in this review used hypothetical patient cases.”
Quote in the paperSubstantial claim - Premisecitation
Currently, CHAS are inconsistent in their design, analysis, and reporting.
“Currently, these studies are inconsistent in their design, analysis, and reporting.14,15,16”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [14] Gilson A, Safranek CW, Huang T,. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? the implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023; 9: e45312. doi: 10.2196/45312 36753318 PMC9947764 source
Not checked in the free run.
- cites [15] Williams SC, Starup-Hansen J, Funnell JP,. Can ChatGPT outperform a neurosurgical trainee? a prospective comparative study. Br J Neurosurg. 2024: 1- 10. doi: 10.1080/02688697.2024.2308222 38305239 PMC12090375 source
Not checked in the free run.
- cites [16] Ye C, Zweck E, Ma Z, Smith J, Katz S. Doctor versus artificial intelligence: patient and physician evaluation of large language model responses to rheumatology patient questions in a cross-sectional study. Arthritis Rheumatol. 2024; 76(3): 479- 484. doi: 10.1002/art.42737 37902018 source
Not checked in the free run.
- Interpretationown data
The extent of reporting varies considerably among studies evaluating the clinical accuracy of LLMs providing health advice.
“The findings of this study suggest that the extent of reporting varies considerably among studies evaluating the clinical accuracy of LLMs providing health advice.”
Quote in the paperSubstantial claim - Premisereasoning
CHAS report varying levels of chatbot performance.
“Thus, it is unsurprising that CHAS report varying levels of chatbot performance, raising concerns about their limitations and risks.1,12,13”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [1] Singhal K, Azizi S, Tu T,. Large language models encode clinical knowledge. Nature. 2023; 620(7972): 172- 180. doi: 10.1038/s41586-023-06291-2 37438534 PMC10396962 source
Not checked in the free run.
- cites [12] Sallam M. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare (Basel). 2023; 11(6): 887. doi: 10.3390/healthcare11060887 36981544 PMC10048148 source
Not checked in the free run.
- cites [13] Davis R, Eppler M, Ayo-Ajibola O,. Evaluating the effectiveness of artificial intelligence–powered large language models application in disseminating appropriate and readable health information in urology. J Urol. 2023; 210(4): 688- 694. doi: 10.1097/JU.0000000000003615 37428117 source
Not checked in the free run.
- Premisecitation
Several examples of bias have been reported among LLMs.
“Several examples of bias have been reported among LLMs.184”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [184] Qu Y, Wang J. Performance and biases of large language models in public opinion simulation. Humanit Soc Sci Commun. 2024; 11(1). doi: 10.1057/s41599-024-03609-x
Not checked in the free run.
- Premisecitation
Some LLMs have already been integrated into electronic medical record systems.
“Some LLMs have already been integrated into electronic medical record systems,174 such as the Nuance DAX Copilot, a generative AI-program embedded in an electronic health record software called Epic, which records patient interactions and automatically produces clinical documentation.175,176”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [174] Wornow M, Xu Y, Thapa R,. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit Med. 2023; 6(1): 135. doi: 10.1038/s41746-023-00879-8 37516790 PMC10387101 source
Not checked in the free run.
- cites [175] Microsoft. Microsoft and Epic expand strategic collaboration with integration of Azure OpenAI Service. Accessed March 29, 2024. https://news.microsoft.com/2023/04/17/microsoft-and-epic-expand-strategic-collaboration-with-integration-of-azure-openai-service/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [176] Dahdah R. Microsoft makes the promise of AI in healthcare real through new collaborations with healthcare organizations and partners. Accessed March 28, 2024. https://blogs.microsoft.com/blog/2024/03/11/microsoft-makes-the-promise-of-ai-in-healthcare-real-through-new-collaborations-with-healthcare-organizations-and-partners/
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Several chatbots have experienced inconsistent accuracy by clinical practice guideline standards.
“Currently, few studies use clinical practice guidelines to define successful performance, and several chatbots have experienced inconsistent accuracy by these standards.168,169”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [168] Ahmed W, Saturno M, Rajjoub R,. ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis. Eur Spine J. 2024; 33(11): 4182- 4203. doi: 10.1007/s00586-024-08198-6 38489044 source
Not checked in the free run.
- cites [169] Gianola S, Bargeri S, Castellini G,. Performance of ChatGPT compared to clinical practice guidelines in making informed decisions for lumbosacral radicular pain: a cross-sectional study. J Orthop Sports Phys Ther. 2024; 54(3): 222- 228. doi: 10.2519/jospt.2024.12151 38284363 source
Not checked in the free run.
- Premisecitation
Prompt engineering improves chatbot performance by tailoring LLM output to be focused and helpful for a specific task.
“Prompt engineering improves chatbot performance by tailoring LLM output to be focused and helpful for a specific task.162,163”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [162] Tian S, Jin Q, Yeganova L,. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Brief Bioinform. 2023; 25(1): 1- 13. doi: 10.1093/bib/bbad493 38168838 PMC10762511 source
Not checked in the free run.
- cites [163] Russe MF, Reisert M, Bamberg F, Rau A. Improving the use of LLMs in radiology through prompt engineering: from precision prompts to zero-shot learning. Rofo. 2024; 196(11): 1166- 1170. doi: 10.1055/a-2264-5631 38408477 source
Not checked in the free run.
- Premisecitation
LLMs have in-context learning capabilities that allow them to adapt to the given prompt.
“LLMs have in-context learning capabilities that allow them to adapt to the given prompt, so prompts can be intentionally designed to support performance on further tasks, such as patient communication, administrative tasks, risk assessments, or clinical decision-making.160,161”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [160] Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res. 2023; 25(1): e50638. doi: 10.2196/50638 37792434 PMC10585440 source
Not checked in the free run.
- cites [161] Nguyen D, Swanson D, Newbury A, Kim YH. Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms. Acad Radiol. 2024; 31(5): 1799- 1804. doi: 10.1016/j.acra.2023.11.002 38103973 source
Not checked in the free run.
- Interpretationreasoning
The frequent use of subjective measures of chatbot performance in CHAS limits the generalizability of their methods and conclusions.
“Authors frequently apply subjective measures of chatbot performance, such as expert opinion, in CHAS, limiting the generalizability of their methods and conclusions.”
Quote in the paperSubstantial claim - Premisecitation
Other regulated AI medical technologies, such as radiology image analysis or diagnosis, are trained on domain-specific medical data for targeted uses.
“While other regulated AI medical technologies, such as radiology image analysis177 or diagnosis,178 are trained on domain-specific medical data for targeted uses, LLMs have broader complexity, applicability, and an ability to adapt in real time, which complicates their regulation.2,179,180”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [177] Bitkina OV, Park J, Kim HK. Application of artificial intelligence in medical technologies: a systematic review of main trends. Digit Health. 2023; 9: 20552076231189331. doi: 10.1177/20552076231189331 37485326 PMC10359663 source
Not checked in the free run.
- cites [178] Saenz AD, Harned Z, Banerjee O, Abràmoff MD, Rajpurkar P. Autonomous AI systems in the face of liability, regulations and costs. NPJ Digit Med. 2023; 6(1): 185. doi: 10.1038/s41746-023-00929-1 37803209 PMC10558567 source
Not checked in the free run.
- Premisecitation
LLM applications can be excluded from regulation as medical devices if they were not designed for medical purposes or intended to replace clinical judgment.
“However, LLM applications can be excluded from this regulation if they were not designed for medical purposes or intended to replace clinical judgment.11,171,172”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [11] Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023; 6(1): 120. doi: 10.1038/s41746-023-00873-0 37414860 PMC10326069 source
Not checked in the free run.
- cites [171] FDA. Artificial intelligence and machine learning in software as a medical device. 2024. Accessed March 29, 2024. https://www.fda.gov/medical-devices/software-medical-device-samd/artificialintelligence-and-machine-learning-software-medical-device
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [172] Health Canada. Guidance document: software as a medical device (SaMD): classification examples. 2022. Accessed March 29, 2024. https://www.canada.ca/en/health-canada/services/drugs-health-products/medical-devices/application-information/guidance-documents/software medical-device-guidance/examples.html#a4.3
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation
Without full knowledge of the intervention being applied, it is difficult to interpret LLM performance.
“Without full knowledge of the intervention being applied, it is difficult to interpret LLM performance.9,17,156”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [9] Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N Engl J Med. 2023; 388(13): 1233- 1239. doi: 10.1056/NEJMsr2214184 36988602 source
Not checked in the free run.
- cites [17] Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010; 7(2): e1000217. doi: 10.1371/journal.pmed.1000217 20169112 PMC2821895 source
Not checked in the free run.
- cites [156] Glasziou P, Meats E, Heneghan C, Shepperd S. What is missing from descriptions of treatment in trials and reviews? BMJ. 2008; 336(7659): 1472- 1474. doi: 10.1136/bmj.39590.732037.47 18583680 PMC2440840 source
Not checked in the free run.
- Premisecitation
Preset model characteristics, such as temperature, token length, fine-tuning availability, penalties, add-on availability, language, and layers, further impact LLM functionality.
“preset model characteristics, such as temperature, token length, fine-tuning availability, penalties, add-on availability, language, and layers, further impact their functionality.157,158,159”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [157] Wang C, Liu SX, Awadallah AH. Cost-effective hyperparameter optimization for large language model generation inference. Preprint published online August 8, 2023. arXiv. doi: 10.48550/arXiv.2303.04673 source
Not checked in the free run.
- cites [158] Wang PH, Hsieh SI, Chang SC,. Contextual temperature for language modeling. Published online December 25, 2020. arXiv. doi: 10.48550/arXiv.2012.12575
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [159] Wang R, Wang H, Mi F,. Enhancing Large language models against inductive instructions with dual-critique prompting. Published online March 7, 2024. arXiv. doi: 10.48550/arXiv.2305.13733 source
Not checked in the free run.
- Premisecitation
No devices that use generative AI or LLMs have been approved by the Food and Drug Administration.
“Although no devices that use generative AI or LLMs have been approved by the Food and Drug Administration,173 LLMs are being used in medical applications through this gap in regulation.11”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [173] FDA. Artificial intelligence and machine learning (AI/ML)–enabled medical devices. Accessed March 29, 2024. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationcitation
LLMs are being used in medical applications through the gap in regulation.
“Although no devices that use generative AI or LLMs have been approved by the Food and Drug Administration,173 LLMs are being used in medical applications through this gap in regulation.11”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [11] Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023; 6(1): 120. doi: 10.1038/s41746-023-00873-0 37414860 PMC10326069 source
Not checked in the free run.
- Premisecitation
LLMs replicate their input data.
“While clinical practice is informed by evidence from peer-reviewed research, LLMs replicate their input data and thus may propagate online health misinformation and prejudices,189,190 compromising patient safety.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [189] Meyrowitsch DW, Jensen AK, Sørensen JB, Varga TV. AI chatbots and (mis)information in public health: impact on vulnerable communities. Front Public Health. 2023; 11: 1226776. doi: 10.3389/fpubh.2023.1226776 38026315 PMC10644115
Not checked in the free run.
- cites [190] Khan B, Fatima H, Qureshi A,. Drawbacks of artificial intelligence and their potential solutions in the healthcare sector. Biomed Mater Devices. 2023; 1(2): 1- 8. doi: 10.1007/s44174-023-00063-2 36785697 PMC9908503
Not checked in the free run.
- Premisecitation
The lack of transparency surrounding the development and external validation of closed-source or proprietary LLMs presents additional challenges to their evaluation for health care purposes.
“Moreover, the lack of transparency surrounding the development and external validation of closed-source or proprietary LLMs presents additional challenges to their evaluation for health care purposes.179,180”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [179] Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021; 27(4): 582- 584. doi: 10.1038/s41591-021-01312-x 33820998 source
Not checked in the free run.
- cites [180] Nagendran M, Chen Y, Lovejoy CA,. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies in medical imaging. BMJ. 2020; 368:m689. doi: 10.1136/bmj.m689 PMC7190037 32213531 source
Not checked in the free run.
- Premisecitation
Reported biases among LLMs have included propagation of biases based on race and ethnicity, gender, educational level, social class, sexuality, and age.
“These have included the propagation of biases based on race and ethnicity, gender, educational level, social class, sexuality, and age.184,185,186”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [184] Qu Y, Wang J. Performance and biases of large language models in public opinion simulation. Humanit Soc Sci Commun. 2024; 11(1). doi: 10.1057/s41599-024-03609-x source
Not checked in the free run.
- cites [185] Omiye JA, Lester JC, Spichak S, Rotemberg V, Daneshjou R. Large language models propagate race-based medicine. NPJ Digit Med. 2023; 6(1): 195. doi: 10.1038/s41746-023-00939-z 37864012 PMC10589311 source
Not checked in the free run.
- cites [186] Ayoub NF, Balakrishnan K, Ayoub MS, Barrett TF, David AP, Gray ST. Inherent bias in large language models: a random sampling analysis. Mayo Clin Proc Digit Health. 2024; 2(2): 186- 191. doi: 10.1016/j.mcpdig.2024.03.003 PMC11975844 40207170 source
Not checked in the free run.
- Premisecitation
Many of these biases stem from the data used to train LLMs.
“Many of these biases stem from the data used to train LLMs.184”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [184] Qu Y, Wang J. Performance and biases of large language models in public opinion simulation. Humanit Soc Sci Commun. 2024; 11(1). doi: 10.1057/s41599-024-03609-x
Not checked in the free run.
- Premisecitation
LLMs have been known to produce hallucinations.
“Additionally, LLMs have been known to produce hallucinations.2,187”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [2] Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023; 29(8): 1930- 1940. doi: 10.1038/s41591-023-02448-8 37460753 source
Not checked in the free run.
- cites [187] Huo B, Calabrese E, Sylla P,. The performance of artificial intelligence large language model–linked chatbots in surgical decision-making for gastroesophageal reflux disease. Surg Endosc. 2024; 38(5): 2320- 2330. doi: 10.1007/s00464-024-10807-w 38630178
Not checked in the free run.
- Premisecitation
Large language model performance is largely influenced by the content used for pretraining and fine-tuning.
“Large language model performance is largely influenced by the content used for pretraining and fine-tuning, yet LLMs are generally not trained exclusively with high-quality medical literature or real-world clinical data.188”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [188] Au Yeung J, Kraljevic Z, Luintel A,. AI chatbots not yet ready for clinical use. Front Digit Health. 2023; 5: 1161098. doi: 10.3389/fdgth.2023.1161098 37122812 PMC10130576
Not checked in the free run.
- Premisecitation
LLMs are generally not trained exclusively with high-quality medical literature or real-world clinical data.
“Large language model performance is largely influenced by the content used for pretraining and fine-tuning, yet LLMs are generally not trained exclusively with high-quality medical literature or real-world clinical data.188”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [188] Au Yeung J, Kraljevic Z, Luintel A,. AI chatbots not yet ready for clinical use. Front Digit Health. 2023; 5: 1161098. doi: 10.3389/fdgth.2023.1161098 37122812 PMC10130576
Not checked in the free run.
- Classified as probably not a substantial claim. 24
- May combine two claims. 9
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataSources used to develop prompts included expert opinion (43 [31.4%]).
“Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataSources used to develop prompts included clinical practice guidelines (24 [17.5%]).
“Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data9 (6.6%) were on radiology topics.
“Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenThere were no ongoing studies or studies awaiting classification.
“There were no ongoing studies or studies awaiting classification.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataSources used to develop prompts included professional society websites (28 [20.4%]).
“Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenThere is a paucity of literature that comments in detail on the ethical and regulatory considerations of the clinical integration of LLMs that impact patient safety.
“In addition, there is a paucity of literature that comments in detail on the ethical and regulatory considerations of the clinical integration of LLMs that impact patient safety.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationThis systematic review adheres to the PRISMA reporting guideline.
“This systematic review adheres to the reporting standards of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guideline.19”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [19] Page MJ, McKenzie JE, Bossuyt PM,. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021; 372(71): n71. doi: 10.1136/bmj.n71 33782057 PMC8005924 source
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductnone givenThe quality of the evidence was not evaluated due to the lack of high-quality, validated risk-of-bias tools available for CHAS.
“The quality of the evidence was not evaluated due to the lack of high-quality, validated risk-of-bias tools available for CHAS.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataAn academic health sciences librarian with expertise in systematic reviews aided the study team in designing a comprehensive search strategy.
“To capture all relevant studies, an academic health sciences librarian (S.S.) with expertise in systematic reviews aided the study team in designing a comprehensive search strategy.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationChatbots are typically not designed for medical application and currently lack regulation for their use in medicine.
“However, chatbots are typically not designed for medical application and currently lack regulation for their use in medicine.10”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [10] Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: a taxonomy and systematic review. Comput Methods Programs Biomed. 2024; 245: 108013. doi: 10.1016/j.cmpb.2024.108013 38262126 source
Not checked in the free run.
May combine two claims.
PremisecitationGenerative AI-driven chatbots present unique risks of hallucinating false results and propagating misleading information and advice.
“Furthermore, generative AI-driven chatbots present unique risks of hallucinating (ie, confident but incorrect) false results and propagating misleading information and advice.11”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [11] Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023; 6(1): 120. doi: 10.1038/s41746-023-00873-0 37414860 PMC10326069 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationThe protocol was prospectively registered on Open Sciences Framework.
“Our protocol was prospectively registered on Open Sciences Framework.20”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [20] Registries OSF. Protocol for a scoping review of chatbot assessment studies: guidance for the CHART tool. February 25, 2024. Accessed December 9, 2024. https://osf.io/cxsk3
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
PremisecitationWith a lack of standardized reporting, the assessment of the quality of CHAS and the interpretation of their findings remain challenging.
“With a lack of standardized reporting, the assessment of the quality of CHAS and the interpretation of their findings remain challenging.17”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [17] Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010; 7(2): e1000217. doi: 10.1371/journal.pmed.1000217 20169112 PMC2821895 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPatients have reported interest in seeking medical advice from chatbots.
“Patients have reported interest in seeking medical advice from chatbots,7”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [7] Shahsavar Y, Choudhury A. User intentions to use ChatGPT for self-diagnosis and health-related purposes: cross-sectional survey study. JMIR Hum Factors. 2023; 10: e47564. doi: 10.2196/47564 37195756 PMC10233444 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataTwo independent researchers conducted screening, with a third independent researcher casting a deciding vote to resolve disagreements.
“In all cases, 2 independent researchers (W.T. and J.P.S.) conducted screening, with a third independent researcher casting a deciding vote to resolve cases of disagreement.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe existing literature base emphasizes a structured approach to performance evaluation.
“Additionally, the existing literature base emphasizes a structured approach to performance evaluation.165”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [165] Guyatt GH, Oxman AD, Kunz R,. GRADE guidelines: 2. framing the question and deciding on important outcomes. J Clin Epidemiol. 2011; 64(4): 395- 400. doi: 10.1016/j.jclinepi.2010.09.012 21194891 source
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningAs these publications are largely prepared by clinicians, study teams may not have access to expertise in machine learning.
“As these publications are largely prepared by clinicians, study teams may not have access to expertise in machine learning.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPrompts can be intentionally designed to support performance on further tasks, such as patient communication, administrative tasks, risk assessments, or clinical decision-making.
“LLMs have in-context learning capabilities that allow them to adapt to the given prompt, so prompts can be intentionally designed to support performance on further tasks, such as patient communication, administrative tasks, risk assessments, or clinical decision-making.160,161”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [160] Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res. 2023; 25(1): e50638. doi: 10.2196/50638 37792434 PMC10585440 source
Not checked in the free run.
- cites [161] Nguyen D, Swanson D, Newbury A, Kim YH. Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms. Acad Radiol. 2024; 31(5): 1799- 1804. doi: 10.1016/j.acra.2023.11.002 38103973 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationArtificial intelligence-linked medical devices are considered software as a medical device and are regulated under federal departments such as Health Canada and the US Food and Drug Administration.
“Artificial intelligence–linked medical devices are considered software as a medical device and are regulated under federal departments such as Health Canada and the US Food and Drug Administration.170”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [170] Da Silva M, Flood CM, Goldenberg A, Singh D. Regulating the safety of health–related artificial intelligence. Healthc Policy. 2022; 17(4): 63- 77. doi: 10.12927/hcpol.2022.26824 35686827 PMC9170055 source
Not checked in the free run.
May combine two claims.
PremisecitationLLMs have broader complexity, applicability, and an ability to adapt in real time, which complicates their regulation.
“While other regulated AI medical technologies, such as radiology image analysis177 or diagnosis,178 are trained on domain-specific medical data for targeted uses, LLMs have broader complexity, applicability, and an ability to adapt in real time, which complicates their regulation.2,179,180”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [2] Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023; 29(8): 1930- 1940. doi: 10.1038/s41591-023-02448-8 37460753 source
Not checked in the free run.
- cites [179] Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021; 27(4): 582- 584. doi: 10.1038/s41591-021-01312-x 33820998 source
Not checked in the free run.
- cites [180] Nagendran M, Chen Y, Lovejoy CA,. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies in medical imaging. BMJ. 2020; 368:m689. doi: 10.1136/bmj.m689 PMC7190037 32213531 source
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationDefining a ground truth or evaluation dataset is an essential step in LLM performance evaluation.
“Many studies describe this as defining a ground truth or evaluation dataset, and it is an essential step in LLM performance evaluation.166,167”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [166] Chang Y, Wang X, Wang J,. A survey on evaluation of large language models. ACM Trans Intell Syst Technol. 2024; 15(3): 1- 45. doi: 10.1145/3641289 source
Not checked in the free run.
- cites [167] Liang JJ, Tsou CH, Devarakonda MV. Ground truth creation for complex clinical NLP tasks: an iterative vetting approach and lessons learned. AMIA Jt Summits Transl Sci Proc. 2017; 2017: 203- 212. 28815130 PMC5543376
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningJust as sufficient detail about medical or surgical interventions must be reported in traditional studies to reproduce experiments and guide critical appraisal, LLM characteristics are similarly vital.
“Just as sufficient detail about medical or surgical interventions must be reported in traditional studies to reproduce experiments and guide critical appraisal, LLM characteristics are similarly vital.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).
Study design and conductnone givenBy engineering a comprehensive search strategy, we aimed to capture an accurate representation of the literature.
“By engineering a comprehensive search strategy, we aimed to capture an accurate representation of the literature.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationThe World Health Organization released guidance on ethics and governance of AI for health, which overlaps substantially with LLMs, which are unimodal models.
“The World Health Organization released guidance on ethics and governance of AI for health,182 which overlaps substantially with LLMs, which are unimodal models.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [182] World Health Organization. Ethics and governance of artificial intelligence for health. Accessed November 4, 2024. https://www.who.int/publications/i/item/9789240029200
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Study design and conductreasoningThe reporting used in CHAS may change beyond the time of this review due to the dynamic nature of this field.
“First, the reporting used in CHAS may change beyond the time of this review due to the dynamic nature of this field.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningLLMs may propagate online health misinformation and prejudices, compromising patient safety.
“While clinical practice is informed by evidence from peer-reviewed research, LLMs replicate their input data and thus may propagate online health misinformation and prejudices,189,190 compromising patient safety.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [189] Meyrowitsch DW, Jensen AK, Sørensen JB, Varga TV. AI chatbots and (mis)information in public health: impact on vulnerable communities. Front Public Health. 2023; 11: 1226776. doi: 10.3389/fpubh.2023.1226776 38026315 PMC10644115
Not checked in the free run.
- cites [190] Khan B, Fatima H, Qureshi A,. Drawbacks of artificial intelligence and their potential solutions in the healthcare sector. Biomed Mater Devices. 2023; 1(2): 1- 8. doi: 10.1007/s44174-023-00063-2 36785697 PMC9908503
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationConcerns have been raised related to data privacy of patient health information among proprietary LLMs.
“Concerns have also been raised related to data privacy of patient health information among proprietary LLMs.182,191”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [182] World Health Organization. Ethics and governance of artificial intelligence for health. Accessed November 4, 2024. https://www.who.int/publications/i/item/9789240029200
Not checked in the free run.
- cites [191] Yao Y, Duan J, Xu K, Cai Y, Sun Z, Zhang Y. A survey on large language model (LLM) security and privacy: the good, the bad, and the ugly. High-Confidence Computing. 2024; 4(2). doi: 10.1016/j.hcc.2024.100211
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
InterpretationcitationTaken together, these factors raise major issues as we move from the technological development of LLMs to their clinical validation for use in health care settings.
“Taken together, these factors raise major issues as we move from the technological development of LLMs to their clinical validation for use in health care settings.182”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [182] World Health Organization. Ethics and governance of artificial intelligence for health. Accessed November 4, 2024. https://www.who.int/publications/i/item/9789240029200
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationStandardized frameworks are under development to establish detailed reporting standards to help authors design and report these studies.
“Standardized frameworks are under development to establish detailed reporting standards to help authors design and report these studies.18”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [18] Huo B, Cacciamani GE, Collins GS, McKechnie T, Lee Y, Guyatt G. Reporting standards for the use of large language model-linked chatbots for health advice. Nat Med. 2023; 29(12): 2988. doi: 10.1038/s41591-023-02656-2 37957381 source
Not checked in the free run.
May combine two claims.
PremisecitationThere is a potential for misuse of patient data, medicolegal implications, and loss of physician-patient trust.
“There is a potential for misuse of patient data, medicolegal implications, and loss of physician-patient trust.181,192”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [181] Ong JCL, Chang SYH, William W,. Ethical and regulatory challenges of large language models in medicine. Lancet Digit Health. 2024; 6(6): e428- e432. doi: 10.1016/S2589-7500(24)00061-X 38658283 source
Not checked in the free run.
- cites [192] Wang C, Liu S, Yang H, Guo J, Wu Y, Liu J. Ethical considerations of using ChatGPT in health care. J Med Internet Res. 2023; 25: e48009. doi: 10.2196/48009 37566454 PMC10457697 source
Not checked in the free run.
The paper relies on this without support.
The generated output depends greatly on the initial input.
“However, the generated output depends greatly on the initial input.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work on patient safety of patient-facing LLM applications has primarily focused on particular diagnoses or specialties.
“Prior work to date on patient safety of patient-facing applications has primarily been focused on particular diagnoses or specialties”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [10]
The Results (Study Characteristics; page not available) classify most of the 137 chatbot health-advice studies by medical or surgical specialty, consistent with a specialty-focused literature. But the Findings say fewer than a third addressed patient-safety implications, so the paper does not establish that prior patient-safety work specifically was primarily focused on particular diagnoses or specialties. (LLM, conf=0.79)
Experimental — accuracy not yet measuredThere have been no previous studies on patient safety of patient-facing applications for a wide range of primary care questions that patients are likely to ask.
“To the best of our knowledge, there have been no previous studies on patient safety of patient-facing applications for a wide range of primary care questions that patients are likely to ask.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [10]
The Discussion > Limitations section says that much of the reviewed literature used hypothetical patient cases and calls for patient-centered, prospective research; it also identifies a paucity of work addressing patient-safety considerations. It does not establish that no previous study assessed the safety of patient-facing applications across a wide range of likely primary care questions. No page number is available in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measuredA recent review found that the majority of studies (68%) on LLMs for medical advice did not consider patient safety.
“Within this review, the authors noted that the majority of studies (68%) did not consider patient safety.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Percentage for omitted chatbot responses is incorrect
MinorTwenty-nine of 137 studies is 21.2%, not 21.7%.
Reported 21.7 · recomputed 21.167883
“| No | 29 (21.7)21,22,28,35,36,37,42,51,55,60,76,78,79,80,82,88,93,108,110,113,128,130,138,140,143,150,151,152,154 |”
Patient-safety percentage is incorrect in Table 2
MinorThe table gives 32.8% for 44 of 137 studies; the percentage is 32.1%, as stated later in the text.
Reported 32.8 · recomputed 32.116788
“| Patient safety | 44 (32.8)”
A study is listed as both giving and not giving a chatbot-selection reason
MinorStudy 33 appears under selection based on accessibility and under selection method "Not described."
“| Accessibility (open vs closed) | 4 (2.9)33,37,123,147 |”
“| Not described | 121 (88.3)6,13,21,22,24,27,28,29,30,33,34,35”
Power-calculation count conflicts with its study list
MinorThe row says 3 studies performed a power calculation but lists five study references.
“| Yes | 3 (2.2)6,55,68,76,127 |”
“6,55,68,76,127”
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · see citing works
Self-citationRevolutionizing healthcare: the role of artificial intelligence in clinical practice
Alowais, S. A., Alghamdi, S. S., Alsuhebany, N., Alqahtani, T., Alshaya, A. I., Almohareb, S. N., Aldairem, A., Alrashed, M., Bin Saleh, K., Badreldin, H. A., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04698-z · Cited by 3,146 · see citing works
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
GRADE guidelines: 2. Framing the question and deciding on important outcomes
Guyatt, G. H., Oxman, A. D., Kunz, R., Atkins, D., Brozek, J., Vist, G., Alderson, P., Glasziou, P., Falck-Ytter, Y., & Schünemann, H. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.012 · Cited by 1,911 · see citing works
Self-citationNatural language processing: state of the art, current trends and challenges
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13428-4 · Cited by 1,823 · see citing works
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738 · see citing works
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437 · see citing works
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152 · see citing works
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · see citing works
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709 · see citing works
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 655 · see citing works
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614 · see citing works
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598 · see citing works
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541 · see citing works
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494 · see citing works
Accuracy and Reliability of Chatbot Responses to Physician Questions
Goodman, R. S., Patrinely, J. R., Stone, C. A., Zimmerman, E., Donald, R. R., Chang, S. S., Berkowitz, S. T., Finn, A. P., Jahangir, E., Scoville, E. A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36483 · Cited by 484 · see citing works
Rao, A. S., Kim, J., Kamineni, M., Pang, M., Lie, W., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.05.003 · Cited by 481 · see citing works
Drawbacks of artificial intelligence and their potential solutions in the healthcare sector.
khan, B., Fatima, H., Qureshi, A., Kumar, S., Hanan, A., Hussain, J., & Abdullah, S. (2023) · Biomedical Materials & Devices · doi:10.1007/s44174-023-00063-2 · Cited by 461 · see citing works
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417 · see citing works
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412 · see citing works
Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397 · see citing works
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 345 · see citing works
Ethical and regulatory challenges of large language models in medicine
Ong, J. C. L., Chang, S. Y.-H., William, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi-Velez, F., Lu, W., Savulescu, J., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00061-x · Cited by 342 · see citing works
Shahsavar, Y., & Choudhury, A. (2023) · JMIR Human Factors · doi:10.2196/47564 · Cited by 341 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 316 · see citing works
How AI Responds to Common Lung Cancer Questions: ChatGPT versus Google Bard
Rahsepar, A. A., Tavakoli, N., Kim, G. H. J., Hassani, C., Abtin, F. G., & Bedayat, A. (2023) · Radiology · doi:10.1148/radiol.230922 · Cited by 283 · see citing works
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282 · see citing works
Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About Cancer
Pan, A., Musheyev, D., Bockelman, D., Loeb, S., & Kabarriti, A. E. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2947 · Cited by 278 · see citing works
Assessing ChatGPT Responses to Common Patient Questions Regarding Total Hip Arthroplasty
Mika, A. P., Martin, J. R., Engstrom, S. M., Polkowski, G. G., & Wilson, J. M. (2023) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.00209 · Cited by 273 · see citing works
Hopkins, A. M., Logan, J. M., Kichenadasse, G., & Sorich, M. J. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad010 · Cited by 259 · see citing works
Use of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251 · see citing works
Bernstein, I. A., Zhang, Y., Govil, D., Majid, I., Chang, R. T., Sun, Y., Shue, A., Chou, J. C., Schehlein, E. M., Christopher, K. L., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.30320 · Cited by 250 · see citing works
Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT
Haver, H. L., Ambinder, E. B., Bahl, M., Oluyemi, E. T., Jeudy, J., & Yi, P. H. (2023) · Radiology · doi:10.1148/radiol.230424 · Cited by 247 · see citing works
Gravel, J., D’Amours-Gravel, M., & Osmanlliu, E. (2023) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2023.05.004 · Cited by 222 · see citing works
Large language model (ChatGPT) as a support tool for breast tumor board
Sorin, V., Klang, E., Sklair‐Levy, M., Cohen, I., Zippel, D. B., Lahat, N. B., Konen, E., & Barash, Y. (2023) · npj Breast Cancer · doi:10.1038/s41523-023-00557-8 · Cited by 216 · see citing works
Can ChatGPT be used in oral and maxillofacial surgery?
Balel, Y. (2023) · Journal of Stomatology Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2023.101471 · Cited by 201 · see citing works
Xie, Y., Seth, I., Hunter-Smith, D. J., Rozen, W. M., Ross, R., & Lee, M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03338-7 · Cited by 198 · see citing works
Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K. Y., Mansour, H. A., Abishek, R. M., Xu, D. S., Sridhar, J. S., et al. (2023) · Ophthalmology Retina · doi:10.1016/j.oret.2023.05.022 · Cited by 179 · see citing works
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176 · see citing works
Factors affecting the quality and reliability of online health information
Battineni, G., Baldoni, S., Chintalapudi, N., Sagaro, G. G., Pallotta, G., Nittari, G., & Amenta, F. (2020) · Digital Health · doi:10.1177/2055207620948996 · Cited by 174 · see citing works
Deiana, G., Dettori, M., Arghittu, A., Azara, A., Gabutti, G., & Castiglia, P. (2023) · Vaccines · doi:10.3390/vaccines11071217 · Cited by 168 · see citing works
Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., & Tortora, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121186 · Cited by 154 · see citing works
Rau, A., Rau, S., Zöller, D., Fink, A., Tran, H. D., Wilpert, C., Nattenmüller, J., Neubauer, J., Bamberg, F., Reisert, M., et al. (2023) · Radiology · doi:10.1148/radiol.230970 · Cited by 152 · see citing works
Artificial intelligence‐based chatbot patient information on common retinal diseases using ChatGPT
Potapenko, I. O., Boberg‐Ans, L. C., Hansen, M. S., Klefter, O. N., van Dijk, E. H. C., & Subhi, Y. (2023) · Acta Ophthalmologica · doi:10.1111/aos.15661 · Cited by 148 · see citing works
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146 · see citing works
Zhu, L., Mou, W., & Chen, R. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04123-5 · Cited by 144 · see citing works
Bulck, L. V., & Moons, P. (2023) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvad038 · Cited by 143 · see citing works
Quality of information and appropriateness of ChatGPT outputs for urology patients
Cocci, A., Pezzoli, M., Re, M. L., Russo, G. I., Asmundo, M. G., Fode, M. M., Cacciamani, G. E., Cimino, S., Minervini, A., & Durukan, E. (2023) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-023-00705-y · Cited by 131 · see citing works
Hirosawa, T., Kawamura, R., Harada, Y., Mizuta, K., Tokumasu, K., Kaji, Y., Suzuki, T., & Shimizu, T. (2023) · JMIR Medical Informatics · doi:10.2196/48808 · Cited by 130 · see citing works
The Potential of ChatGPT as a Self-Diagnostic Tool in Common Orthopedic Diseases: Exploratory Study
Kuroiwa, T., Sarcon, A. K., Ibara, T., Yamada, E., Yamamoto, A., Tsukamoto, K., & Fujita, K. (2023) · Journal of Medical Internet Research · doi:10.2196/47621 · Cited by 129 · see citing works
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128 · see citing works
Musheyev, D., Pan, A., Loeb, S., & Kabarriti, A. E. (2023) · European Urology · doi:10.1016/j.eururo.2023.07.004 · Cited by 126 · see citing works
Wagner, M. W., & Ertl-Wagner, B. B. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231171125 · Cited by 125 · see citing works
Seth, I., Cox, A., Xie, Y., Bulloch, G., Hunter‐Smith, D. J., Rozen, W. M., & Ross, R. J. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad140 · Cited by 125 · see citing works
Al-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N. (2023) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s425858 · Cited by 125 · see citing works
Moshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124 · see citing works
Vaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Bolzoni, A. R., Committeri, U., Crimi, S., et al. (2023) · Otolaryngology · doi:10.1002/ohn.489 · Cited by 123 · see citing works
Davis, R., Eppler, M., Ayo-Ajibola, O., Loh-Doyle, J. C., Nabhani, J., Samplaski, M., Gill, I., & Cacciamani, G. E. (2023) · Journal of Urology · doi:10.1097/ju.0000000000003615 · Cited by 119 · see citing works
Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study
Levkovich, I., & Elyoseph, Z. (2023) · JMIR Mental Health · doi:10.2196/51232 · Cited by 119 · see citing works
Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4
Krusche, M., Callhoff, J., Knitza, J., & Ruffer, N. (2023) · Rheumatology International · doi:10.1007/s00296-023-05464-6 · Cited by 116 · see citing works
Evaluating Artificial Intelligence Responses to Public Health Questions
Ayers, J. W., Zhu, Z., Poliak, A., Leas, E. C., Dredze, M. H., Hogarth, M. A., & Smith, D. M. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.17517 · Cited by 115 · see citing works
Chiesa-Estomba, C. M., Lechien, J. R., Vaira, L. A., Brunet, A., Cammaroto, G., Mayo‐Yáñez, M., Barrueco, Á. S., & Saga-Gutiérrez, C. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08104-8 · Cited by 106 · see citing works
Hurley, E. T., Crook, B. S., Lorentz, S. G., Danilkowicz, R. M., Lau, B. C., Taylor, D. C., Dickens, J. F., Anakwenze, O. A., & Klifto, C. S. (2023) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2023.07.048 · Cited by 103 · see citing works
Gordon, E. B., Towbin, A. J., Wingrove, P. M., Shafique, U., Haas, B. M., Kitts, A. B., Feldman, J., & Furlan, A. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.09.011 · Cited by 102 · see citing works
Artificial intelligence chatbot performance in triage of ophthalmic conditions
Lyons, R. J., Arepalli, S. R., Fromal, O., Choi, J. D., & Jain, N. (2024) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2023.07.016 · Cited by 101 · see citing works
Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty
Dubin, J. A., Bains, S. S., Chen, Z., Hameed, D., Nace, J., Mont, M. A., & Delanois, R. E. (2023) · The Journal of Arthroplasty · doi:10.1016/j.arth.2023.04.007 · Cited by 100 · see citing works
Haemmerli, J., Sveikata, L., Nouri, A., May, A. T., Egervari, K. L., Freyschlag, C. F., Lobrinus, J. A., Migliorini, D., Momjian, S., Sanda, N., et al. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100775 · Cited by 99 · see citing works
Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions
Ayoub, N., Lee, Y.-J., Grimm, D. R., & Balakrishnan, K. (2023) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2023.0704 · Cited by 98 · see citing works
Self-citationCoşkun, B., Ocakoğlu, G., Yetemen, M., & Kaygısız, O. (2023) · Urology · doi:10.1016/j.urology.2023.05.040 · Cited by 97 · see citing works
Nastasi, A. J., Courtright, K. R., Halpern, S. D., & Weissman, G. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-45223-y · Cited by 96 · see citing works
ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results
Hristidis, V., Ruggiano, N., Brown, E. L., Ganta, S. R. R., & Stewart, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48966 · Cited by 93 · see citing works
Lahat, A., Shachar, E., Avidan, B., Glicksberg, B. S., & Klang, E. (2023) · Diagnostics · doi:10.3390/diagnostics13111950 · Cited by 92 · see citing works
Harskamp, R. E., & Clercq, L. D. (2024) · Acta cardiologica. Supplementum · doi:10.1080/00015385.2024.2303528 · Cited by 92 · see citing works
Harskamp, R. E., & De Clercq, L. (2023) · doi:10.1101/2023.03.25.23285475 · Cited by 92 · see citing works
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90 · see citing works
Chervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.05.151 · Cited by 87 · see citing works
Seth, I., Lim, B., Xie, Y., Cevik, J., Rozen, W. M., Ross, R. J., & Lee, M. (2023) · Aesthetic Surgery Journal Open Forum · doi:10.1093/asjof/ojad084 · Cited by 86 · see citing works
Gebrael, G., Sahu, K. K., Chigarira, B., Tripathi, N., Mathew Thomas, V., Sayegh, N., Maughan, B. L., Agarwal, N., Swami, U., & Li, H. (2023) · Cancers · doi:10.3390/cancers15143717 · Cited by 85 · see citing works
Biswas, S., Logan, N. S., Davies, L. N., Sheppard, A. L., & Wolffsohn, J. S. (2023) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13207 · Cited by 84 · see citing works
Autonomous AI systems in the face of liability, regulations and costs.
Saenz, A. D., Harned, Z., Banerjee, O., Abràmoff, M. D., & Rajpurkar, P. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00929-1 · Cited by 83 · see citing works
Evaluating ChatGPT responses on obstructive sleep apnea for patient education
Campbell, D. J., Estephan, L. E., Mastrolonardo, E. V., Amin, D. R., Huntley, C. T., & Boon, M. S. (2023) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.10728 · Cited by 82 · see citing works
Levkovich, I., & Elyoseph, Z. (2023) · Family Medicine and Community Health Journal · doi:10.1136/fmch-2023-002391 · Cited by 81 · see citing works
Utilizing ChatGPT-4 for Providing Medical Information on Blepharoplasties to Patients
Cox, A., Seth, I., Xie, Y., Hunter‐Smith, D. J., & Rozen, W. M. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad096 · Cited by 79 · see citing works
Performance and biases of large language models in public opinion simulation.
Qu, Y., & Wang, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03609-x · Cited by 79 · see citing works
Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettes
Franco D’Souza, R., Amanullah, S., Mathew, M., & Surapaneni, K. M. (2023) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2023.103770 · Cited by 77 · see citing works
Rasmussen, M. L. R., Larsen, A., Subhi, Y., & Potapenko, I. O. (2023) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-023-06078-1 · Cited by 76 · see citing works
ChatGPT as a Complementary Mental Health Resource: A Boon or a Bane
Farhat, F. (2023) · doi:10.20944/preprints202307.1479.v1 · Cited by 76 · see citing works
ChatGPT and Lacrimal Drainage Disorders: Performance and Scope of Improvement
Ali, M. J. (2023) · Ophthalmic Plastic and Reconstructive Surgery · doi:10.1097/iop.0000000000002418 · Cited by 75 · see citing works
AI chatbots and (mis)information in public health: impact on vulnerable communities.
Meyrowitsch, D. W., Jensen, A. K., Sørensen, J. B., & Varga, T. V. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1226776 · Cited by 73 · see citing works
Ye, C., Zweck, E., Ma, Z., Smith, J., & Katz, S. (2024) · Arthritis & Rheumatology · doi:10.1002/art.42737 · Cited by 69 · see citing works
Evaluating the performance of ChatGPT in answering questions related to pediatric urology
Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., Gelmiş, M., Sarılar, Ö., & Özgör, F. (2023) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2023.08.003 · Cited by 67 · see citing works
Is ChatGPT an Evidence-based Doctor?
Zhou, Z., Wang, X., Li, X., & Liao, L. (2023) · European Urology · doi:10.1016/j.eururo.2023.03.037 · Cited by 66 · see citing works
Application of artificial intelligence in medical technologies: a systematic review of main trends.
Bitkina, O. V., Park, J., & Kim, H. K. (2023) · Digital Health · doi:10.1177/20552076231189331 · Cited by 66 · see citing works
Sezgin, E., Chekeni, F., Lee, J., & Keim, S. (2023) · Journal of Medical Internet Research · doi:10.2196/49240 · Cited by 65 · see citing works
Coşkun, B. N., Yağız, B., Ocakoğlu, G., Dalkılıç, E., & Pehlivan, Y. (2023) · Rheumatology International · doi:10.1007/s00296-023-05473-5 · Cited by 65 · see citing works
Large language models will not replace healthcare professionals: curbing popular fears and hype.
Thirunavukarasu, A. J. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231173123 · Cited by 63 · see citing works
Self-citationSzczesniewski, J. J., Fouz, C. T., Alba, A. R., Díaz-Goizueta, F. J., García-Tello, A., & González, L. L. (2023) · World Journal of Urology · doi:10.1007/s00345-023-04563-0 · Cited by 63 · see citing works
Seth, I., Xie, Y., Rodwell, A., Gracias, D., Bulloch, G., Hunter‐Smith, D. J., & Rozen, W. M. (2023) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2023.07.003 · Cited by 62 · see citing works
Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios
Qu, R. W., Qureshi, U., Petersen, G., & Lee, S. C. (2023) · OTO Open · doi:10.1002/oto2.67 · Cited by 62 · see citing works
Maillard, A., Micheli, G., Lefèvre, L., Guyonnet, C., Poyart, C., Canouï, É., Belan, M., & Charlier, C. (2023) · Clinical Infectious Diseases · doi:10.1093/cid/ciad632 · Cited by 62 · see citing works
Gianola, S., Bargeri, S., Castellini, G., Cook, C., Palese, A., Pillastrini, P., Salvalaggio, S., Turolla, A., & Rossettini, G. (2024) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2024.12151 · Cited by 62 · see citing works
ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases
Mehnen, L., Gruarin, S., Vasileva, M., & Knapp, B. (2023) · doi:10.1101/2023.04.20.23288859 · Cited by 60 · see citing works
ChatGPT: Can You Prepare My Patients for [18F]FDG PET/CT and Explain My Reports?
Rogasch, J. M. M., Metzger, G., Preisler, M., Galler, M., Thiele, F., Brenner, W., FELDHAUS, F. W., Wetz, C., Amthauer, H., Furth, C., et al. (2023) · Journal of Nuclear Medicine · doi:10.2967/jnumed.123.266114 · Cited by 59 · see citing works
Kao, H.-J., Chien, T.-W., Wang, W.-C., Chou, W., & Chow, J. C. (2023) · Medicine · doi:10.1097/md.0000000000034068 · Cited by 57 · see citing works
Gabriel, J., Shafik, L., Alanbuki, A., & Larner, T. (2023) · International Urology and Nephrology · doi:10.1007/s11255-023-03729-4 · Cited by 57 · see citing works
Evaluating the performance of ChatGPT in answering questions related to urolithiasis
Çakır, H., Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., & Özgör, F. (2023) · International Urology and Nephrology · doi:10.1007/s11255-023-03773-0 · Cited by 57 · see citing works
Nielsen, J. P. S., von Buchwald, C., & Grønhøj, C. (2023) · Acta Oto-Laryngologica · doi:10.1080/00016489.2023.2254809 · Cited by 55 · see citing works
Kusunose, K., Kashima, S., & Sata, M. (2023) · Circulation Journal · doi:10.1253/circj.cj-23-0308 · Cited by 54 · see citing works
Salazar, G. Z., Zúñiga, D., Vindel-Zuñiga, C., Yoong, A. M., Hincapié-Arias, S. M., Zúñiga, A. B., Zúñiga, P., Salazar, E., & Zúñiga, B. (2023) · Cureus · doi:10.7759/cureus.45473 · Cited by 54 · see citing works
Stroop, A., Stroop, T., Alsofy, S. Z., Nakamura, M., Möllmann, F., Greiner, C., & Stroop, R. (2023) · European Spine Journal · doi:10.1007/s00586-023-07975-z · Cited by 53 · see citing works
Lim, D. Y. Z., Tan, Y. B., Koh, J. T. E., Tung, J. Y. M., Sng, G. G. R., Tan, D. M. Y., & Tan, C. (2023) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.16375 · Cited by 53 · see citing works
Reporting standards for the use of large language model-linked chatbots for health advice.
Huo, B., Cacciamani, G. E., Collins, G. S., McKechnie, T., Lee, Y., & Guyatt, G. (2023) · Nature Medicine · doi:10.1038/s41591-023-02656-2 · Cited by 53 · see citing works
Self-citationChatGPT vs. web search for patient questions: what does ChatGPT do better?
Shen, S. A., Perez-Heydrich, C. A., Xie, D. X., & Nellis, J. C. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08524-0 · Cited by 53 · see citing works
Russe, M. F., Reisert, M., Bamberg, F., & Rau, A. (2024) · RöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren · doi:10.1055/a-2264-5631 · Cited by 52 · see citing works
Capacity of ChatGPT to Identify Guideline-Based Treatments for Advanced Solid Tumors
Schulte, B. C. (2023) · Cureus · doi:10.7759/cureus.37938 · Cited by 50 · see citing works
Lower, K., Seth, I., Lim, B., & Seth, N. (2023) · Indian Journal of Orthopaedics · doi:10.1007/s43465-023-00967-7 · Cited by 50 · see citing works
ChatGPT performance in laryngology and head and neck surgery: a clinical case-series
Lechien, J. R., Georgescu, B. M., Hans, S., & Chiesa-Estomba, C. M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08282-5 · Cited by 50 · see citing works
Emile, S. H., Horesh, N., Freund, M. R., Pellino, G., Oliveira, L. C. C., Wignakumar, A., & Wexner, S. D. (2023) · Surgery · doi:10.1016/j.surg.2023.06.005 · Cited by 49 · see citing works
Shao, C.-Y., Li, H., Liu, X.-L., Li, C., Yang, L.-Q., Zhang, Y.-J., Luo, J., & Zhao, J. (2023) · Interactive Journal of Medical Research · doi:10.2196/46900 · Cited by 49 · see citing works
Dwyer, T., Hoit, G., Burns, D., Higgins, J. P., Chang, J. S., Whelan, D. B., Kiroplis, I., & Chahal, J. (2023) · Arthroscopy Sports Medicine and Rehabilitation · doi:10.1016/j.asmr.2023.01.020 · Cited by 48 · see citing works
Inherent bias in large language models: a random sampling analysis.
Ayoub, N. F., Balakrishnan, K., Ayoub, M. S., Barrett, T. F., David, A. P., & Gray, S. T. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.03.003 · Cited by 47 · see citing works
Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis
Rojas-Carabali, W. A., Cifuentes‐González, C., Wei, X., Putera, I., Sen, A. C., Thng, Z. X., Agrawal, R., Elze, T., Sobrin, L., Kempen, J. H., et al. (2023) · Ocular Immunology and Inflammation · doi:10.1080/09273948.2023.2253471 · Cited by 46 · see citing works
Long, C., Subburam, D., Lowe, K., dos Santos, A., Zhang, J. J., Hwang, S. W., Saduka, N., Horev, Y., Su, T., Côté, D. W. J., et al. (2024) · Otolaryngology · doi:10.1002/ohn.864 · Cited by 46 · see citing works
BPPV Information on Google Versus AI (ChatGPT)
Bellinger, J. R., Chapa, J. S. D. L., Kwak, M. W., Ramos, G. A., Morrison, D. R., & Kesser, B. W. (2023) · Otolaryngology · doi:10.1002/ohn.506 · Cited by 45 · see citing works
Yun, J. Y., Kim, D. J., Lee, N., & Kim, E. K. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105219 · Cited by 45 · see citing works
Appropriateness of ChatGPT in Answering Heart Failure Related Questions
King, R. C., Samaan, J. S., Yeo, Y. H., Mody, B. P., Lombardo, D. M., & Ghashghaei, R. (2024) · Heart Lung and Circulation · doi:10.1016/j.hlc.2024.03.005 · Cited by 45 · see citing works
A descriptive study based on the comparison of ChatGPT and evidence-based neurosurgeons
Liu, J., Zheng, J., Cai, X., Wu, D., & Yin, C. (2023) · iScience · doi:10.1016/j.isci.2023.107590 · Cited by 44 · see citing works
Levartovsky, A., Ben‐Horin, S., Kopylov, U., Klang, E., & Barash, Y. (2023) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002483 · Cited by 44 · see citing works
The potential of chatbots in chronic venous disease patient management
Athavale, A., Baier, J., Ross, E., & Fukaya, E. (2023) · JVS-Vascular Insights · doi:10.1016/j.jvsvi.2023.100019 · Cited by 43 · see citing works
Comparative Evaluation of Diagnostic Accuracy Between Google Bard and Physicians
Hirosawa, T., Mizuta, K., Harada, Y., & Shimizu, T. (2023) · The American Journal of Medicine · doi:10.1016/j.amjmed.2023.08.003 · Cited by 43 · see citing works
Quality of ChatGPT Responses to Questions Related To Liver Transplantation
Endo, Y., Sasaki, K., Moazzam, Z., Lima, H. A., Schenk, A. D., Limkemann, A. J. P., Washburn, W. K., & Pawlik, T. M. (2023) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-023-05714-9 · Cited by 42 · see citing works
Rizwan, A., & Sadiq, T. (2023) · Cureus · doi:10.7759/cureus.43106 · Cited by 39 · see citing works
Luykx, J. J., Gerritse, F. L., Habets, P. C., & Vinkers, C. H. (2023) · World Psychiatry · doi:10.1002/wps.21145 · Cited by 38 · see citing works
Rosen, S. R., & Saban, M. (2023) · European Radiology · doi:10.1007/s00330-023-10230-0 · Cited by 38 · see citing works
Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms
Nguyen, D., Swanson, D., Newbury, A., & Kim, Y. H. (2024) · Academic Radiology · doi:10.1016/j.acra.2023.11.002 · Cited by 38 · see citing works
Kataoka, Y., Takemura, T., Sasajima, M., & Katoh, N. (2021) · JMIR Cancer · doi:10.2196/26911 · Cited by 37 · see citing works
Mishra, A., Begley, S. L., Chen, A., Rob, M., Pelcher, I., Ward, M., & Schulder, M. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002598 · Cited by 37 · see citing works
ChatGPT’s ability to comprehend and answer cirrhosis related questions in Arabic
Samaan, J. S., Yeo, Y. H., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Yang, J. D., Liran, O., Spiegel, B. M. R., Kuo, A., et al. (2023) · Arab Journal of Gastroenterology · doi:10.1016/j.ajg.2023.08.001 · Cited by 37 · see citing works
ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis.
Ahmed, W., Saturno, M., Rajjoub, R., Duey, A. H., Zaidat, B., Hoang, T., Restrepo Mejia, M., Gallate, Z. S., Shrestha, N., Tang, J., et al. (2024) · European Spine Journal · doi:10.1007/s00586-024-08198-6 · Cited by 37 · see citing works
Li, W., Chen, J., Chen, F., Liang, J., & Yu, H. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03660-0 · Cited by 36 · see citing works
ChatGPT Surpasses 1000 Publications on PubMed: Envisioning the Road Ahead
Temsah, M.-H. A., Altamimi, I., Jamal, A. A., Alhasan, K. A., & Al-Eyadhy, A. A. (2023) · Cureus · doi:10.7759/cureus.44769 · Cited by 33 · see citing works
Liu, H. Y., Bonetti, M. A., Jeong, T., Pandya, S., Nguyen, V. T., & Egro, F. M. (2023) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2023.07.039 · Cited by 32 · see citing works
Galido, P. V., Butala, S., Chakerian, M., & Agustines, D. (2023) · Cureus · doi:10.7759/cureus.38166 · Cited by 29 · see citing works
Altamimi, I., Altamimi, A., Alhumimidi, A. S., Altamimi, A., & Temsah, M.-H. A. (2023) · Cureus · doi:10.7759/cureus.40351 · Cited by 29 · see citing works
Artificial Intelligence-Based ChatGPT Responses for Patient Questions on Optic Disc Drusen
Potapenko, I. O., Malmqvist, L., Subhi, Y., & Hamann, S. (2023) · Ophthalmology and Therapy · doi:10.1007/s40123-023-00800-2 · Cited by 29 · see citing works
Stevenson, E., Walsh, C., & Hibberd, L. (2023) · Annals of Clinical Biochemistry International Journal of Laboratory Medicine · doi:10.1177/00045632231203473 · Cited by 28 · see citing works
Huo, B., Calabrese, E., Sylla, P., Kumar, S., Ignacio, R. C., Oviedo, R., Hassan, I., Slater, B. J., Kaiser, A., Walsh, D. S., et al. (2024) · Surgical Endoscopy · doi:10.1007/s00464-024-10807-w · Cited by 28 · see citing works
Self-citationChatGPT in Occupational Medicine: A Comparative Study with Human Experts
Padovan, M., Cosci, B., Petillo, A., Nerli, G., Porciatti, F., Scarinci, S., Carlucci, F., Dell’Amico, L., Meliani, N., Necciari, G., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11010057 · Cited by 26 · see citing works
Mind + Machine: ChatGPT as a Basic Clinical Decisions Support Tool
Ayoub, M., Ballout, A. A., Zayek, R. A., & Ayoub, N. F. (2023) · Cureus · doi:10.7759/cureus.43690 · Cited by 25 · see citing works
Janopaul-Naylor, J. R., Koo, A., Qian, D. C., McCall, N. S., Liu, Y., & Patel, S. A. (2023) · American Journal of Clinical Oncology · doi:10.1097/coc.0000000000001050 · Cited by 25 · see citing works
Regulating the safety of health–related artificial intelligence.
Silva, M., Flood, C., Goldenberg, A., & Singh, D. (2022) · Healthcare Policy | Politiques de Santé · doi:10.12927/hcpol.2022.26824 · Cited by 23 · see citing works
Training ChatGPT Models in Assisting Urologists in Daily Practice
Manolitsis, I. S., Feretzakis, G., Tzelves, L. I., Kalles, D., Katsimperis, S. N., Angelopoulos, P., Αναστασίου, Α., Koutsouris, D., Kosmidis, T., Verykios, V. S., et al. (2023) · Studies in health technology and informatics · doi:10.3233/shti230562 · Cited by 22 · see citing works
Benirschke, R. C., Wodskow, J., Prasai, K., Freeman, A., Lee, J. M., & Groth, J. (2023) · American Journal of Clinical Pathology · doi:10.1093/ajcp/aqad106 · Cited by 20 · see citing works
“Vertigo, likely peripheral”: the dizzying rise of ChatGPT
Chee, J., Kwa, E. D., & Goh, X. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08135-1 · Cited by 18 · see citing works
Zhou, Y., Moon, C. N., Szatkowski, J., Moore, D., & Stevens, J. M. (2023) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-023-03742-4 · Cited by 18 · see citing works
From Understanding to Utilization: A Survey on Explainability for Large Language Models
Luo, H., & Specia, L. (2024) · arXiv · doi:10.48550/arxiv.2401.12874 · Cited by 18 · see citing works
Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?
Chowdhury, M., Lim, E., Higham, A., McKinnon, R., Ventoura, N., He, Y., & De Pennington, N. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.17 · Cited by 17 · see citing works
O’Hagan, R., Kim, R. H., Abittan, B. J., Caldas, S. A., Ungar, J. P., & Ungar, B. (2023) · Dermatology · doi:10.1159/000534005 · Cited by 17 · see citing works
Ali, H., Patel, P., Obaitan, I., Mohan, B. P., Sohail, A. H., Smith-Martinez, L., Lambert, K., Gangwani, M. K., Easler, J. J., & Adler, D. G. (2023) · iGIE · doi:10.1016/j.igie.2023.10.001 · Cited by 15 · see citing works
Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference
Wang, C., Liu, S. X., & Awadallah, A. H. (2023) · arXiv · doi:10.48550/arxiv.2303.04673 · Cited by 13 · see citing works
Evaluating ChatGPT's Accuracy in Lung Cancer Prevention and Screening Recommendations
Haver, H. L., Lin, C. T., Sirajuddin, A., Yi, P. H., & Jeudy, J. J. (2023) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.230115 · Cited by 13 · see citing works
AI's deep dive into complex pediatric inguinal hernia issues: a challenge to traditional guidelines?
Wang, G., Liu, Q., Chen, G., Xia, B., Zeng, D., Chen, G., & Guo, C. (2023) · Hernia · doi:10.1007/s10029-023-02900-1 · Cited by 13 · see citing works
Gracias, D., Siu, A. H. Y., Seth, I., Dooreemeah, D., & Lee, A. (2023) · ANZ Journal of Surgery · doi:10.1111/ans.18736 · Cited by 13 · see citing works
Can ChatGPT outperform a neurosurgical trainee? a prospective comparative study.
Williams, S. C., Starup-Hansen, J., Funnell, J. P., Hanrahan, J. G., Valetopoulou, A., Singh, N., Sinha, S., Muirhead, W. R., & Marcus, H. J. (2025) · British Journal of Neurosurgery · doi:10.1080/02688697.2024.2308222 · Cited by 13 · see citing works
Li, J., Gao, X., Dou, T., Gao, Y., Xu, L., & Zhu, W. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082344 · Cited by 13 · see citing works
(2017) · PubMed Central · Cited by 12 · see citing works
Utility of GPT-4 as an Informational Patient Resource in Otolaryngology
Suresh, K., Rathi, V., Nwosu, O., Partain, M. P., Glicksman, J. T., Jowett, N., & Crowson, M. G. (2023) · doi:10.1101/2023.05.14.23289944 · Cited by 11 · see citing works
Gwak, G., Hwang, U., Jung, S. H., & Kim, J. H. (2023) · Journal of Musculoskeletal Science and Technology · doi:10.29273/jmst.2023.7.1.8 · Cited by 11 · see citing works
Assessing the Soft Tissue Infection Expertise of ChatGPT and Bard Compared to IDSA Recommendations
Bonetti, M. A., Giorgino, R., Naegeli, M., Liu, H. Y., & Egro, F. M. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03372-1 · Cited by 10 · see citing works
Leveraging ChatGPT to aid patient education on coronary angiogram
Koh, S. J. Q., Yeo, K. K., & Yap, J. J. L. (2023) · Annals of the Academy of Medicine Singapore · doi:10.47102/annals-acadmedsg.2023138 · Cited by 9 · see citing works
Yıldız, M. S., & Alper, A. (2023) · Journal of Consumer Health on the Internet · doi:10.1080/15398285.2023.2240652 · Cited by 9 · see citing works
Can ChatGPT assist with the initial triage? A case study of stroke in young females
Rosen, S., & Saban, M. (2023) · International Emergency Nursing · doi:10.1016/j.ienj.2023.101340 · Cited by 7 · see citing works
Assessing the performance of GPT-4 in the filed of osteoarthritis and orthopaedic case consultation
Li, J., Gao, X., Dou, T., Gao, Y., & Zhu, W. (2023) · doi:10.1101/2023.08.06.23293735 · Cited by 4 · see citing works
Relevance of medical information obtained from ChatGPT: Are large language models friends or foes?
Mesnier, J., Suc, G., Sayah, N., Abtan, J., & Steg, P. G. (2023) · Archives of cardiovascular diseases · doi:10.1016/j.acvd.2023.07.009 · Cited by 4 · see citing works
Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting
Wang, R., Wang, H., Mi, F., Xue, B., Chen, Y., Wong, K.-F., & Xu, R. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.299 · Cited by 4 · see citing works
Can ChatGPT Provide Quality Information on Integrative Oncology? A Brief Report
Lam, C. S., Hua, R., Koon, H. K., Zhou, K. R., Lam, T. N., Lee, C. P., Lin, W. L., Wong, C. L., Lau, Y. M., Loong, H. H.-F., et al. (2023) · Journal of Integrative and Complementary Medicine · doi:10.1089/jicm.2023.0290 · Cited by 3 · see citing works
Graph coverings and twisted operators
Cimasoni, D., & Kassel, A. (2023) · Algebraic Combinatorics · doi:10.5802/alco.258 · Cited by 2 · see citing works
Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting
Wang, R., Wang, H., Mi, F., Chen, Y., Xue, B., Wong, K.-F., & Xu, R. (2023) · arXiv · doi:10.48550/arxiv.2305.13733 · Cited by 2 · see citing works
Graph coverings and twisted operators
Cimasoni, D., & Kassel, A. (2020) · arXiv · doi:10.48550/arxiv.2012.12575
Flagged first, then by citations
Research works printed but not linked: 2
- Wang? Y, Visweswaran? S, Kappor? S, Kooragayalu? S, Wu? X. ChatGPT, enhanced with clinical practice guidelines, is a superior decision support tool. [published online August 13, 2023].? MedRxiv.
- Zhou? J, He? X, Sun? L, . SkinGPT-4: an interactive dermatology diagnostic system with visual large language model. Published online April 20, 2023.MedRxiv.
Cited by
Sorted by citations
Large Language Models in Healthcare and Medical Applications: A Review
Maity, S., & Saikia, M. J. (2025) · Bioengineering · doi:10.3390/bioengineering12060631 · Cited by 249 · see citing works
Generative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119 · see citing works
Self-citationChow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84 · see citing works
DeepSeek’s “Low-Cost” Adoption Across China’s Hospital Systems
Zeng, D., Qin, Y., Sheng, B., & Wong, T. Y. (2025) · JAMA · doi:10.1001/jama.2025.6571 · Cited by 73 · see citing works
Collaborative, T. C. (2025) · BMJ Medicine · doi:10.1136/bmjmed-2025-001632 · Cited by 63 · see citing works
High-performance automated abstract screening with large language model ensembles
Sanghera, R., Thirunavukarasu, A. J., Khoury, M. E., O’Logbon, J., Chen, Y., Watt, A., Mahmood, M., Butt, H., Nishimura, G., & Soltan, A. A. S. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf050 · Cited by 62 · see citing works
Self-citationReporting Guideline for Chatbot Health Advice Studies
Collaborative, T. C., Huo, B., Collins, P. G. S., Chartash, D., Thirunavukarasu, A. J., Flanagin, A., Iorio, A., Cacciamani, G. E., Chen, X., Liu, N., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.30220 · Cited by 60 · see citing works
Self-citationCollaborative, T. C. (2025) · BMJ · doi:10.1136/bmj-2024-083305 · Cited by 50 · see citing works
ESMO guidance on the use of Large Language Models in Clinical Practice (ELCAP)
Wong, E. Y. T., Verlingue, L., Aldea, M., Franzoi, M. A., Umeton, R., Halabi, S., Harbeck, N., Indini, A., Prelaj, A., Romano, E., et al. (2025) · Annals of Oncology · doi:10.1016/j.annonc.2025.09.001 · Cited by 44 · see citing works
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 45 · see citing works
Large language models provide unsafe answers to patient-posed medical questions
Draelos, R. L., Afreen, S., Blasko, B., Brazile, T. L., Chase, N., Desai, D. P., Evert, J., Gardner, H. L., Herrmann, L., House, A. V., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02428-5 · Cited by 42 · see citing works
Tiller, N. B., Marcon, A. R., Zenone, M. A., Kidd, K. E., Jeukendrup, A. E., Master, Z., & Caulfield, T. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-112695 · Cited by 43 · see citing works
Park, S. H., Suh, C. H., Lee, J. H., Tejani, A. S., You, S. C., Kahn, C. E., & Moy, L. P. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.1522 · Cited by 38 · see citing works
Personalizing prostate cancer education for patients using an EHR-Integrated LLM agent
Hao, Y., Holmes, J., Waddle, M. R., Davis, B. J., Yu, N. Y., Vickers, K. S., Preston, H. R., Margolin, D., Löckenhoff, C. E., Vashistha, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02166-0 · Cited by 29 · see citing works
Public use of a generalist LLM chatbot for health queries
Costa-Gomes, B., Tolmachev, P., Taysom, E., Sounderajah, V., Richardson, H. B., Schoenegger, P., Liu, X., Nour, M. M., Spielman, S., Way, S. F., et al. (2026) · Nature Health · doi:10.1038/s44360-026-00117-x · Cited by 25 · see citing works
Wei, J., Wang, X., Huang, M., Xu, Y., & Yang, W. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02227-7 · Cited by 23 · see citing works
Kong, M., Fok, E. H. W., & Yiu, C. K. Y. (2025) · International Dental Journal · doi:10.1016/j.identj.2025.103854 · Cited by 23 · see citing works
Can artificial intelligence improve the readability of patient education information in gynecology?
Daram, N. R., Maxwell, R. A., D'Amato, J., & Massengill, J. C. (2025) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2025.06.047 · Cited by 22 · see citing works
Zhang, Y., & Zhang, H. (2025) · Journal of Technology Informatics and Engineering · doi:10.51903/jtie.v4i2.547 · Cited by 22 · see citing works
POC Sensor Systems and Artificial Intelligence—Where We Are Now and Where We Are Going?
Prashanthi, K., Kovur, K. M., Rayat, D. Y., & Wishart, D. S. (2025) · Biosensors · doi:10.3390/bios15090589 · Cited by 21 · see citing works
Ohu, F. C., Burrell, D. N., & Jones, L. A. (2025) · Healthcare · doi:10.3390/healthcare13212721 · Cited by 18 · see citing works
Huo, B., Collins, P. G. S., Cacciamani, G. E., & Guyatt, G. H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02113-z · Cited by 18 · see citing works
Self-citationCollaborative, T. C. (2025) · The Annals of Family Medicine · doi:10.1370/afm.250386 · Cited by 18 · see citing works
Awasthi, R., Bhattad, A., Ramachandran, S. P., Mishra, S. V., Khanna, A. K., Cywiński, J. B., Maheshwari, K. K., Mahapatra, D., DiRosa, I., Cohen, A., et al. (2025) · npj Health Systems · doi:10.1038/s44401-025-00043-2 · Cited by 17 · see citing works
RAG-Enhanced Open SLMs for Hypertension Management Chatbots
Aguzzi, G., Magnini, M., Farahmand, A., Ferretti, S., Pengo, M. F., & Montagna, S. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02297-7 · Cited by 16 · see citing works
van Kessel, R., Anderson, M., McMillan, B., Matthews, M. R., Rust, P., Pearcy, P., Nasir, K., & Mossialos, E. A. (2026) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2025-101959 · Cited by 16 · see citing works
Abdalla, M., Saad, M., Abazia, D. T., Advani, A., Alhammad, A., Bernknopf, A. C., Bitra, V. R., Blommel, M. L., Conn, K. M., Goldwire, M. A., et al. (2025) · JACCP JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · doi:10.1002/jac5.70131 · Cited by 14 · see citing works
Kim, Y., Shin, S., & Yoo, S.-H. (2025) · BMC Medical Ethics · doi:10.1186/s12910-025-01316-z · Cited by 14 · see citing works
ChatMyopia: An AI agent for myopia-related consultation in primary eye care settings
Wu, Y. Y., Chen, X., Zhang, W., Liu, S., Sum, W. M. R., Wu, X., Shang, X., Kee, C., He, M., & Shi, D. (2025) · iScience · doi:10.1016/j.isci.2025.113768 · Cited by 13 · see citing works
Ralla, B., Biernath, N., Lichy, I. M., Kurz, L., Friedersdorff, F., Schlomm, T., Schmidt, J., Plage, H., & Jeutner, J. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14165697 · Cited by 12 · see citing works
Effective prompt design for large language models in clinical practice
Callens, S. F. J. (2026) · Acta Clinica Belgica · doi:10.1080/17843286.2026.2613903 · Cited by 12 · see citing works
Kottlors, J., Iuga, A.-I., Blüthgen, C., Bressem, K. K., Kather, J. N., Moy, L. P., Wald, C., Wang, W., Liu, T., Ranschaert, E. R., et al. (2026) · Radiology · doi:10.1148/radiol.250913 · Cited by 12 · see citing works
Reporting guideline for chatbot health advice studies: The CHART statement
Huo, B., Collins, P. G. S., Chartash, D., Thirunavukarasu, A. J., Flanagin, A., Iorio, A., Cacciamani, G. E., Chen, X., Liu, N., Mathur, P., et al. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103222 · Cited by 11 · see citing works
Self-citationYou, C., Ghosh, R., Vilaro, M. J., Venkatakrishnan, R., Venkatakrishnan, R., Maxim, A., Peng, X., Tamboli, D. N. A., & Lok, B. C. L. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1655860 · Cited by 10 · see citing works
Akgün, F. E., & Akgün, M. (2026) · Healthcare · doi:10.3390/healthcare14081098 · Cited by 10 · see citing works
Critical Appraisal Tools for Evaluating Artificial Intelligence in Clinical Studies: Scoping Review
Cabello, J. B., García, V. R., Torralba, M., Fernandez, M. M., Ubeda, M., Ansuategi, E., Ramos-Ruperto, L., Emparanza, J. I., Urreta, I., Iglesias, M. T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/77110 · Cited by 9 · see citing works
Clinical pathway-aware large language models for reliable and transparent medical dialogue
Wu, J., Wu, X., Zheng, Y., & Yang, J. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104942 · Cited by 9 · see citing works
Goh, H. L., Sancenón, V., Chu, B. M. X., Koh, G. C. H., Koh, L., Teo, D., Ooi, M. S. L., Thng, C. N., Tan, C.-Z., Chua, D. W. L., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02514-8 · Cited by 9 · see citing works
Chow, J. C. L., & Li, K. (2026) · Information · doi:10.3390/info17040361 · Cited by 9 · see citing works
Large language model-generated clinical practice guideline for appendicitis
Boyle, A., Huo, B., Sylla, P. A., Calabrese, E., Kumar, S. S., Slater, B. J., Walsh, D. S., & Vosburg, R. W. (2025) · Surgical Endoscopy · doi:10.1007/s00464-025-11723-3 · Cited by 8 · see citing works
Self-citationPerformance of Foundation Models vs Physicians in Textual and Multimodal Ophthalmological Questions
Rocha, H., Chong, Y. J., Thirunavukarasu, A. J., Wong, Y. L., Wong, S. W., Chang, Y., Azzopardi, M., Tan, B. K. J., Song, A., Malem, A. H., et al. (2025) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2025.4255 · Cited by 8 · see citing works
Self-citationReis, F., Agha-Mir-Salim, L., Hickstein, R., Reis, M., Piper, S. K., Balzer, F., & Boie, S. D. (2026) · Journal of Medical Internet Research · doi:10.2196/84668 · Cited by 8 · see citing works
Hassan, A. M., Coert, J. H., Clemens, M. W., Hassanein, A. H., Waljee, J. F., Nelson, J. A., Mehrara, B. J., & Selber, J. C. (2026) · Journal of the American College of Surgeons · doi:10.1097/xcs.0000000000001834 · Cited by 8 · see citing works
Hamamoto, R., Koyama, T., Takahashi, S., Yasuda, T., Kobayashi, K., Akagi, Y., Kouno, N., Sudo, K., Hirata, M., Sunami, K., et al. (2026) · Journal of Hematology & Oncology · doi:10.1186/s13045-026-01781-y · Cited by 8 · see citing works
AI for Personalized Learning in Medical Education
Niazai, H., & Monib, W. K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-1519-5.ch004 · Cited by 7 · see citing works
Artificial intelligence in emergency medicine: a narrative review
Rego, A., Arango-Ibañez, J. P., Taylor, R. A., Smith, M. E., Jones, D. D., Pelletier, J., Colletti, J. E., Gottlieb, M. A., & Long, B. (2026) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2026.01.028 · Cited by 7 · see citing works
Hu, M., Wang, Z., Zhang, Z., & Muwei, L. I. (2026) · Digital Health · doi:10.1177/20552076261464724 · Cited by 7 · see citing works
Angyal, V., Bertalan, Á., Domján, P., & Dinya, E. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03088-3 · Cited by 6 · see citing works
Collaborative, T. C., Huo, B., Collins, P. G. S., Chartash, D., Thirunavukarasu, A. J., Flanagin, A., Iorio, A., Cacciamani, G. E., Chen, X., Liu, N., et al. (2025) · British journal of surgery · doi:10.1093/bjs/znaf142 · Cited by 6 · see citing works
Self-citationBurla, L., Metzler, J. M., Kalaitzopoulos, D. R., Kamm, S., Ormos, M., Passweg, D. E., Schraag, S. M., Samartzis, E. P., Samartzis, N., Witzel, I., et al. (2025) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2025.114625 · Cited by 6 · see citing works
Li, W., & Li, Q. (2025) · Digital Health · doi:10.1177/20552076251393385 · Cited by 6 · see citing works
Al-Haddad, A., Alrabadi, M., Sa'deh, O., Alrabadi, G., & Hassona, Y. M. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1710159 · Cited by 7 · see citing works
Artificial intelligence in medicine: a position paper by the Italian Society of Internal Medicine
Balsano, C., Cabitza, F., Cicco, S., Gori, M., Malerba, D., Montagna, M., Tarquini, R., Vacca, A., & The Working Group on Artificial Intelligence, D. T. O. T. I. S. O. I. M. (. (2025) · Internal and Emergency Medicine · doi:10.1007/s11739-025-04146-4 · Cited by 6 · see citing works
Gong, Y., Wang, M.-K., Tu, Y.-F., Huang, C.-Q., & Zhang, D. (2026) · Entertainment Computing · doi:10.1016/j.entcom.2026.101082 · Cited by 6 · see citing works
ChatGPT-5 versus other mainstream large language models in core diabetic retinopathy patient queries
Cang, X., MengXia, N., Song, C., Zhao, J., Guo, Y., Zou, Y., Zhang, Z., & Jiang, L. (2026) · Frontiers in Cell and Developmental Biology · doi:10.3389/fcell.2026.1754221 · Cited by 6 · see citing works
Hazhir, S., Langarizadeh, M., Bahariniya, S., Laktarashi, H. V., Hami, A., & Aghda, S. A. F. (2025) · BMC Public Health · doi:10.1186/s12889-025-25814-2 · Cited by 5 · see citing works
Artificial intelligence and the future of evidence-based medicine
Car, L. T., Wong, T. Y., Majeed, A., & Collins, P. G. S. (2026) · Nature Health · doi:10.1038/s44360-025-00002-z · Cited by 5 · see citing works
Coronado-Vázquez, V., Allande‐Cussó, R., Caparrós-González, R. A., & Gómez-Salgado, J. C. (2026) · Atención Primaria · doi:10.1016/j.aprim.2026.103460 · Cited by 5 · see citing works
Pan, J., Jian, B., Hager, P., Zhang, Y., Liu, C., Jungmann, F., Li, H., Canisius, J., You, C., Wu, J., et al. (2026) · Nature Health · doi:10.1038/s44360-026-00152-8 · Cited by 5 · see citing works
Yu, Y. Q., Li, L., & Li, Y. (2025) · medRxiv · doi:10.1101/2025.10.17.25338266 · Cited by 5 · see citing works
Liang, J., Xing, M., Xiang, P., Wang, G., Chen, M., Fang, Q., Zhou, T., Lu, Z., Leng, X., Huang, J., et al. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02537-1 · Cited by 4 · see citing works
The top 100 most-cited articles on large language models in medicine: A bibliometric analysis
Li, Z., Xu, R., Gong, X.-R., Wang, C. L., & Liu, J. (2025) · Digital Health · doi:10.1177/20552076251365059 · Cited by 4 · see citing works
Grant, J. E. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00527-y · Cited by 4 · see citing works
Evaluation of DeepSeek-R1 for Ophthalmic Diagnosis and Reasoning: A Comparison with OpenAI o1 and o3
Ming, S., Yao, X., Guo, Q., Chen, D., Guo, X., Xie, K., & Lei, B. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02264-2 · Cited by 4 · see citing works
Ma, Z., Chen, R., Wang, A., Xi, Y., Liang, M., Yuan, S., Fan, N., Zang, J., Wang, T., & Zang, L. (2026) · Journal of Medical Internet Research · doi:10.2196/92315 · Cited by 4 · see citing works
Thirunavukarasu, A. J., Lim, E., & Huo, B. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100910 · Cited by 3 · see citing works
Self-citationMartin, G. L., & Létinier, L. (2025) · Therapies · doi:10.1016/j.therap.2025.10.001 · Cited by 3 · see citing works
Attanasio, F., Fazio, V., Antonini, C., Lanzano, N., Obumselu, G., Prato, M., Flutti, E., Pacchioni, F., Fregna, L., Franchini, L. A. M., et al. (2025) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2025.11.039 · Cited by 3 · see citing works
Christiansen, I. B. J., Zoch-Lesniak, B., Lindberg, B. H., Björkman, A., Ramerman, L., Payne, R. E., Herrmann, T., Homar, V., & Huibers, L. (2026) · Scandinavian Journal of Primary Health Care · doi:10.1080/02813432.2025.2600632 · Cited by 3 · see citing works
Khosravi, M., Yousefi-Roobiyat, M., Asghari, Z., Nakhaei, M., & Khosravi, F. (2026) · Scientific Reports · doi:10.1038/s41598-026-47666-5 · Cited by 3 · see citing works
Chatbot psychosis: moving beyond recognition to mechanistic understanding and harm reduction
Kumari, V., & Otermans, P. C. J. (2026) · The British Journal of Psychiatry · doi:10.1192/bjp.2026.10541 · Cited by 3 · see citing works
CARE CONNECT: Revolutionizing Healthcare through AI
Babu, C. M., Bhaskar, N., Bhavika, B., Shivateja, A., Annabeemoju, V., Chowdari, S., & Manisharan, C. (2025) · International Conference on Circuit, Power and Computing Technologies · doi:10.1109/iccpct65132.2025.11176716 · Cited by 2 · see citing works
Wang, G., Zhang, Y., Wang, W., Zhu, Y., Lu, W., Wang, C., Bi, H., & Yang, X. (2025) · JMIR AI · doi:10.2196/76372 · Cited by 2 · see citing works
Jiang, W., Wang, D., Zeng, Y., Huang, J., Xu, C., & Liu, C. (2025) · Journal of Medical Internet Research · doi:10.2196/80770 · Cited by 2 · see citing works
Xie, X. Y., Zhang, M. J., Cheung, Z. J. K., Luk, T. T., Wang, M. P., Lam, T. H., & Cheung, Y. T. D. (2025) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntaf267 · Cited by 2 · see citing works
Medi Portal AI: An Intelligent Hospital and Patient Portal System Enhancing HealthCare Management
Lakshmi, M. S., Reddy, B. J., Yadav, D. A., Dinesh, G., & Ramavamshi, A. (2025) · International Conference on Sustainable Communication Networks and Application (ICSCNA) · doi:10.1109/icscn67106.2025.11308346 · Cited by 2 · see citing works
Building a code of conduct for AI-driven clinical consultations
Lim, E., Thirunavukarasu, A. J., He, Y. V., Monkhouse, H. E., Fu, D. J., de Pennington, N., Higham, A., Tham, Y. C., Jia, Y., & Habli, I. (2026) · Nature Medicine · doi:10.1038/s41591-025-04068-w · Cited by 2 · see citing works
Self-citationCarou‐Senra, P., Rodríguez‐Pombo, L., Alvarez‐Lorenzo, C., & Goyanes, Á. (2026) · International Journal of Pharmaceutics · doi:10.1016/j.ijpharm.2026.126587 · Cited by 2 · see citing works
Digital patient decision aids for endometriosis management: a scoping review protocol
Pittet, O., Delvallée, M., Pluchino, N., Selby, K., Elwyn, G. J., & Durand, M. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-109888 · Cited by 2 · see citing works
Richter, S., Buszello, C. H., Prem, M. G., Willkommen, S., Hasani, E., Uckermann, O., Sandi-Gahun, S., Juratli, T. A., Eyüpoglu, I. Y., & Polanski, W. H. (2026) · Journal of Medical Internet Research · doi:10.2196/77307 · Cited by 2 · see citing works
Villarreal‐Zegarra, D., Paredes-Gonzales, Y., Dámaso-Román, A., Quiñones-Inga, J., Centeno-Terrazas, G., & Lozada, Y. P. A.-M. (2026) · PLoS ONE · doi:10.1371/journal.pone.0344939 · Cited by 2 · see citing works
Large language models for primary care ophthalmic education: a systematic review
Li, S., Wang, X., Chen, Y., Tian, M., Lin, P., Lai, M., & Jiang, L. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1810098 · Cited by 2 · see citing works
Examining user-AI interaction patterns in health-Information queries
Wang, J. T., Chung, H. W., Do, G. N., & Yang, A. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106453 · Cited by 2 · see citing works
Martini, S., Schluessel, S., Aghamaliyev, U., Rippl, M., Deißler, L., Tausendfreund, O., Nuebler, D., Mueller, K., Schmidmaier, R., & Drey, M. (2026) · JMIR AI · doi:10.2196/91369 · Cited by 2 · see citing works
Han, G. I., Lee, H. J., & Son, Y.-J. (2026) · Digital Health · doi:10.1177/20552076261450385 · Cited by 2 · see citing works
Babun, H., & Oluş, F. (2026) · Frontiers in Health Services · doi:10.3389/frhs.2026.1830227 · Cited by 2 · see citing works
Çınar, B., Safran, E., & Ayyildiz-Eroglu, Z. (2026) · International Urogynecology Journal · doi:10.1007/s00192-026-06727-z · Cited by 2 · see citing works
Zhong, W., Zheng, G., Li, Q., Huang, Y., & Zhao, J. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1818821 · Cited by 2 · see citing works
Bilgiç, B., Turan, S., Avcıl, S., & Pekcanlar, A. A. (2026) · BMC Psychiatry · doi:10.1186/s12888-026-08280-x · Cited by 2 · see citing works
Ethical Considerations in Personal Health Large Language Models
Liu, J., & Liu, S. (2026) · Journal of Medical Internet Research · doi:10.2196/92240 · Cited by 2 · see citing works
Yost, C. D., Callas, B., Halligan, P., Palumbo, P., Rawal, S. C., Leyfman, Y., & Nguyen, R. H.-T. (2026) · Cancers · doi:10.3390/cancers18121981 · Cited by 2 · see citing works
Chang, J., Peng, R., Chen, X., Zhu, Y., Miao, R., Cao, Z., & Feng, H. (2026) · Journal of Medical Internet Research · doi:10.2196/84726 · Cited by 2 · see citing works
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 2 · see citing works
Çiftçi, M. U., Koluman, A., Çiftçi, E. A., Çakmur, B. B., & Ziroğlu, N. (2026) · The Knee · doi:10.1016/j.knee.2026.104579 · Cited by 2 · see citing works
Gong, Z., Na, Y., Guo, Y., Zhu, X., & Ma, X. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1924162 · Cited by 2 · see citing works
Zhang, X., & Wang, J. (2025) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2025.02.015 · Cited by 2 · see citing works
Does the use of generative AI chatbots by patients introduce risk of adverse drug events?
McIntyre, R. S. (2025) · Expert Opinion on Drug Safety · doi:10.1080/14740338.2025.2493351 · Cited by 2 · see citing works
From Information Overload to Patient-Centered Oncology Care Through Apomediation
Sarfraz, Z., Subbiah, V., & Ahluwalia, M. S. (2026) · JCO Oncology Practice · doi:10.1200/op-25-01299 · Cited by 1 · see citing works
AI in the Chair: A Multi‐Centre Study of Doctor AI Answering Orthodontic Patient Questions
Zhang, R., Liu, M., Liong, O., Liu, L., Guan, S., Chen, K., Li, Y., Han, X., & Mei, L. (2026) · Journal of the Royal Society of New Zealand · doi:10.1002/snz2.70035 · Cited by 1 · see citing works
Laypersons’ perceptions and misconceptions of AI-chatbots: a systematic review
Schneller, M., Seier, J., & Albert, D. (2026) · Universal Access in the Information Society · doi:10.1007/s10209-026-01323-w · Cited by 1 · see citing works
- Authors
- Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., Sanger, S., Ramji, K., & Guyatt, G.
- Affiliations
- McMaster University (CA)
- H. Ross University School of Medicine, Miramar, Florida
- University of Toronto (CA)
- Universidad Complutense de Madrid (ES)
- Papageorgiou General Hospital (GR)
- University of Oxford (GB)
- Corresponding author
- Huo, Bright
- ORCID
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2025-02-04
- Volume
- 8
- Issue
- 2
- Pages
- e2457879
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC11795331
- OpenAlex
- W4407135263
- Semantic Scholar
- 8e5b1084560f7b370363a2ad302c602e7a1ba78c
- Cited by
- 283
- References
- 184
- Citations by year
- 2026: 210 · 2025: 73
- FWCI
- 16.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, chatbots, health advice, reporting quality, generative AI, clinical integration, prompt engineering, patient safety, ethical and regulatory considerations, clinical accuracy
- MeSH terms
- Large Language Models, Generative Artificial Intelligence, Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
