The DOI matches its Crossref record; 189 of the 190 references it cites were found in a registry or an index; none of the 65 sources its claims cite is checked for support yet.
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Open accessAbstract
Importance: There is much interest in the clinical integration of large language models (LLMs) in health care. Many studies have assessed the ability of LLMs to provide health advice, but the quality of their reporting is uncertain. Objective: To perform a systematic review to examine the reporting variability among peer-reviewed studies evaluating the performance of generative artificial intelligence (AI)–driven chatbots for summarizing evidence and providing health advice to inform the development of the Chatbot Assessment Reporting Tool (CHART). Evidence Review: A search of MEDLINE via Ovid, Embase via Elsevier, and Web of Science from inception to October 27, 2023, was conducted with the help of a health sciences librarian to yield 7752 articles. Two reviewers screened articles by title and abstract followed by full-text review to identify primary studies evaluating the clinical accuracy of generative AI-driven chatbots in providing health advice (chatbot health advice studies). Two reviewers then performed data extraction for 137 eligible studies. Findings: A total of 137 studies were included. Studies examined topics in surgery (55 [40.1%]), medicine (51 [37.2%]), and primary care (13 [9.5%]). Many studies focused on treatment (91 [66.4%]), diagnosis (60 [43.8%]), or disease prevention (29 [21.2%]). Most studies (136 [99.3%]) evaluated inaccessible, closed-source LLMs and did not provide enough information to identify the version of the LLM under evaluation. All studies lacked a sufficient description of LLM characteristics, including temperature, token length, fine-tuning availability, layers, and other details. Most studies (136 [99.3%]) did not describe a prompt engineering phase in their study. The date of LLM querying was reported in 54 (39.4%) studies. Most studies (89 [65.0%]) used subjective means to define the successful performance of the chatbot, while less than one-third addressed the ethical, regulatory, and patient safety implications of the clinical integration of LLMs. Conclusions and Relevance: In this systematic review of 137 chatbot health advice studies, the reporting quality was heterogeneous and may inform the development of the CHART reporting standards. Ethical, regulatory, and patient safety considerations are crucial as interest grows in the clinical integration of LLMs.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This systematic review of 137 chatbot health advice studies found substantial heterogeneity and incomplete reporting of LLM characteristics, prompt engineering, query strategy, and performance evaluation. Most studies evaluated closed-source ChatGPT models and used investigator opinion rather than evidence-based reference standards, with less than one-third addressing ethics, patient safety, or regulation.
Key findings
- Of 137 CHAS, 135 (98.5%) assessed ChatGPT and 136 (99.3%) evaluated inaccessible/closed-source models; only 1 (0.7%) fully described the LLM version, and 36 (26.3%) described none of the LLM characteristics.
- Performance evaluation was often subjective: 89 studies (65.0%) used investigators’ opinion without a reference standard, while 21 (15.3%) used clinical practice guideline recommendations; 18 (13.1%) described a standardized evaluation process and 16 (11.7%) reported blinding.
- Reporting of methods was incomplete: 121 (88.3%) did not describe chatbot selection, 40 (29.2%) did not describe query strategy, 37 (27.0%) did not describe prompt source, 136 (99.3%) did not describe prompt testing/engineering, and 20 (14.6%) used standardized prompts.
- Ethical, safety, and regulatory implications were addressed by 45 (32.8%) for ethics, 44 (32.1%) for patient safety, and 22 (16.1%) for regulation of LLMs.
Methods
- Systematic review following PRISMA; protocol prospectively registered on Open Science Framework; searched MEDLINE via Ovid, Embase via Elsevier, and Web of Science on October 27, 2023.
- Two independent researchers screened titles/abstracts and full texts, with a third resolving disagreements; 7,752 articles reviewed yielded 137 eligible CHAS.
- Data extracted in Covidence; descriptive statistics and narrative synthesis were used, with no meta-analysis planned due anticipated heterogeneity; quality of evidence was not evaluated due lack of validated risk-of-bias tools for CHAS.
Limitations
- Stated: reporting in CHAS may change beyond review time due the dynamic nature of the field; much included literature used hypothetical patient cases; there is a paucity of literature detailing ethical and regulatory considerations; future patient-centered prospective research is needed.
- Stated: quality of evidence was not evaluated because no high-quality validated risk-of-bias tools were available for CHAS.
- Evident from the text: heterogeneity in study subjects and designs precluded meta-analysis; the review describes reporting practices and cannot establish causal effects of reporting on clinical outcomes.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
49 Own result · 35 Not checked yet · 7 No basis given · 5 Own method · 4 Own interpretation · 2 Sources not found
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.”
§ Large Language Models for Chatbot Health Advice Studies > Results > Performance Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.”
§ Large Language Models for Chatbot Health Advice Studies > Results > Performance Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Articles most commonly examined advice regarding treatment (91 [66.4%]), diagnosis (60 [43.8%]), and differential diagnosis (23 [16.8%]).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.”
§ Large Language Models for Chatbot Health Advice Studies > Results > Performance Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most (64.5%) studies used subjective means as the ground truth to define the successful performance of the LLM, while less than a third addressed the ethical, regulatory, and patient safety implications of clinically integrating LLMs.”
§ Large Language Models for Chatbot Health Advice Studies > Abstract > Findings
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, 16 studies (11.7%) explained why the specific LLMs were selected to be studied.”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most (64.5%) studies used subjective means as the ground truth to define the successful performance of the LLM, while less than a third addressed the ethical, regulatory, and patient safety implications of clinically integrating LLMs.”
§ Large Language Models for Chatbot Health Advice Studies > Abstract > Findings
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With respect to performance evaluation of chatbots, few studies described a standardized evaluation process (18 [13.1%]) or blinding (16 [11.7%]), and many studies did not describe a structured approach to performance evaluation (39 [28.5%]).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Performance Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Study performance was suboptimal in noting the number of chat windows (32 [23.4%]) and the number of queries (48 [35.0%]).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Performance Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Few studies noted the date (54 [39.4%]) and location (5 [3.6%]) of their query.”
§ Large Language Models for Chatbot Health Advice Studies > Results > Performance Evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 9
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“There were no ongoing studies or studies awaiting classification.”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, there is a paucity of literature that comments in detail on the ethical and regulatory considerations of the clinical integration of LLMs that impact patient safety.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“This systematic review adheres to the reporting standards of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guideline.19”
§ Large Language Models for Chatbot Health Advice Studies > Methods
What the sources say
- Not checked yetcites [19] Page MJ, McKenzie JE, Bossuyt PM,. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021; 372(71): n71. doi: 10.1136/bmj.n71 33782057 PMC8005924 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The quality of the evidence was not evaluated due to the lack of high-quality, validated risk-of-bias tools available for CHAS.”
§ Large Language Models for Chatbot Health Advice Studies > Methods > Statistical Analysis
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To capture all relevant studies, an academic health sciences librarian (S.S.) with expertise in systematic reviews aided the study team in designing a comprehensive search strategy.”
§ Large Language Models for Chatbot Health Advice Studies > Methods > Search Strategy and Selection Criteria
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“However, chatbots are typically not designed for medical application and currently lack regulation for their use in medicine.10”
§ Large Language Models for Chatbot Health Advice Studies > Introduction
What the sources say
- Not checked yetcites [10] Li J, Dada A, Puladi B, Kleesiek J, Egger J. ChatGPT in healthcare: a taxonomy and systematic review. Comput Methods Programs Biomed. 2024; 245: 108013. doi: 10.1016/j.cmpb.2024.108013 38262126 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Furthermore, generative AI-driven chatbots present unique risks of hallucinating (ie, confident but incorrect) false results and propagating misleading information and advice.11”
§ Large Language Models for Chatbot Health Advice Studies > Introduction
What the sources say
- Not checked yetcites [11] Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023; 6(1): 120. doi: 10.1038/s41746-023-00873-0 37414860 PMC10326069 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Our protocol was prospectively registered on Open Sciences Framework.20”
§ Large Language Models for Chatbot Health Advice Studies > Methods
What the sources say
- Not matchedcites [20] Registries OSF. Protocol for a scoping review of chatbot assessment studies: guidance for the CHART tool. February 25, 2024. Accessed December 9, 2024. https://osf.io/cxsk3
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“With a lack of standardized reporting, the assessment of the quality of CHAS and the interpretation of their findings remain challenging.17”
§ Large Language Models for Chatbot Health Advice Studies > Introduction
What the sources say
- Not checked yetcites [17] Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010; 7(2): e1000217. doi: 10.1371/journal.pmed.1000217 20169112 PMC2821895 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Patients have reported interest in seeking medical advice from chatbots,7”
§ Large Language Models for Chatbot Health Advice Studies > Introduction
What the sources say
- Not checked yetcites [7] Shahsavar Y, Choudhury A. User intentions to use ChatGPT for self-diagnosis and health-related purposes: cross-sectional survey study. JMIR Hum Factors. 2023; 10: e47564. doi: 10.2196/47564 37195756 PMC10233444 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In all cases, 2 independent researchers (W.T. and J.P.S.) conducted screening, with a third independent researcher casting a deciding vote to resolve cases of disagreement.”
§ Large Language Models for Chatbot Health Advice Studies > Methods > Search Strategy and Selection Criteria
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Additionally, the existing literature base emphasizes a structured approach to performance evaluation.165”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Reporting of LLM Characteristics and Study Method
What the sources say
- Not checked yetcites [165] Guyatt GH, Oxman AD, Kunz R,. GRADE guidelines: 2. framing the question and deciding on important outcomes. J Clin Epidemiol. 2011; 64(4): 395- 400. doi: 10.1016/j.jclinepi.2010.09.012 21194891 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As these publications are largely prepared by clinicians, study teams may not have access to expertise in machine learning.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Reporting of LLM Characteristics and Study Method
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“LLMs have in-context learning capabilities that allow them to adapt to the given prompt, so prompts can be intentionally designed to support performance on further tasks, such as patient communication, administrative tasks, risk assessments, or clinical decision-making.160,161”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Reporting of LLM Characteristics and Study Method
What the sources say
- Not checked yetcites [160] Meskó B. Prompt engineering as an important emerging skill for medical professionals: tutorial. J Med Internet Res. 2023; 25(1): e50638. doi: 10.2196/50638 37792434 PMC10585440 source
- Not checked yetcites [161] Nguyen D, Swanson D, Newbury A, Kim YH. Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms. Acad Radiol. 2024; 31(5): 1799- 1804. doi: 10.1016/j.acra.2023.11.002 38103973 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Artificial intelligence–linked medical devices are considered software as a medical device and are regulated under federal departments such as Health Canada and the US Food and Drug Administration.170”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [170] Da Silva M, Flood CM, Goldenberg A, Singh D. Regulating the safety of health–related artificial intelligence. Healthc Policy. 2022; 17(4): 63- 77. doi: 10.12927/hcpol.2022.26824 35686827 PMC9170055 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“While other regulated AI medical technologies, such as radiology image analysis177 or diagnosis,178 are trained on domain-specific medical data for targeted uses, LLMs have broader complexity, applicability, and an ability to adapt in real time, which complicates their regulation.2,179,180”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [2] Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023; 29(8): 1930- 1940. doi: 10.1038/s41591-023-02448-8 37460753 source
- Not checked yetcites [179] Wu E, Wu K, Daneshjou R, Ouyang D, Ho DE, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat Med. 2021; 27(4): 582- 584. doi: 10.1038/s41591-021-01312-x 33820998 source
- Not checked yetcites [180] Nagendran M, Chen Y, Lovejoy CA,. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies in medical imaging. BMJ. 2020; 368:m689. doi: 10.1136/bmj.m689 PMC7190037 32213531 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Many studies describe this as defining a ground truth or evaluation dataset, and it is an essential step in LLM performance evaluation.166,167”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Reporting of LLM Characteristics and Study Method
What the sources say
- Not checked yetcites [166] Chang Y, Wang X, Wang J,. A survey on evaluation of large language models. ACM Trans Intell Syst Technol. 2024; 15(3): 1- 45. doi: 10.1145/3641289 source
- Not checked yetcites [167] Liang JJ, Tsou CH, Devarakonda MV. Ground truth creation for complex clinical NLP tasks: an iterative vetting approach and lessons learned. AMIA Jt Summits Transl Sci Proc. 2017; 2017: 203- 212. 28815130 PMC5543376
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Just as sufficient detail about medical or surgical interventions must be reported in traditional studies to reproduce experiments and guide critical appraisal, LLM characteristics are similarly vital.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Reporting of LLM Characteristics and Study Method
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“By engineering a comprehensive search strategy, we aimed to capture an accurate representation of the literature.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Limitations
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“The World Health Organization released guidance on ethics and governance of AI for health,182 which overlaps substantially with LLMs, which are unimodal models.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [182] World Health Organization. Ethics and governance of artificial intelligence for health. Accessed November 4, 2024. https://www.who.int/publications/i/item/9789240029200
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First, the reporting used in CHAS may change beyond the time of this review due to the dynamic nature of this field.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Limitations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“While clinical practice is informed by evidence from peer-reviewed research, LLMs replicate their input data and thus may propagate online health misinformation and prejudices,189,190 compromising patient safety.”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [189] Meyrowitsch DW, Jensen AK, Sørensen JB, Varga TV. AI chatbots and (mis)information in public health: impact on vulnerable communities. Front Public Health. 2023; 11: 1226776. doi: 10.3389/fpubh.2023.1226776 38026315 PMC10644115
- Not checked yetcites [190] Khan B, Fatima H, Qureshi A,. Drawbacks of artificial intelligence and their potential solutions in the healthcare sector. Biomed Mater Devices. 2023; 1(2): 1- 8. doi: 10.1007/s44174-023-00063-2 36785697 PMC9908503
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Concerns have also been raised related to data privacy of patient health information among proprietary LLMs.182,191”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [182] World Health Organization. Ethics and governance of artificial intelligence for health. Accessed November 4, 2024. https://www.who.int/publications/i/item/9789240029200
- Not checked yetcites [191] Yao Y, Duan J, Xu K, Cai Y, Sun Z, Zhang Y. A survey on large language model (LLM) security and privacy: the good, the bad, and the ugly. High-Confidence Computing. 2024; 4(2). doi: 10.1016/j.hcc.2024.100211
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Taken together, these factors raise major issues as we move from the technological development of LLMs to their clinical validation for use in health care settings.182”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [182] World Health Organization. Ethics and governance of artificial intelligence for health. Accessed November 4, 2024. https://www.who.int/publications/i/item/9789240029200
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“Standardized frameworks are under development to establish detailed reporting standards to help authors design and report these studies.18”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Limitations
What the sources say
- Not checked yetcites [18] Huo B, Cacciamani GE, Collins GS, McKechnie T, Lee Y, Guyatt G. Reporting standards for the use of large language model-linked chatbots for health advice. Nat Med. 2023; 29(12): 2988. doi: 10.1038/s41591-023-02656-2 37957381 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“There is a potential for misuse of patient data, medicolegal implications, and loss of physician-patient trust.181,192”
§ Large Language Models for Chatbot Health Advice Studies > Discussion > Ethics, Patient Safety, and LLM Regulation
What the sources say
- Not checked yetcites [181] Ong JCL, Chang SYH, William W,. Ethical and regulatory challenges of large language models in medicine. Lancet Digit Health. 2024; 6(6): e428- e432. doi: 10.1016/S2589-7500(24)00061-X 38658283 source
- Not checked yetcites [192] Wang C, Liu S, Yang H, Guo J, Wu Y, Liu J. Ethical considerations of using ChatGPT in health care. J Med Internet Res. 2023; 25: e48009. doi: 10.2196/48009 37566454 PMC10457697 source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The generated output depends greatly on the initial input.
“However, the generated output depends greatly on the initial input.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work to date on patient safety of patient-facing applications has primarily been focused on particular diagnoses or specialties”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To the best of our knowledge, there have been no previous studies on patient safety of patient-facing applications for a wide range of primary care questions that patients are likely to ask.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Within this review, the authors noted that the majority of studies (68%) did not consider patient safety.”
From Large language models provide unsafe answers to patient-posed medical questions (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| No | 29 (21.7)21,22,28,35,36,37,42,51,55,60,76,78,79,80,82,88,93,108,110,113,128,130,138,140,143,150,151,152,154 |”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
Reported 21.7 · recomputed 21.167883
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Patient safety | 44 (32.8)”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
Reported 32.8 · recomputed 32.116788
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
“| Accessibility (open vs closed) | 4 (2.9)33,37,123,147 |”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
“| Not described | 121 (88.3)6,13,21,22,24,27,28,29,30,33,34,35”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
“| Yes | 3 (2.2)6,55,68,76,127 |”
§ Large Language Models for Chatbot Health Advice Studies > Results > Study Characteristics
“6,55,68,76,127”
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Self-citationRevolutionizing healthcare: the role of artificial intelligence in clinical practice
Alowais, S. A., Alghamdi, S. S., Alsuhebany, N., Alqahtani, T., Alshaya, A. I., Almohareb, S. N., Aldairem, A., Alrashed, M., Bin Saleh, K., Badreldin, H. A., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04698-z · Cited by 3,146
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
GRADE guidelines: 2. Framing the question and deciding on important outcomes
Guyatt, G. H., Oxman, A. D., Kunz, R., Atkins, D., Brozek, J., Vist, G., Alderson, P., Glasziou, P., Falck-Ytter, Y., & Schünemann, H. J. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.09.012 · Cited by 1,911
Self-citationNatural language processing: state of the art, current trends and challenges
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13428-4 · Cited by 1,823
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047
Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial
Meskó, B. (2023) · Journal of Medical Internet Research · doi:10.2196/50638 · Cited by 709
Yeo, Y. H., Samaan, J. S., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Ayoub, W. S., Yang, J. D., Liran, O., Spiegel, B. M. R., et al. (2023) · Clinical and Molecular Hepatology · doi:10.3350/cmh.2023.0089 · Cited by 669
What is missing from descriptions of treatment in trials and reviews?
Glasziou, P., Meats, E., Heneghan, C., & Shepperd, S. (2008) · BMJ · doi:10.1136/bmj.39590.732037.47 · Cited by 614
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al. (2023) · doi:10.21203/rs.3.rs-2566942/v1 · Cited by 598
Ethical Considerations of Using ChatGPT in Health Care
Wang, C., Liu, S., Yang, H., Guo, J., Wu, Y., & Liu, J. (2023) · Journal of Medical Internet Research · doi:10.2196/48009 · Cited by 541
Rao, A. S., Kim, J., Kamineni, M., Pang, M., Lie, W., Dreyer, K. J., & Succi, M. D. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.05.003 · Cited by 509
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021) · Nature Medicine · doi:10.1038/s41591-021-01312-x · Cited by 494
Accuracy and Reliability of Chatbot Responses to Physician Questions
Goodman, R. S., Patrinely, J. R., Stone, C. A., Zimmerman, E., Donald, R. R., Chang, S. S., Berkowitz, S. T., Finn, A. P., Jahangir, E., Scoville, E. A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.36483 · Cited by 484
Drawbacks of artificial intelligence and their potential solutions in the healthcare sector.
khan, B., Fatima, H., Qureshi, A., Kumar, S., Hanan, A., Hussain, J., & Abdullah, S. (2023) · Biomedical Materials & Devices · doi:10.1007/s44174-023-00063-2 · Cited by 461
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Large language models propagate race-based medicine
Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00939-z · Cited by 417
Rao, A., Pang, M., Kim, J., Kamineni, M., Lie, W., Prasad, A. K., Landman, A., Dreyer, K., & Succi, M. D. (2023) · Journal of Medical Internet Research · doi:10.2196/48659 · Cited by 412
Using AI-generated suggestions from ChatGPT to optimize clinical decision support
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad072 · Cited by 397
Walker, H. L., Ghani, S., Kuemmerli, C., Nebiker, C. A., Müller, B. P., Raptis, D. A., & Staubli, S. M. (2023) · Journal of Medical Internet Research · doi:10.2196/47479 · Cited by 363
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359
Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., & Shimizu, T. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20043378 · Cited by 351
Ethical and regulatory challenges of large language models in medicine
Ong, J. C. L., Chang, S. Y.-H., William, W., Butte, A. J., Shah, N. H., Chew, L. S. T., Liu, N., Doshi-Velez, F., Lu, W., Savulescu, J., et al. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00061-x · Cited by 342
Shahsavar, Y., & Choudhury, A. (2023) · JMIR Human Factors · doi:10.2196/47564 · Cited by 341
Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C., Lam, J. S. H., Chen, D. Z., Goh, J. H. L., Tan, M. C. J., Sheng, B., et al. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104770 · Cited by 329
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About Cancer
Pan, A., Musheyev, D., Bockelman, D., Loeb, S., & Kabarriti, A. E. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2947 · Cited by 289
How AI Responds to Common Lung Cancer Questions: ChatGPT versus Google Bard
Rahsepar, A. A., Tavakoli, N., Kim, G. H. J., Hassani, C., Abtin, F. G., & Bedayat, A. (2023) · Radiology · doi:10.1148/radiol.230922 · Cited by 283
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282
Assessing ChatGPT Responses to Common Patient Questions Regarding Total Hip Arthroplasty
Mika, A. P., Martin, J. R., Engstrom, S. M., Polkowski, G. G., & Wilson, J. M. (2023) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.23.00209 · Cited by 273
Bernstein, I. A., Zhang, Y., Govil, D., Majid, I., Chang, R. T., Sun, Y., Shue, A., Chou, J. C., Schehlein, E. M., Christopher, K. L., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.30320 · Cited by 260
Hopkins, A. M., Logan, J. M., Kichenadasse, G., & Sorich, M. J. (2023) · JNCI Cancer Spectrum · doi:10.1093/jncics/pkad010 · Cited by 259
Use of Artificial Intelligence Chatbots for Cancer Treatment Information
Chen, S., Kann, B. H., Foote, M. B., Aerts, H. J. W. L., Savova, G. K., Mak, R. H., & Bitterman, D. S. (2023) · JAMA Oncology · doi:10.1001/jamaoncol.2023.2954 · Cited by 251
Appropriateness of breast cancer prevention and screening recommendations provided by ChatGPT
Haver, H. L., Ambinder, E. B., Bahl, M., Oluyemi, E. T., Jeudy, J., & Yi, P. H. (2023) · Radiology · doi:10.1148/radiol.230424 · Cited by 247
Large language model (ChatGPT) as a support tool for breast tumor board
Sorin, V., Klang, E., Sklair‐Levy, M., Cohen, I., Zippel, D. B., Lahat, N. B., Konen, E., & Barash, Y. (2023) · npj Breast Cancer · doi:10.1038/s41523-023-00557-8 · Cited by 223
Gravel, J., D’Amours-Gravel, M., & Osmanlliu, E. (2023) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2023.05.004 · Cited by 222
Can ChatGPT be used in oral and maxillofacial surgery?
Balel, Y. (2023) · Journal of Stomatology Oral and Maxillofacial Surgery · doi:10.1016/j.jormas.2023.101471 · Cited by 201
Xie, Y., Seth, I., Hunter-Smith, D. J., Rozen, W. M., Ross, R., & Lee, M. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03338-7 · Cited by 198
Momenaei, B., Wakabayashi, T., Shahlaee, A., Durrani, A. F., Pandit, S. A., Wang, K. Y., Mansour, H. A., Abishek, R. M., Xu, D. S., Sridhar, J. S., et al. (2023) · Ophthalmology Retina · doi:10.1016/j.oret.2023.05.022 · Cited by 189
Factors affecting the quality and reliability of online health information
Battineni, G., Baldoni, S., Chintalapudi, N., Sagaro, G. G., Pallotta, G., Nittari, G., & Amenta, F. (2020) · Digital Health · doi:10.1177/2055207620948996 · Cited by 186
AI chatbots not yet ready for clinical use.
Au Yeung, J., Kraljevic, Z., Luintel, A., Balston, A., Idowu, E., Dobson, R. J., & Teo, J. T. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2023.1161098 · Cited by 176
Caruccio, L., Cirillo, S., Polese, G., Solimando, G., Sundaramurthy, S., & Tortora, G. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121186 · Cited by 171
Deiana, G., Dettori, M., Arghittu, A., Azara, A., Gabutti, G., & Castiglia, P. (2023) · Vaccines · doi:10.3390/vaccines11071217 · Cited by 168
Rau, A., Rau, S., Zöller, D., Fink, A., Tran, H. D., Wilpert, C., Nattenmüller, J., Neubauer, J., Bamberg, F., Reisert, M., et al. (2023) · Radiology · doi:10.1148/radiol.230970 · Cited by 159
Zhu, L., Mou, W., & Chen, R. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04123-5 · Cited by 151
Artificial intelligence‐based chatbot patient information on common retinal diseases using ChatGPT
Potapenko, I. O., Boberg‐Ans, L. C., Hansen, M. S., Klefter, O. N., van Dijk, E. H. C., & Subhi, Y. (2023) · Acta Ophthalmologica · doi:10.1111/aos.15661 · Cited by 148
ChatGPT Answers Common Patient Questions About Colonoscopy
Lee, T.-C., Staller, K., Botoman, V., Pathipati, M. P., Varma, S., & Kuo, B. (2023) · Gastroenterology · doi:10.1053/j.gastro.2023.04.033 · Cited by 146
Bulck, L. V., & Moons, P. (2023) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvad038 · Cited by 144
Nastasi, A. J., Courtright, K. R., Halpern, S. D., & Weissman, G. E. (2023) · Scientific Reports · doi:10.1038/s41598-023-45223-y · Cited by 132
Quality of information and appropriateness of ChatGPT outputs for urology patients
Cocci, A., Pezzoli, M., Re, M. L., Russo, G. I., Asmundo, M. G., Fode, M. M., Cacciamani, G. E., Cimino, S., Minervini, A., & Durukan, E. (2023) · Prostate Cancer and Prostatic Diseases · doi:10.1038/s41391-023-00705-y · Cited by 131
Hirosawa, T., Kawamura, R., Harada, Y., Mizuta, K., Tokumasu, K., Kaji, Y., Suzuki, T., & Shimizu, T. (2023) · JMIR Medical Informatics · doi:10.2196/48808 · Cited by 130
The Potential of ChatGPT as a Self-Diagnostic Tool in Common Orthopedic Diseases: Exploratory Study
Kuroiwa, T., Sarcon, A. K., Ibara, T., Yamada, E., Yamamoto, A., Tsukamoto, K., & Fujita, K. (2023) · Journal of Medical Internet Research · doi:10.2196/47621 · Cited by 129
Fraser, H., Crossland, D., Bacher, I., Ranney, M., Madsen, T., & Hilliard, R. (2023) · JMIR mHealth and uHealth · doi:10.2196/49995 · Cited by 128
Musheyev, D., Pan, A., Loeb, S., & Kabarriti, A. E. (2023) · European Urology · doi:10.1016/j.eururo.2023.07.004 · Cited by 126
Wagner, M. W., & Ertl-Wagner, B. B. (2024) · Canadian Association of Radiologists Journal · doi:10.1177/08465371231171125 · Cited by 125
Seth, I., Cox, A., Xie, Y., Bulloch, G., Hunter‐Smith, D. J., Rozen, W. M., & Ross, R. J. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad140 · Cited by 125
Al-Ashwal, F. Y., Zawiah, M., Gharaibeh, L., Abu-Farha, R., & Bitar, A. N. (2023) · Drug Healthcare and Patient Safety · doi:10.2147/dhps.s425858 · Cited by 125
Moshirfar, M., Altaf, A. W., Stoakes, I. M., Tuttle, J. J., & Hoopes, P. C. (2023) · Cureus · doi:10.7759/cureus.40822 · Cited by 124
Vaira, L. A., Lechien, J. R., Abbate, V., Allevi, F., Audino, G., Beltramini, G. A., Bergonzani, M., Bolzoni, A. R., Committeri, U., Crimi, S., et al. (2023) · Otolaryngology · doi:10.1002/ohn.489 · Cited by 123
Davis, R., Eppler, M., Ayo-Ajibola, O., Loh-Doyle, J. C., Nabhani, J., Samplaski, M., Gill, I., & Cacciamani, G. E. (2023) · Journal of Urology · doi:10.1097/ju.0000000000003615 · Cited by 119
Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study
Levkovich, I., & Elyoseph, Z. (2023) · JMIR Mental Health · doi:10.2196/51232 · Cited by 119
Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4
Krusche, M., Callhoff, J., Knitza, J., & Ruffer, N. (2023) · Rheumatology International · doi:10.1007/s00296-023-05464-6 · Cited by 116
Evaluating Artificial Intelligence Responses to Public Health Questions
Ayers, J. W., Zhu, Z., Poliak, A., Leas, E. C., Dredze, M. H., Hogarth, M. A., & Smith, D. M. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.17517 · Cited by 115
Hurley, E. T., Crook, B. S., Lorentz, S. G., Danilkowicz, R. M., Lau, B. C., Taylor, D. C., Dickens, J. F., Anakwenze, O. A., & Klifto, C. S. (2023) · Arthroscopy The Journal of Arthroscopic and Related Surgery · doi:10.1016/j.arthro.2023.07.048 · Cited by 107
Chiesa-Estomba, C. M., Lechien, J. R., Vaira, L. A., Brunet, A., Cammaroto, G., Mayo‐Yáñez, M., Barrueco, Á. S., & Saga-Gutiérrez, C. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08104-8 · Cited by 106
Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty
Dubin, J. A., Bains, S. S., Chen, Z., Hameed, D., Nace, J., Mont, M. A., & Delanois, R. E. (2023) · The Journal of Arthroplasty · doi:10.1016/j.arth.2023.04.007 · Cited by 103
Coşkun, B., Ocakoğlu, G., Yetemen, M., & Kaygısız, O. (2023) · Urology · doi:10.1016/j.urology.2023.05.040 · Cited by 102
Haemmerli, J., Sveikata, L., Nouri, A., May, A. T., Égervári, K., Freyschlag, C. F., Lobrinus, J. A., Migliorini, D., Momjian, S., Sanda, N., et al. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100775 · Cited by 102
Gordon, E. B., Towbin, A. J., Wingrove, P. M., Shafique, U., Haas, B. M., Kitts, A. B., Feldman, J., & Furlan, A. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.09.011 · Cited by 102
Artificial intelligence chatbot performance in triage of ophthalmic conditions
Lyons, R. J., Arepalli, S. R., Fromal, O., Choi, J. D., & Jain, N. (2024) · Canadian Journal of Ophthalmology · doi:10.1016/j.jcjo.2023.07.016 · Cited by 101
ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results
Hristidis, V., Ruggiano, N., Brown, E. L., Ganta, S. R. R., & Stewart, S. (2023) · Journal of Medical Internet Research · doi:10.2196/48966 · Cited by 100
Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions
Ayoub, N., Lee, Y.-J., Grimm, D. R., & Balakrishnan, K. (2023) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2023.0704 · Cited by 98
Self-citationHarskamp, R. E., & Clercq, L. D. (2024) · Acta cardiologica. Supplementum · doi:10.1080/00015385.2024.2303528 · Cited by 95
Lahat, A., Shachar, E., Avidan, B., Glicksberg, B. S., & Klang, E. (2023) · Diagnostics · doi:10.3390/diagnostics13111950 · Cited by 92
Harskamp, R. E., & De Clercq, L. (2023) · doi:10.1101/2023.03.25.23285475 · Cited by 92
Levkovich, I., & Elyoseph, Z. (2023) · Family Medicine and Community Health · doi:10.1136/fmch-2023-002391 · Cited by 91
Assessing the Value of ChatGPT for Clinical Decision Support Optimization
Liu, S., Wright, A. P., Patterson, B. L., Wanderer, J. P., Turer, R. W., Nelson, S. D., McCoy, A. B., Sittig, D. F., & Wright, A. (2023) · doi:10.1101/2023.02.21.23286254 · Cited by 90
Chervenak, J., Lieman, H., Blanco-Breindel, M., & Jindal, S. (2023) · Fertility and Sterility · doi:10.1016/j.fertnstert.2023.05.151 · Cited by 87
Seth, I., Lim, B., Xie, Y., Cevik, J., Rozen, W. M., Ross, R. J., & Lee, M. (2023) · Aesthetic Surgery Journal Open Forum · doi:10.1093/asjof/ojad084 · Cited by 86
Gebrael, G., Sahu, K. K., Chigarira, B., Tripathi, N., Mathew Thomas, V., Sayegh, N., Maughan, B. L., Agarwal, N., Swami, U., & Li, H. (2023) · Cancers · doi:10.3390/cancers15143717 · Cited by 85
Biswas, S., Logan, N. S., Davies, L. N., Sheppard, A. L., & Wolffsohn, J. S. (2023) · Ophthalmic and Physiological Optics · doi:10.1111/opo.13207 · Cited by 84
Autonomous AI systems in the face of liability, regulations and costs.
Saenz, A. D., Harned, Z., Banerjee, O., Abràmoff, M. D., & Rajpurkar, P. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00929-1 · Cited by 83
Evaluating ChatGPT responses on obstructive sleep apnea for patient education
Campbell, D. J., Estephan, L. E., Mastrolonardo, E. V., Amin, D. R., Huntley, C. T., & Boon, M. S. (2023) · Journal of Clinical Sleep Medicine · doi:10.5664/jcsm.10728 · Cited by 82
Utilizing ChatGPT-4 for Providing Medical Information on Blepharoplasties to Patients
Cox, A., Seth, I., Xie, Y., Hunter‐Smith, D. J., & Rozen, W. M. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad096 · Cited by 79
Performance and biases of large language models in public opinion simulation.
Qu, Y., & Wang, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03609-x · Cited by 79
Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettes
Franco D’Souza, R., Amanullah, S., Mathew, M., & Surapaneni, K. M. (2023) · Asian Journal of Psychiatry · doi:10.1016/j.ajp.2023.103770 · Cited by 77
Rasmussen, M. L. R., Larsen, A., Subhi, Y., & Potapenko, I. O. (2023) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-023-06078-1 · Cited by 76
ChatGPT as a Complementary Mental Health Resource: A Boon or a Bane
Farhat, F. (2023) · doi:10.20944/preprints202307.1479.v1 · Cited by 76
ChatGPT and Lacrimal Drainage Disorders: Performance and Scope of Improvement
Ali, M. J. (2023) · Ophthalmic Plastic and Reconstructive Surgery · doi:10.1097/iop.0000000000002418 · Cited by 75
AI chatbots and (mis)information in public health: impact on vulnerable communities.
Meyrowitsch, D. W., Jensen, A. K., Sørensen, J. B., & Varga, T. V. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1226776 · Cited by 73
Ye, C., Zweck, E., Ma, Z., Smith, J., & Katz, S. (2024) · Arthritis & Rheumatology · doi:10.1002/art.42737 · Cited by 69
Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios
Qu, R. W., Qureshi, U., Petersen, G., & Lee, S. C. (2023) · OTO Open · doi:10.1002/oto2.67 · Cited by 68
Evaluating the performance of ChatGPT in answering questions related to pediatric urology
Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., Gelmiş, M., Sarılar, Ö., & Özgör, F. (2023) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2023.08.003 · Cited by 67
Szczesniewski, J. J., Fouz, C. T., Alba, A. R., Díaz-Goizueta, F. J., García-Tello, A., & González, L. L. (2023) · World Journal of Urology · doi:10.1007/s00345-023-04563-0 · Cited by 67
Is ChatGPT an Evidence-based Doctor?
Zhou, Z., Wang, X., Li, X., & Liao, L. (2023) · European Urology · doi:10.1016/j.eururo.2023.03.037 · Cited by 66
Application of artificial intelligence in medical technologies: a systematic review of main trends.
Bitkina, O. V., Park, J., & Kim, H. K. (2023) · Digital Health · doi:10.1177/20552076231189331 · Cited by 66
Sezgin, E., Chekeni, F., Lee, J., & Keim, S. (2023) · Journal of Medical Internet Research · doi:10.2196/49240 · Cited by 65
Coşkun, B. N., Yağız, B., Ocakoğlu, G., Dalkılıç, E., & Pehlivan, Y. (2023) · Rheumatology International · doi:10.1007/s00296-023-05473-5 · Cited by 65
Large language models will not replace healthcare professionals: curbing popular fears and hype.
Thirunavukarasu, A. J. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231173123 · Cited by 63
Self-citationSeth, I., Xie, Y., Rodwell, A., Gracias, D., Bulloch, G., Hunter‐Smith, D. J., & Rozen, W. M. (2023) · The Journal Of Hand Surgery · doi:10.1016/j.jhsa.2023.07.003 · Cited by 62
Maillard, A., Micheli, G., Lefèvre, L., Guyonnet, C., Poyart, C., Canouï, É., Belan, M., & Charlier, C. (2023) · Clinical Infectious Diseases · doi:10.1093/cid/ciad632 · Cited by 62
Gianola, S., Bargeri, S., Castellini, G., Cook, C., Palese, A., Pillastrini, P., Salvalaggio, S., Turolla, A., & Rossettini, G. (2024) · Journal of Orthopaedic & Sports Physical Therapy · doi:10.2519/jospt.2024.12151 · Cited by 62
ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases
Mehnen, L., Gruarin, S., Vasileva, M., & Knapp, B. (2023) · doi:10.1101/2023.04.20.23288859 · Cited by 60
Gabriel, J., Shafik, L., Alanbuki, A., & Larner, T. (2023) · International Urology and Nephrology · doi:10.1007/s11255-023-03729-4 · Cited by 59
ChatGPT: Can You Prepare My Patients for [18F]FDG PET/CT and Explain My Reports?
Rogasch, J. M. M., Metzger, G., Preisler, M., Galler, M., Thiele, F., Brenner, W., FELDHAUS, F. W., Wetz, C., Amthauer, H., Furth, C., et al. (2023) · Journal of Nuclear Medicine · doi:10.2967/jnumed.123.266114 · Cited by 59
Kao, H.-J., Chien, T.-W., Wang, W.-C., Chou, W., & Chow, J. C. (2023) · Medicine · doi:10.1097/md.0000000000034068 · Cited by 57
Evaluating the performance of ChatGPT in answering questions related to urolithiasis
Çakır, H., Çağlar, U., Yıldız, O., Meriç, A., Ayrancı, A., & Özgör, F. (2023) · International Urology and Nephrology · doi:10.1007/s11255-023-03773-0 · Cited by 57
Nielsen, J. P. S., von Buchwald, C., & Grønhøj, C. (2023) · Acta Oto-Laryngologica · doi:10.1080/00016489.2023.2254809 · Cited by 55
Salazar, G. Z., Zúñiga, D., Vindel-Zuñiga, C., Yoong, A. M., Hincapié-Arias, S. M., Zúñiga, A. B., Zúñiga, P., Salazar, E., & Zúñiga, B. (2023) · Cureus · doi:10.7759/cureus.45473 · Cited by 55
Lim, D. Y. Z., Tan, Y. B., Koh, J. T. E., Tung, J. Y. M., Sng, G. G. R., Tan, D. M. Y., & Tan, C. (2023) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.16375 · Cited by 55
Kusunose, K., Kashima, S., & Sata, M. (2023) · Circulation Journal · doi:10.1253/circj.cj-23-0308 · Cited by 54
Stroop, A., Stroop, T., Alsofy, S. Z., Nakamura, M., Möllmann, F., Greiner, C., & Stroop, R. (2023) · European Spine Journal · doi:10.1007/s00586-023-07975-z · Cited by 53
Reporting standards for the use of large language model-linked chatbots for health advice.
Huo, B., Cacciamani, G. E., Collins, G. S., McKechnie, T., Lee, Y., & Guyatt, G. (2023) · Nature Medicine · doi:10.1038/s41591-023-02656-2 · Cited by 53
Self-citationChatGPT vs. web search for patient questions: what does ChatGPT do better?
Shen, S. A., Perez-Heydrich, C. A., Xie, D. X., & Nellis, J. C. (2024) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-024-08524-0 · Cited by 53
Russe, M. F., Reisert, M., Bamberg, F., & Rau, A. (2024) · RöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren · doi:10.1055/a-2264-5631 · Cited by 52
Lower, K., Seth, I., Lim, B., & Seth, N. (2023) · Indian Journal of Orthopaedics · doi:10.1007/s43465-023-00967-7 · Cited by 50
ChatGPT performance in laryngology and head and neck surgery: a clinical case-series
Lechien, J. R., Georgescu, B. M., Hans, S., & Chiesa-Estomba, C. M. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08282-5 · Cited by 50
Emile, S. H., Horesh, N., Freund, M. R., Pellino, G., Oliveira, L. C. C., Wignakumar, A., & Wexner, S. D. (2023) · Surgery · doi:10.1016/j.surg.2023.06.005 · Cited by 49
Shao, C.-Y., Li, H., Liu, X.-L., Li, C., Yang, L.-Q., Zhang, Y.-J., Luo, J., & Zhao, J. (2023) · Interactive Journal of Medical Research · doi:10.2196/46900 · Cited by 49
Dwyer, T., Hoit, G., Burns, D., Higgins, J. P., Chang, J. S., Whelan, D. B., Kiroplis, I., & Chahal, J. (2023) · Arthroscopy Sports Medicine and Rehabilitation · doi:10.1016/j.asmr.2023.01.020 · Cited by 48
Capacity of ChatGPT to Identify Guideline-Based Treatments for Advanced Solid Tumors
Schulte, B. C. (2023) · Cureus · doi:10.7759/cureus.37938 · Cited by 48
BPPV Information on Google Versus AI (ChatGPT)
Bellinger, J. R., Chapa, J. S. D. L., Kwak, M. W., Ramos, G. A., Morrison, D. R., & Kesser, B. W. (2023) · Otolaryngology · doi:10.1002/ohn.506 · Cited by 47
Inherent bias in large language models: a random sampling analysis.
Ayoub, N. F., Balakrishnan, K., Ayoub, M. S., Barrett, T. F., David, A. P., & Gray, S. T. (2024) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2024.03.003 · Cited by 47
Long, C., Subburam, D., Lowe, K., dos Santos, A., Zhang, J. J., Hwang, S. W., Saduka, N., Horev, Y., Su, T., Côté, D. W. J., et al. (2024) · Otolaryngology · doi:10.1002/ohn.864 · Cited by 47
Levartovsky, A., Ben‐Horin, S., Kopylov, U., Klang, E., & Barash, Y. (2023) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000002483 · Cited by 46
Comparative Evaluation of Diagnostic Accuracy Between Google Bard and Physicians
Hirosawa, T., Mizuta, K., Harada, Y., & Shimizu, T. (2023) · The American Journal of Medicine · doi:10.1016/j.amjmed.2023.08.003 · Cited by 46
Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis
Rojas-Carabali, W. A., Cifuentes‐González, C., Wei, X., Putera, I., Sen, A. C., Thng, Z. X., Agrawal, R., Elze, T., Sobrin, L., Kempen, J. H., et al. (2023) · Ocular Immunology and Inflammation · doi:10.1080/09273948.2023.2253471 · Cited by 46
Yun, J. Y., Kim, D. J., Lee, N., & Kim, E. K. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105219 · Cited by 45
Appropriateness of ChatGPT in Answering Heart Failure Related Questions
King, R. C., Samaan, J. S., Yeo, Y. H., Mody, B. P., Lombardo, D. M., & Ghashghaei, R. (2024) · Heart Lung and Circulation · doi:10.1016/j.hlc.2024.03.005 · Cited by 45
A descriptive study based on the comparison of ChatGPT and evidence-based neurosurgeons
Liu, J., Zheng, J., Cai, X., Wu, D., & Yin, C. (2023) · iScience · doi:10.1016/j.isci.2023.107590 · Cited by 44
The potential of chatbots in chronic venous disease patient management
Athavale, A., Baier, J., Ross, E., & Fukaya, E. (2023) · JVS-Vascular Insights · doi:10.1016/j.jvsvi.2023.100019 · Cited by 43
Rizwan, A., & Sadiq, T. (2023) · Cureus · doi:10.7759/cureus.43106 · Cited by 43
Quality of ChatGPT Responses to Questions Related To Liver Transplantation
Endo, Y., Sasaki, K., Moazzam, Z., Lima, H. A., Schenk, A. D., Limkemann, A. J. P., Washburn, W. K., & Pawlik, T. M. (2023) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-023-05714-9 · Cited by 42
Mishra, A., Begley, S. L., Chen, A., Rob, M., Pelcher, I., Ward, M., & Schulder, M. (2023) · Neurosurgery · doi:10.1227/neu.0000000000002598 · Cited by 38
Luykx, J. J., Gerritse, F. L., Habets, P. C., & Vinkers, C. H. (2023) · World Psychiatry · doi:10.1002/wps.21145 · Cited by 38
Rosen, S. R., & Saban, M. (2023) · European Radiology · doi:10.1007/s00330-023-10230-0 · Cited by 38
Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms
Nguyen, D., Swanson, D., Newbury, A., & Kim, Y. H. (2024) · Academic Radiology · doi:10.1016/j.acra.2023.11.002 · Cited by 38
Kataoka, Y., Takemura, T., Sasajima, M., & Katoh, N. (2021) · JMIR Cancer · doi:10.2196/26911 · Cited by 37
ChatGPT’s ability to comprehend and answer cirrhosis related questions in Arabic
Samaan, J. S., Yeo, Y. H., Ng, W. H., Ting, P., Trivedi, H. D., Vipani, A., Yang, J. D., Liran, O., Spiegel, B. M. R., Kuo, A., et al. (2023) · Arab Journal of Gastroenterology · doi:10.1016/j.ajg.2023.08.001 · Cited by 37
ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis.
Ahmed, W., Saturno, M., Rajjoub, R., Duey, A. H., Zaidat, B., Hoang, T., Restrepo Mejia, M., Gallate, Z. S., Shrestha, N., Tang, J., et al. (2024) · European Spine Journal · doi:10.1007/s00586-024-08198-6 · Cited by 37
Li, W., Chen, J., Chen, F., Liang, J., & Yu, H. (2023) · Aesthetic Plastic Surgery · doi:10.1007/s00266-023-03660-0 · Cited by 36
Altamimi, I., Altamimi, A., Alhumimidi, A. S., Altamimi, A., & Temsah, M.-H. A. (2023) · Cureus · doi:10.7759/cureus.40351 · Cited by 33
ChatGPT Surpasses 1000 Publications on PubMed: Envisioning the Road Ahead
Temsah, M.-H. A., Altamimi, I., Jamal, A. A., Alhasan, K. A., & Al-Eyadhy, A. A. (2023) · Cureus · doi:10.7759/cureus.44769 · Cited by 33
Liu, H. Y., Bonetti, M. A., Jeong, T., Pandya, S., Nguyen, V. T., & Egro, F. M. (2023) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2023.07.039 · Cited by 32
Galido, P. V., Butala, S., Chakerian, M., & Agustines, D. (2023) · Cureus · doi:10.7759/cureus.38166 · Cited by 29
Artificial Intelligence-Based ChatGPT Responses for Patient Questions on Optic Disc Drusen
Potapenko, I. O., Malmqvist, L., Subhi, Y., & Hamann, S. (2023) · Ophthalmology and Therapy · doi:10.1007/s40123-023-00800-2 · Cited by 29
Stevenson, E., Walsh, C., & Hibberd, L. (2023) · Annals of Clinical Biochemistry International Journal of Laboratory Medicine · doi:10.1177/00045632231203473 · Cited by 28
ChatGPT in Occupational Medicine: A Comparative Study with Human Experts
Padovan, M., Cosci, B., Petillo, A., Nerli, G., Porciatti, F., Scarinci, S., Carlucci, F., Dell’Amico, L., Meliani, N., Necciari, G., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11010057 · Cited by 28
Huo, B., Calabrese, E., Sylla, P., Kumar, S., Ignacio, R. C., Oviedo, R., Hassan, I., Slater, B. J., Kaiser, A., Walsh, D. S., et al. (2024) · Surgical Endoscopy · doi:10.1007/s00464-024-10807-w · Cited by 28
Self-citationMind + Machine: ChatGPT as a Basic Clinical Decisions Support Tool
Ayoub, M., Ballout, A. A., Zayek, R. A., & Ayoub, N. F. (2023) · Cureus · doi:10.7759/cureus.43690 · Cited by 25
Janopaul-Naylor, J. R., Koo, A., Qian, D. C., McCall, N. S., Liu, Y., & Patel, S. A. (2023) · American Journal of Clinical Oncology · doi:10.1097/coc.0000000000001050 · Cited by 25
Regulating the safety of health–related artificial intelligence.
Silva, M., Flood, C., Goldenberg, A., & Singh, D. (2022) · Healthcare Policy | Politiques de Santé · doi:10.12927/hcpol.2022.26824 · Cited by 23
Training ChatGPT Models in Assisting Urologists in Daily Practice
Manolitsis, I. S., Feretzakis, G., Tzelves, L. I., Kalles, D., Katsimperis, S. N., Angelopoulos, P., Αναστασίου, Α., Koutsouris, D., Kosmidis, T., Verykios, V. S., et al. (2023) · Studies in health technology and informatics · doi:10.3233/shti230562 · Cited by 22
“Vertigo, likely peripheral”: the dizzying rise of ChatGPT
Chee, J., Kwa, E. D., & Goh, X. (2023) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-023-08135-1 · Cited by 20
Benirschke, R. C., Wodskow, J., Prasai, K., Freeman, A., Lee, J. M., & Groth, J. (2023) · American Journal of Clinical Pathology · doi:10.1093/ajcp/aqad106 · Cited by 20
Zhou, Y., Moon, C. N., Szatkowski, J., Moore, D., & Stevens, J. M. (2023) · European Journal of Orthopaedic Surgery & Traumatology · doi:10.1007/s00590-023-03742-4 · Cited by 18
From Understanding to Utilization: A Survey on Explainability for Large Language Models
Luo, H., & Specia, L. (2024) · arXiv · doi:10.48550/arxiv.2401.12874 · Cited by 18
Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?
Chowdhury, M., Lim, E., Higham, A., McKinnon, R., Ventoura, N., He, Y., & De Pennington, N. (2023) · Proceedings of the 5th Clinical Natural Language Processing Workshop · doi:10.18653/v1/2023.clinicalnlp-1.17 · Cited by 17
O’Hagan, R., Kim, R. H., Abittan, B. J., Caldas, S. A., Ungar, J. P., & Ungar, B. (2023) · Dermatology · doi:10.1159/000534005 · Cited by 17
Ali, H., Patel, P., Obaitan, I., Mohan, B. P., Sohail, A. H., Smith-Martinez, L., Lambert, K., Gangwani, M. K., Easler, J. J., & Adler, D. G. (2023) · iGIE · doi:10.1016/j.igie.2023.10.001 · Cited by 15
AI's deep dive into complex pediatric inguinal hernia issues: a challenge to traditional guidelines?
Wang, G., Liu, Q., Chen, G., Xia, B., Zeng, D., Chen, G., & Guo, C. (2023) · Hernia · doi:10.1007/s10029-023-02900-1 · Cited by 14
Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference
Wang, C., Liu, S. X., & Awadallah, A. H. (2023) · arXiv · doi:10.48550/arxiv.2303.04673 · Cited by 13
Evaluating ChatGPT's Accuracy in Lung Cancer Prevention and Screening Recommendations
Haver, H. L., Lin, C. T., Sirajuddin, A., Yi, P. H., & Jeudy, J. J. (2023) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.230115 · Cited by 13
Gracias, D., Siu, A. H. Y., Seth, I., Dooreemeah, D., & Lee, A. (2023) · ANZ Journal of Surgery · doi:10.1111/ans.18736 · Cited by 13
Can ChatGPT outperform a neurosurgical trainee? a prospective comparative study.
Williams, S. C., Starup-Hansen, J., Funnell, J. P., Hanrahan, J. G., Valetopoulou, A., Singh, N., Sinha, S., Muirhead, W. R., & Marcus, H. J. (2025) · British Journal of Neurosurgery · doi:10.1080/02688697.2024.2308222 · Cited by 13
Li, J., Gao, X., Dou, T., Gao, Y., Xu, L., & Zhu, W. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082344 · Cited by 13
(2017) · PubMed Central · Cited by 12
Utility of GPT-4 as an Informational Patient Resource in Otolaryngology
Suresh, K., Rathi, V., Nwosu, O., Partain, M. P., Glicksman, J. T., Jowett, N., & Crowson, M. G. (2023) · doi:10.1101/2023.05.14.23289944 · Cited by 11
Gwak, G., Hwang, U., Jung, S. H., & Kim, J. H. (2023) · Journal of Musculoskeletal Science and Technology · doi:10.29273/jmst.2023.7.1.8 · Cited by 11
Assessing the Soft Tissue Infection Expertise of ChatGPT and Bard Compared to IDSA Recommendations
Bonetti, M. A., Giorgino, R., Naegeli, M., Liu, H. Y., & Egro, F. M. (2023) · Annals of Biomedical Engineering · doi:10.1007/s10439-023-03372-1 · Cited by 10
Leveraging ChatGPT to aid patient education on coronary angiogram
Koh, S. J. Q., Yeo, K. K., & Yap, J. J. L. (2023) · Annals of the Academy of Medicine Singapore · doi:10.47102/annals-acadmedsg.2023138 · Cited by 9
Yıldız, M. S., & Alper, A. (2023) · Journal of Consumer Health on the Internet · doi:10.1080/15398285.2023.2240652 · Cited by 9
Can ChatGPT assist with the initial triage? A case study of stroke in young females
Rosen, S., & Saban, M. (2023) · International Emergency Nursing · doi:10.1016/j.ienj.2023.101340 · Cited by 7
Assessing the performance of GPT-4 in the filed of osteoarthritis and orthopaedic case consultation
Li, J., Gao, X., Dou, T., Gao, Y., & Zhu, W. (2023) · doi:10.1101/2023.08.06.23293735 · Cited by 4
Relevance of medical information obtained from ChatGPT: Are large language models friends or foes?
Mesnier, J., Suc, G., Sayah, N., Abtan, J., & Steg, P. G. (2023) · Archives of cardiovascular diseases · doi:10.1016/j.acvd.2023.07.009 · Cited by 4
Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting
Wang, R., Wang, H., Mi, F., Xue, B., Chen, Y., Wong, K.-F., & Xu, R. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.299 · Cited by 4
Can ChatGPT Provide Quality Information on Integrative Oncology? A Brief Report
Lam, C. S., Hua, R., Koon, H. K., Zhou, K. R., Lam, T. N., Lee, C. P., Lin, W. L., Wong, C. L., Lau, Y. M., Loong, H. H.-F., et al. (2023) · Journal of Integrative and Complementary Medicine · doi:10.1089/jicm.2023.0290 · Cited by 3
Graph coverings and twisted operators
Cimasoni, D., & Kassel, A. (2023) · Algebraic Combinatorics · doi:10.5802/alco.258 · Cited by 2
Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting
Wang, R., Wang, H., Mi, F., Chen, Y., Xue, B., Wong, K.-F., & Xu, R. (2023) · arXiv · doi:10.48550/arxiv.2305.13733 · Cited by 2
Graph coverings and twisted operators
Cimasoni, D., & Kassel, A. (2020) · arXiv · doi:10.48550/arxiv.2012.12575
Research works printed but not linked: 2
- Wang? Y, Visweswaran? S, Kappor? S, Kooragayalu? S, Wu? X. ChatGPT, enhanced with clinical practice guidelines, is a superior decision support tool. [published online August 13, 2023].? MedRxiv.
- Zhou? J, He? X, Sun? L, . SkinGPT-4: an interactive dermatology diagnostic system with visual large language model. Published online April 20, 2023.MedRxiv.
Cited by
Sorted by citations
Large Language Models in Healthcare and Medical Applications: A Review
Maity, S., & Saikia, M. J. (2025) · Bioengineering · doi:10.3390/bioengineering12060631 · Cited by 249
Generative artificial intelligence in medicine
Teo, Z. L., Thirunavukarasu, A. J., Elangovan, K., Cheng, H., Moova, P., Soetikno, B., Nielsen, C., Pollreisz, A., Ting, D. S. J., Morris, R. J. T., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03983-2 · Cited by 119
Self-citationChow, J. C. L., & Li, K. (2025) · Information · doi:10.3390/info16070549 · Cited by 84
DeepSeek’s “Low-Cost” Adoption Across China’s Hospital Systems
Zeng, D., Qin, Y., Sheng, B., & Wong, T. Y. (2025) · JAMA · doi:10.1001/jama.2025.6571 · Cited by 73
Collaborative, T. C. (2025) · BMJ Medicine · doi:10.1136/bmjmed-2025-001632 · Cited by 70
Metadata
- Authors
- Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., Sanger, S., Ramji, K., & Guyatt, G.
- Affiliations
- McMaster University (CA)
- H. Ross University School of Medicine, Miramar, Florida
- University of Toronto (CA)
- Universidad Complutense de Madrid (ES)
- Papageorgiou General Hospital (GR)
- University of Oxford (GB)
- Corresponding author
- Huo, Bright
- ORCID
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2025-02-04
- Volume
- 8
- Issue
- 2
- Pages
- e2457879
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC11795331
- OpenAlex
- W4407135263
- Semantic Scholar
- 8e5b1084560f7b370363a2ad302c602e7a1ba78c
- Cited by
- 283
- References
- 184
- Citations by year
- 2026: 210 · 2025: 73
- FWCI
- 16.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, chatbots, health advice, reporting quality, generative AI, clinical integration, prompt engineering, patient safety, ethical and regulatory considerations, clinical accuracy
- MeSH terms
- Large Language Models, Generative Artificial Intelligence, Artificial Intelligence, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
