CoChat
Sign Up
Cited by 246Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareTopic Modeling

Large language models in medical and healthcare fields: applications, advances, and challenges

Open access
Wang, Dandan, Zhang, Shiqing·2024·Artificial Intelligence Review
DOI 10.1007/s10462-024-10921-0
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are increasingly recognized for their advanced language capabilities, offering significant assistance in diverse areas like medical communication, patient data optimization, and surgical planning. Our survey meticulously searched for papers with keywords such as “medical,” “clinical,” “healthcare,” and “LLMs” across various databases, including ACM and Google Scholar. It sought to delve into the latest trends and applications of LLMs in healthcare, analyzing 175 relevant publications to support both practitioners and researchers in the field. We have compiled 56 experimental datasets, various evaluation methods and reviewed cutting-edge LLMs across tasks. Our comprehensive analysis of LLMs in healthcare applications, including medical question-answering, dialogue summarization, electronic health record generation, scientific research, medical education, medical product safety monitoring, clinical health reasoning, and clinical decision support. Furthermore, we have identified the challenges, including data security, inaccurate information, fairness and bias, plagiarism, copyrights, and accountability, and the potential solutions, namely de-identification framework, references,counterfactually fair prompting,opening and ending control codes, and establishing normative standards,to address these open issues,respectively. The findings of this survey exert a profound impact on spurring innovation in practical applications and addressing inherent challenges within the academic and medical communities.

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews large language models in medical and healthcare fields (LLMMs), covering applications, datasets, evaluation methods, model performance, and ethical/safety challenges. It analyzes 175 relevant publications and compiles 56 experimental datasets to guide practitioners and researchers.

Key findings

  • The survey organizes LLMM applications into medical question-answering, medical dialog summarization, EHR/clinical letter/note generation, scientific research, medical education and translation, medical imaging recognition, clinical health reasoning, medical product safety monitoring and disease diagnosis, clinical decision support, and administrative tasks assistance.
  • It reports that GPT-4 with retrieval-augmented generation (GPT4-RAG) achieved 91.4% accuracy in preoperative guidance, 5.1% higher than human-generated answers at 86.3%, with retrieval in about 1 s and generation in 15–20 s versus human physicians’ average 10 min.
  • For clinical question-answering, ChatGPT reached 87% accuracy without options and 97% in multiple-choice settings on the NEJM “Image Challenge”; GPT-4 achieved 96% accuracy in disease classification with chain-of-thought and few-shot prompts.
  • The paper identifies five challenge areas: data security and privacy preservation, incorrect or misleading information, fairness and bias, transparency/explainability/trustworthiness, and plagiarism/copyright/accountability, with proposed solutions such as de-identification frameworks, references, counterfactually fair prompting, opening and ending control codes, and normative standards.

Methods

  • Survey design: searched Google Scholar and multi-source databases including ACM, Springer, Elsevier, arXiv, and medRxiv for papers from January 2022 to January 2024, focusing on models after ChatGPT’s November 2022 launch and parameters exceeding 10^9.
  • Search strategy: combined application domains (“medical,” “clinical,” “healthcare”) with specific LLM names such as ChatGPT, LLaMA, and PaLM; reviewed titles, abstracts, and keywords, excluded papers without specific medical/healthcare LLMs, and included articles from review papers, arriving at 175 papers.
  • Analysis: categorized applications, compiled 56 experimental datasets, summarized evaluation metrics (e.g., ROUGE, BERTScore, BLEU, accuracy, precision, recall, F1-score, MCC, AUM, AKCM, ASRE), and compared state-of-the-art LLMM performance across tasks.

Limitations

  • The paper states that LLMMs face data security and privacy risks, inaccurate or fabricated information, fairness and bias, limited transparency/explainability/trustworthiness, and plagiarism/copyright/accountability concerns.
  • It notes that extensive training of LLMMs is typically based on English-related datasets, causing lack of medical knowledge and poor performance on Chinese tasks such as disease diagnosis, drug recommendation, and clinical decision support.
  • The text also makes evident that the survey is not a systematic review with a formal PRISMA-style protocol, and several sections appear truncated or garbled, limiting reproducibility of the search and synthesis process.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10462-024-10921-0

Cites

128 of 128 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 91 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The survey compiled 56 experimental datasets.

    “We have compiled 56 experimental datasets, various evaluation methods and reviewed cutting-edge LLMs across tasks.”
    Quote in the paperSubstantial claim
  • Findingcitation

    The assessments and medical plans created by GPT-4 demonstrate a significant association between demographic attributes and patient differences in recommendations for expensive procedures.

    “The assessments and medical plans created by GPT-4 demonstrate a significant association between demographic attributes and patient differences in recommendations for expensive procedures (Zack et al. 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zack et al. 2023

      Not checked in the free run.

  • Findingown data

    For clinical decision support on COPD, PBC, and CKD, both ChatGPT and GPT-4, when equipped with an elaborate clinical guideline, consistently achieved higher F1 scores than without.

    “The results reveal that both ChatGPT and GPT-4, when equipped with an elaborate clinical guideline, consistently achieved higher F1 scores across the board, as detailed in Table 7.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the multiple-choice QA task, Med-PaLM 2 achieved the top performance on the MedQA (USMLE) and PubMedQA datasets.

    “In the Multiple-choice QA task, Med-PaLM 2 achieved the top performance on the MedQA (USMLE) and PubMedQA datasets, while GPT-4 excelled on MedMCQA, MMLU-Medical Genetics, and MMLU-College Biology.”
    Quote in the paperNot a substantial claim
  • Findingown data

    For the MediQA shared task, T5 SAMSum achieved the highest F1 Score.

    “For the MediQA shared task, the BART-Large model yielded the highest BLEU score, and T5 SAMSum achieved the highest F1 Score.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 45
  • Classified as probably not a substantial claim. 33
  • The quote only partly states this claim. 5

  • May combine two claims.

    Findingown data

    The survey identified challenges including data security, inaccurate information, fairness and bias, plagiarism, copyrights, and accountability.

    “Furthermore, we have identified the challenges, including data security, inaccurate information, fairness and bias, plagiarism, copyrights, and accountability, and the potential solutions, namely de-identification framework, references,counterfactually fair prompting,opening and ending control codes, and establishing normative standards,to address these open issues,respectively.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The survey identified potential solutions including de-identification framework, references, counterfactually fair prompting, opening and ending control codes, and establishing normative standards.

    “Furthermore, we have identified the challenges, including data security, inaccurate information, fairness and bias, plagiarism, copyrights, and accountability, and the potential solutions, namely de-identification framework, references,counterfactually fair prompting,opening and ending control codes, and establishing normative standards,to address these open issues,respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The initial search approach yielded a significant number of survey and overview articles, with only a few papers containing relevant models.

    “We found that this approach yielded a significant number of survey and overview articles, with only a few papers containing relevant models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    The practical utility of LLMMs in clinical and research settings is fraught with challenges including data security, privacy preservation, risk of inaccurate information, fairness and bias, plagiarism, copyright, and accountability.

    “Although scholars and medical practitioners have increasingly expressed interest in the application of LLMMs, the practical utility of LLMMs in clinical and research settings is fraught with distinct challenges.These challenges include data security, privacy preservation, the risk of inaccurate information, fairness and bias issues, plagiarism concerns, copyright considerations, and accountability”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    These datasets encompass a range of tasks, including medical question-answering, medical knowledge representation, clinical evidence understanding and integration, diagnosis generation and summarization, and others.

    “These datasets encompass a range of tasks, including medical question-answering, medical knowledge representation, clinical evidence understanding and integration, diagnosis generation and summarization, and others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    GPT-4 was found not to simulate the demographics of medical conditions in various situations, consistently producing clinical hallucinations, including the differential diagnosis of standardized clinical samples, which are more likely to include stereotypes of specific racial, ethnic, and gender identities.

    “For instance, GPT-4 was found not to simulate the demographics of medical conditions in various situations, consistently producing clinical hallucinations, including the differential diagnosis of standardized clinical samples, which are more likely to include stereotypes of specific racial, ethnic, and gender identities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the multiple-choice QA task, GPT-4 excelled on MedMCQA, MMLU-Medical Genetics, and MMLU-College Biology.

    “In the Multiple-choice QA task, Med-PaLM 2 achieved the top performance on the MedQA (USMLE) and PubMedQA datasets, while GPT-4 excelled on MedMCQA, MMLU-Medical Genetics, and MMLU-College Biology.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    This paper presents 56 experimental datasets that are most widely used by researchers in the medical and healthcare domains.

    “This paper presents 56 experimental datasets that are most widely used by researchers in the medical and healthcare domains.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    GPT-4 achieved a score exceeding the passing threshold on the USMLE by more than 20 points without any professional hints, outperforming GPT-3.5 and specialized medical knowledge models such as Med-PaLM and Flan-PaLM.

    “The results indicated that GPT-4 achieved a score exceeding the passing threshold on the USMLE by more than 20 points without any professional hints, outperforming GPT-3.5 and specialized medical knowledge models such as Med-PaLM and Flan-PaLM.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nori et al. 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    LLMs such as GPT-4 and Med-PaLM have demonstrated the ability to answer questions in the USMLE clinical knowledge test with an accuracy of over 80%.

    “LLMs such as GPT-4 and Med-PaLM have demonstrated the ability to answer questions in the USMLE clinical knowledge test with an accuracy of over 80%.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingcitation

    ChatGPT achieved an accuracy level of approximately 60% without any specialized training on the USMLE.

    “ChatGPT achieved an accuracy level of approximately 60% without any specialized training.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kung et al. 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Scaling inference-time computing can enhance the reasoning performance of LLMs.

    “Their findings suggested that scaling inference-time computing can enhance the reasoning performance of LLMs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All papers retrieved from the searches were included in the candidate corpus.

    “All the papers retrieved from the above searches were included in the candidate corpus.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    LLMMs have exhibited remarkable performance in assisting with USMLE, JMLE, and other medical licensing examinations, and in generating and evaluating multiple-choice tests.

    “In the realm of medical education, LLMMs have exhibited remarkable performance, for instance, in assisting human learners and educators with United States Medical Licensing Examination (USMLE), Japanese Medical Licensing Examination (JMLE), and other medical licensing examinations, as well as in generating and evaluating multiple-choice tests (Sallam 2023b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Sallam 2023b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The deployment of LLMs offers valuable insights for various stakeholders in healthcare domains, such as healthcare providers and patients.

    “The deployment of LLMs offers valuable insights for various stakeholders in healthcare domains, such as healthcare providers and patients (Jin and Dobry 2023)).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Jin and Dobry 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Within medical and healthcare domains, GPT-4 can supply the most current literature, draft discharge summaries, analyze medical image characteristics, and identify objects in photographs.

    “Within the medical and healthcare domains, GPT-4 can supply the most current literature in specific fields, draft discharge summaries for patients post-surgery, analyze medical image characteristics, and identify objects in photographs, revealing its significant potential in clinical trials (Waisberg et al. 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Waisberg et al. 2023

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The authors reviewed titles, abstracts, and keywords of candidate papers and excluded those that did not include specific large models in medical or healthcare fields.

    “We then reviewed the titles, abstracts, and keywords of the candidate papers; those that did not include specific large models in the medical or healthcare fields were excluded.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    MDS aids clinicians in identifying potential health risks for patients and supports informed decision-making.

    “The MDS aids clinicians in identifying potential health risks for patients and supports informed decision-making (Patel and Lam 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Patel and Lam 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Agrawal et al. demonstrated that GPT-3 outperforms other models in clinical natural language processing tasks.

    “Agrawal et al. (2022) utilized GPT-based models to extract critical variables from diverse clinical notes, demonstrating that GPT-3 outperforms other models in clinical natural language processing tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Agrawal et al. 2022

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The study focused on research work from January 2022 up until January 2024, with emphasis on models introduced after ChatGPT's launch in November 2022 and those with parameters exceeding 10^9.

    “The focus of this paper is on research work from January 2022 up until the submission of this paper in January 2024, with a particular emphasis on models introduced after the launch of ChatGPT in November 2022, as well as those with parameters exceeding 10^9.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Prior studies often concentrate on a limited number of LLMs or fail to offer a thorough and expansive analysis of LLM applications and associated issues.

    “Nonetheless, these studies often concentrate on a limited number of LLMs or fail to offer a thorough and expansive analysis of LLM applications and the associated potential issues, such as medical dialog summarization, scientific research, medical product safety monitoring, disease diagnosis, clinical decision support, administrative tasks assistance, and ethical concerns regarding data security”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Wang et al. 2023a

      Not checked in the free run.

    • cites Tian et al. 2023

      Not checked in the free run.

    • cites Liu et al. 2023a

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs can serve as powerful tools for data analysis, literature review, and hypothesis generation in scientific research.

    “In scientific research, LLMs can serve as powerful tools for data analysis (Tao et al. 2022), literature review, and hypothesis generation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tao et al. 2022 source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The authors also included articles discussing LLMs in healthcare from multiple review papers in the candidate set, ultimately arriving at 175 papers.

    “Additionally, to ensure a more comprehensive coverage of research content, we also included articles discussing LLMs in the healthcare domain from multiple review papers in the candidate set, ultimately arriving at 175 papers closely related to our research.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Clinical notes summarization assists healthcare practitioners in identifying potential health risks within patients' electronic health records, thereby reducing errors and facilitating informed decision-making.

    “Clinical notes summarization assists healthcare practitioners in identifying potential health risks within patients' electronic health records (Wornow et al. 2023), thereby reducing errors and facilitating informed decision-making.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wornow et al. 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    This survey is the first to focus on LLMMs in relation to real-world application scenarios, available datasets, evaluation methods, and ethical and safety considerations.

    “To our understanding, our survey is the first to focus on LLMMs in relation to realworld application scenarios, available datasets, evaluation methods, and ethical and safety considerations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMMs can aid doctors in accurately and rapidly detecting anomalies and diagnosing diseases or injuries, thereby alleviating radiologists' workload.

    “LLMMs can also be trained to recognize and analyze medical images (e.g., X-rays, magnetic resonance imaging, and ultrasound), video, audio and remote photoplethysmograph signals to identify features and structures (Fan et al. 2024), aiding doctors in accurately and rapidly detecting anomalies and diagnosing diseases or injuries, thereby alleviating the workload of radiologists (Waisberg et al. 202”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Waisberg et al. 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study searched Google Scholar, which encompasses literature from ACM, Springer, Elsevier, arXiv, medRxiv, and other multi-source databases.

    “This study was conducted by searching for relevant papers on the renowned digital library Google Scholar, which encompasses literature from ACM, Springer, Elsevier, arXiv, medRxiv, and other multi-source databases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMMs have the potential to facilitate communication among healthcare professionals, patients, and their families, streamline collection and analysis of patient health data, and assist in surgical plan development.

    “Large language models in the medical and healthcare domain (LLMMs) have the potential to facilitate communication among healthcare professionals, patients, and their families, streamline the collection and analysis of patient health data, and assist in the development of surgical plans (Zhao et al. 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhao et al. 2023 source

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Premisecitation

    Without options, ChatGPT demonstrated an accuracy rate of 87%, while in multiple-choice scenarios, its accuracy reached 97% on the NEJM Image Challenge quiz.

    “Without options, ChatGPT demonstrated an accuracy rate of 87%, while in multiple-choice scenarios, its accuracy reached 97%.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ueda et al. 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    GPT-4 outperformed ChatGPT in general clinical questions, complex questions, and specific disease-related queries in JMLE.

    “The results revealed that GPT-4 outperformed ChatGPT in general clinical questions, complex questions, and specific disease-related queries.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Takagi et al. 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    GPT-4 achieved a score that met the passing standard of the JMLE, demonstrating robust reliability in clinical reasoning and medical knowledge within the Japanese context.

    “Furthermore, GPT-4 achieved a score that met the passing standard of the JMLE, demonstrating its robust reliability in clinical reasoning and medical knowledge within the Japanese context.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Takagi et al. 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This improvement facilitates a deeper understanding of the internal structure and function of various organisms.

    “This improvement facilitates a deeper understanding of the internal structure and function of various organisms (e.g., (Tao et al. 2020)).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tao et al. 2020 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs have become indispensable tools in scientific writing, draft generation, article summarization, language and grammar checks, and translation of multilingual content.

    “(2) LLMs have become indispensable tools in scientific writing, draft generation, article summarization, language and grammar checks, and translation of multilingual content (Liebrenz et al. 2023), engaging in discussions as virtual collaborators, and offering new perspectives and research directions, thereby enhancing the efficiency and diversity of scientific and academic output (Fatani 2023).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Liebrenz et al. 2023 source

      Not checked in the free run.

    • cites Fatani 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    ChatGPT significantly outperformed GPT-3 and InstructGPT on medical question-answering tasks, with its answer level comparable to third-year medical students.

    “The study found that ChatGPT significantly outperformed GPT-3 and InstructGPT on medical question-answering tasks, with its answer level comparable to that of third-year medical students.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gilson et al. 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The remarkable performance of ChatGPT on USMLE has been a significant milestone in medical education.

    “The remarkable performance of ChatGPT on USMLE has been a significant milestone in medical education (Sallam 2023b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Sallam 2023b

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    GPT-3 yielded superior diagnostic outcomes compared to laypersons without domain-specific expertise, although its performance fell short of that of professional physicians.

    “GPT-3 yielded superior diagnostic outcomes compared to laypersons without domain-specific expertise, although its performance fell short of that of professional physicians.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Levine et al. (2023)

      Not checked in the free run.

  • The quote only partly states this claim.

    Premisecitation

    GPT-4 exhibited higher scores in accuracy and completeness for diagnostic and treatment questions related to orthopedic diseases.

    “GPT-4 exhibited higher scores in terms of accuracy and completeness.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Li et al. 2023b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLMs present an opportunity for researchers to streamline research and craft influential articles by facilitating literature reviews, retrieving latest scientific progress, and retrieving key information tailored to different journals.

    “(1) LLMs present an exciting opportunity for researchers to streamline their research and craft influential articles by facilitating literature reviews (Chen and Li 2023), retrieving and discovering the latest scientific progress, automatically searching for academic papers based on the needs of a given field and retrieving key information tailored to the requirements of different journals.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chen and Li 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs have shown remarkable proficiency in tasks involving clinical health reasoning, real-world medical question-answering, and diagnostic reasoning.

    “LLMs have shown remarkable proficiency in tasks involving clinical health reasoning, realworld medical question-answering, and diagnostic reasoning.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Feng et al. (2022)

      Not checked in the free run.

    • cites Liévin et al. (2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Researchers can gain valuable feedback and suggestions by submitting drafts or manuscripts to LLMs, especially independent researchers without regular peer review access.

    “(4) Researchers are able to gain valuable feedback and suggestions for improvement by submitting drafts or manuscripts, a practice that is especially advantageous for academic researchers who operate independently and do not have regular access to the peer review process.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Castellanos-Gomez 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    GPT-4 presents a transformative opportunity to enhance doctor-patient communication, fostering a better understanding of patients' needs, anxieties, and expectations, thereby improving the overall medical experience.

    “The advanced capabilities of GPT-4 present a transformative opportunity to enhance doctor-patient communication, fostering a better understanding of patients' needs, anxieties, and expectations, thereby improving the overall medical experience (Nashwan et al. 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nashwan et al. 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    GPT-4 exhibited sufficient performance to pass the JNNE, surpassing ChatGPT, suggesting it is suitable for specialized medical training in the Japanese clinical setting.

    “GPT-4 exhibited sufficient performance to pass the JNNE, surpassing ChatGPT, which suggests that GPT-4 is suitable for specialized medical training in the Japanese clinical setting.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kaneda et al. 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs have been utilized in diverse contexts within medical education and language translation, including licensing examinations, generation of multiple-choice questions, evaluation of medical tests, rehabilitation education, pharmacogenomics, and translation of medical imaging reports into layman's terms.

    “The LLMs have been utilized in diverse contexts within medical education and language translation.These include applications in licensing examinations such as the USMLE and the JMLE, the generation of multiple-choice questions, the evaluation of medical tests, educational initiatives in rehabilitation, and pharmacogenomics, as well as the translation of complex medical imaging reports into layman”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Omran et al. 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs can assist experimental design by providing predictions, suggestions, summaries, and interpretations of experimental results, enhancing research efficiency.

    “These models can assist experimental design by providing valuable predictions, suggestions, summaries and interpretations of experimental results, thus enhancing the efficiency of the research process (Huang and Tan 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huang and Tan 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This interdisciplinary approach has the potential to revolutionize fields including media analysis, marketing, and human-computer interaction.

    “This interdisciplinary approach has the potential to revolutionize various fields, including media analysis, marketing, and human-computer interaction (Zhang et al. 2023c).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al. 2023c

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Rehabilitation education plays a vital role in Physical Medicine and Rehabilitation.

    “Rehabilitation education plays a vital role in the field of Physical Medicine and Rehabilitation (Peng et al. 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Peng et al. 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The synergy between LLMs and recognition technologies enables sophisticated multimodal sentiment analysis techniques.

    “The synergy between LLMs and these recognition technologies empowers systems to understand and process multiple forms of data, allowing the development of sophisticated multimodal sentiment analysis techniques (Zhang et al. 2023b).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al. 2023b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some researchers also measure model performance using Medical Concept Coverage to test the importance and negations.

    “Some researchers also measure model performance using Medical Concept Coverage (Chintagunta et al. 2021) to test the importance and negations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Chintagunta et al. 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Commonly used metrics for evaluating LLMM performance include ROUGE, BERTScore, BLEU scores, accuracy, precision, recall, and F1-score for precision evaluation tasks.

    “Commonly used metrics include ROUGE, BERTScore, BLEU scores, accuracy, precision, recall, and F1-score for precision evaluation tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The generation and training of high-quality LLMMs pose significant challenges, necessitating substantial hardware support due to the massive resource consumption and prolonged training times.

    “However, the generation and training of high-quality LLMMs pose significant challenges, necessitating substantial hardware support due to the massive resource consumption and prolonged training times.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Training on extensive internet data may inadvertently integrate misinformation and biased content, potentially leading to significant drawbacks such as the generation of incorrect or fabricated information.

    “However, this training may inadvertently integrate misinformation and biased content, potentially leading to significant drawbacks such as the generation of incorrect or fabricated information (Reddy 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Reddy 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The application of LLMs is currently grappling with a crisis of trustworthiness.

    “The application of LLMs is currently grappling with a crisis of trustworthiness (Liu et al. 2023d).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liu et al. 2023d

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    He et al. found that GPT-4 can significantly support spinal surgeons in diagnosing conditions, managing the perioperative period, conducting scientific research, and enhancing communication with patients, as well as in planning and executing surgical procedures.

    “He et al. (He et al. 2023) conducted a systematic investigation into the use of GPT-4 in lumbar disc herniation surgery and found that GPT-4 can significantly support spinal surgeons in diagnosing conditions, managing the perioperative period, conducting scientific research, and enhancing communication with patients, as well as in planning and executing surgical procedures.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites He et al. 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    By simply concatenating the CFP with the original input prompt, sensitive information in user token embeddings can be eliminated, achieving fairness across a set of sensitive attributes without the need to retrain the entire base model.

    “By simply concatenating the CFP with the original input prompt, sensitive information in user token embeddings can be eliminated, achieving fairness across a set of sensitive attributes without the need to retrain the entire base model.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lin et al. 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMMs have achieved significant advancements in various application scenarios, and the implementation of LLMMs provides critical insights for various parties within healthcare providers and patients.

    “LLMMs have achieved significant advancements in various application scenarios, the implementation of LLMMs provides critical insights for various parties within healthcare providers and patients.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Lee et al. 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The GPT4-RAG model retrieved information in an average of just 1 s and generated results in an average of 15-20 s, while human physicians took an average of 10 min to produce preoperative instructions.

    “The GPT4-RAG model retrieved information in an average of just 1 s and generated results in an average of 15-20 s, while human physicians took an average of 10 min to produce preoperative instructions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ke et al. 2024

      Not checked in the free run.

  • The quote only partly states this claim.

    Interpretationcitation

    The training data of LLMs typically coming from high-income, English-speaking countries can lead to biases in the mechanistic models of health and disease towards understanding this process in high-income countries.

    “This can lead to biases in the mechanistic models of health and disease towards understanding this process in high-income countries.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The training data of LLMs typically comes from institutions in high-income, English-speaking countries, which may severely limit the representativeness of viewpoints from other regions of the world.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The average number of unsafe matches offers a way to measure the effectiveness of the harm or severe consequences of the generated questions.

    “The average number of unsafe matches (Roy et al. 2023) offers a way to measure the effectiveness of the harm or severe consequences of the generated questions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roy et al. 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Li et al. released the largest Chinese medical QA dataset, Huatuo-26 M.

    “Additionally, Li et al. (Li et al. 2023a) released the largest Chinese medical QA dataset, Huatuo-26 M, and Yang et al. (Yang et al. 2023a) pre-trained on this dataset.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Li et al. 2023a

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The complexity of large-scale model architectures has heightened the difficulty in understanding and interpreting these models, particularly within the medical domain where incorrect predictions or biased recommendations could result in substantial harm to patients.

    “Moreover, the complexity of large-scale model architectures has heightened the difficulty in understanding and interpreting these models, particularly within the medical domain where incorrect predictions or biased recommendations could result in substantial harm to patients.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    These strategies enhance the transparency and interpretability of both the LLM's training process and the generated text, thereby reducing errors in medical practitioners and bolstering the credibility of the strategy.

    “These strategies enhance the transparency and interpretability of both the LLM's training process and the generated text, thereby reducing errors in medical practitioners and bolstering the credibility of the strategy (Reddy 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Reddy 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Ke et al. found that the GPT4-RAG model was the most accurate, achieving a performance of 91.4%, which is 5.1% higher than the human-generated answers at 86.3%.

    “The research found that the model augmented by GPT-4 with RAG technology was the most accurate, with the GPT4-RAG model achieving a performance of 91.4%, which is 5.1% higher than the human-generated answers at 86.3%.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ke et al. 2024

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLMs exhibit considerable potential in executing a diverse range of tasks that typically require human capabilities, having been trained on extensive internet data.

    “LLMs exhibit considerable potential in executing a diverse range of tasks that typically require human capabilities, having been trained on extensive internet data (Harrer 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Harrer 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Zhang et al. found that GPT-4 achieved an accuracy rate of 96% in disease classification tasks with a thinking chain and few-shot prompts, and it could be corrected three times for four diagnostic tests.

    “The findings revealed that GPT-4 achieved an accuracy rate of 96% in disease classification tasks with a thinking chain and few-shot prompts, and it could be corrected three times for four diagnostic tests.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al. 2023d

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Interpretationreasoning

    The GPT4-RAG model's accuracy and speed demonstrate its feasibility in the specialized field of healthcare.

    “This demonstrates the feasibility of the GPT4-RAG model in the specialized field of healthcare.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Due to their training on a vast array of internet content, LLMs may inadvertently incorporate biases across the web, posing severe threats in sensitive fields.

    “Due to their training on a vast array of internet content, LLMs may inadvertently incorporate biases (Arora and Arora 2023) (e.g., gender bias, racial bias, geopolitical biases, religious bias, nationality bias, sexual orientation bias, and age bias, etc.) across the web, posing severe threats in sensitive fields (Korngiebel and Mooney 2021).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Arora and Arora 2023 source

      Not checked in the free run.

    • cites Korngiebel and Mooney 2021 source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The findings indicate that LLMs can furnish safe and dependable responses but may not fully address the specific information requirements of a given query.

    “The findings indicate that LLMs can furnish safe and dependable responses but may not fully address the specific information requirements of a given query.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    The digitization of healthcare facilitates the sharing and repurposing of medical data, yet it also increases the risk of critical patient information being compromised.

    “The digitization of healthcare facilitates the sharing and repurposing of medical data, yet it also increases the risk of critical patient information being compromised (Liu et al. 2023c).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Liu et al. 2023c

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The latest achievements have witnessed a technological leap in Chinese question-answering systems, such as Doctor-GLM, Zhongjing, and Huatuo.

    “The latest achievements have witnessed a technological leap in Chinese question-answering systems, such as Doctor-GLM (Xiong et al. 2023), Zhongjing (Yang et al. 2023a), and Huatuo (Li et al. 2023a)”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Xiong et al. 2023

      Not checked in the free run.

    • cites Yang et al. 2023a

      Not checked in the free run.

    • cites Li et al. 2023a

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Given that LLMs retain and train on the information provided, the generated text introduces the potential for plagiarism, which can be illegal and threaten the integrity and copyright of the publication.

    “Given that LLMs retain and train on the information provided, the generated text introduces the potential for plagiarism, which can be illegal and threaten the integrity and copyright of the publication (Nashwan et al. 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Nashwan et al. 2023 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Following adjustments to the output, ChatGPT generated persuasive but unattributed content, complicating the task of identifying plagiarism or original creation.

    “Following adjustments to the output, ChatGPT generated persuasive but unattributed content, complicating the task of identifying plagiarism or original creation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A New York Times report indicates that ChatGPT provided conspiracy theories and misleading responses based on researchers' queries.

    “A New York Times report (Zhang et al. 2023f) indicates that ChatGPT provided conspiracy theories and misleading responses based on researchers' queries.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhang et al. 2023f

      Not checked in the free run.

72 of 72

  • The paper relies on this without support.

    Advanced LLMs do not perform well in Chinese medical question-answering systems because most LLMs are trained on English corpora.

    “However, since most LLMs are trained and learned from English corpora, advanced LLMs do not perform well in Chinese medical question-answering systems.”

  • The paper relies on this without support.

    GPT-4 exhibits superior problem-solving capabilities and an expansive knowledge base compared to ChatGPT.

    “In comparison to ChatGPT, GPT-4 exhibits superior problem-solving capabilities and an expansive knowledge base.”

  • The paper relies on this without support.

    Models such as ChatGPT and GPT-4 can serve as virtual educational companions in rehabilitation.

    “Models such as ChatGPT and GPT-4 can serve as virtual educational companions in rehabilitation.”

  • The paper relies on this without support.

    The LLMs have been trained to recognize and analyze medical images, including x-rays, MRI, and ultrasound.

    “The LLMs have been trained to recognize and analyze medical images, including x-rays, magnetic resonance imaging (MRI), and ultrasound.”

  • The paper relies on this without support.

    These models can interpret features and structures within images, assisting physicians in accurately and rapidly identifying abnormalities, diagnosing diseases, and injuries.

    “These models can interpret features and structures within images, assisting physicians in accurately and rapidly identifying abnormalities, diagnosing diseases, and injuries.”

  • The paper relies on this without support.

    The integration of LLMs can enhance radiologists' interpretive skills, facilitate communication between physicians and patients, and streamline workflow in clinical settings, particularly in hospitals.

    “The integration of LLMs can enhance radiologists' interpretive skills, facilitate communication between physicians and patients, and streamline workflow in clinical settings, particularly in hospitals.”

  • The paper relies on this without support.

    For medical device manufacturers, LLMs can serve as a valuable tool for analyzing user feedback and technical documents, providing insights that inform device development.

    “For medical device manufacturers, LLMs can serve as a valuable tool for analyzing user feedback and technical documents, providing insights that inform device development.”

  • The paper relies on this without support.

    For health insurance companies and providers, LLMs can process and analyze large datasets to identify potential fraud patterns and anomalies, offering tools for insurers to prevent fraud.

    “For health insurance companies and providers, LLMs can process and analyze large datasets to identify potential fraud patterns and anomalies, offering tools for insurers to prevent fraud.”

  • The paper relies on this without support.

    LLMs can address policyholders' queries, provide personalized recommendations to enhance customer experience, and ensure delivery of accurate and valuable information.

    “Additionally, LLMs can address policyholders' queries, provide personalized recommendations to enhance customer experience, and ensure the delivery of accurate and valuable information.”

  • The paper relies on this without support.

    For regulatory bodies, LLMs can strengthen the regulatory review process and assist in detailed scrutiny of medical product submissions.

    “For regulatory bodies, LLMs can strengthen the regulatory review process and assist in the detailed scrutiny of medical product submissions.”

  • The paper relies on this without support.

    Public health authorities can utilize the analytical capabilities of these models to analyze health data, identify disease trends and patterns, and significantly enhance disease surveillance, informing disease control and prevention strategies.

    “Public health authorities can utilize the analytical capabilities of these models to analyze health data, identify disease trends and patterns, and significantly enhance disease surveillance, informing disease control and prevention strategies.”

  • The paper relies on this without support.

    Due to the constrained scope and diversity of clinical trials for novel pharmaceuticals, comprehensive pre-market safety and efficacy assessments are often unattainable.

    “Due to the constrained scope and diversity of clinical trials for novel pharmaceuticals, comprehensive pre-market safety and efficacy assessments are often unattainable.”

  • The paper relies on this without support.

    Virtual mental health assistants are not suitable for use as diagnostic tools because they lack the ability to adhere to essential safety constraints and the professional clinical process knowledge required for accurate diagnosis.

    “However, these assistants are not suitable for use as diagnostic tools because they lack the ability to adhere to essential safety constraints and the professional clinical process knowledge required for accurate diagnosis.”

  • The paper relies on this without support.

    Despite growing interest among scholars and medical professionals in leveraging LLMs in healthcare, the examination and appraisal of their practical application and safety in clinical contexts remain limited.

    “Despite growing interest among scholars and medical professionals in leveraging LLMs in healthcare, the examination and appraisal of their practical application and safety in clinical contexts remain limited.”

  • The paper relies on this without support.

    There is currently a scarcity of data employing large-scale electronic health records (EHR) to assess the performance of LLMs and their utility in providing clinical diagnostic assistance to patients.

    “Despite the exemplary performance of ChatGPT and GPT-4 models in various medical tasks, there is currently a scarcity of data employing large-scale electronic health records (EHR) to assess the performance of LLMs and their utility in providing clinical diagnostic assistance to patients.”

  • The paper relies on this without support.

    The extensive training of LLMMs is typically based on English-related datasets, resulting in a lack of medical knowledge, which can lead to poor performance in tasks such as disease diagnosis, drug recommendation, and clinical decision support.

    “However, the extensive training of LLMMs is typically based on English-related datasets, resulting in a lack of medical knowledge, which can lead to poor performance in tasks such as disease diagnosis, drug recommendation, and clinical decision support.”

  • The paper relies on this without support.

    Existing medical datasets based on English corpora present challenges for conducting accurate experimental analyses of LLMMs on Chinese tasks.

    “Existing medical datasets based on English corpora present challenges for conducting accurate experimental analyses of LLMMs on Chinese tasks.”

  • The paper relies on this without support.

    Medical reports may inadvertently reveal private and demographic details of patient records.

    “Medical reports may inadvertently reveal private and demographic details of patient records.”

  • The paper relies on this without support.

    The training data of LLMs typically comes from institutions in high-income, English-speaking countries, which may severely limit the representativeness of viewpoints from other regions of the world.

    “The training data of LLMs typically comes from institutions in high-income, English-speaking countries, which may severely limit the representativeness of viewpoints from other regions of the world.”

  • The paper relies on this without support.

    The discrepancy between the model's output and the diagnoses of seasoned medical professionals can engender structural bias and inequitable treatment.

    “The discrepancy between the model's output and the diagnoses of seasoned medical professionals can engender structural bias and inequitable treatment.”

  • The paper relies on this without support.

    LLMs suffer from a lack of transparency, hindering their efficiency in assisting humans with complex tasks.

    “Despite their impressive potential in performing various simple tasks, LLMs suffer from a lack of transparency, hindering their efficiency in assisting humans with complex tasks.”

  • The paper relies on this without support.

    A significant challenge for healthcare practitioners is the absence of guidelines for assessing whether LLM outputs align with social norms and regulations.

    “A significant challenge for healthcare practitioners is the absence of guidelines for assessing whether LLM outputs align with social norms and regulations.”

  • The paper relies on this without support.

    As LLMs become more advanced, they are increasingly susceptible to generating harmful or inappropriate content, such as hallucinations, spam, sexist, and racist hate speech.

    “As LLMs become more advanced, they are increasingly susceptible to generating harmful or inappropriate content, such as hallucinations, spam, sexist, and racist hate speech.”

  • The paper relies on this without support.

    These models may also produce responses that sound plausible yet are incorrect or absurd.

    “These models may also produce responses that sound plausible yet are incorrect or absurd.”

  • The paper relies on this without support.

    The Health Insurance Portability and Accountability Act (HIPAA) mandates patient confidentiality and privacy, stipulating that medical records must be sanitized of sensitive information before dissemination.

    “The Health Insurance Portability and Accountability Act (HIPAA) mandates patient confidentiality and privacy, stipulating that medical records must be sanitized of sensitive information before dissemination.”

  • The paper relies on this without support.

    While rule-based and machine learning-based deidentification methods have been extensively implemented in practice, they remain limited in their versatility and effectiveness across diverse scenarios.

    “While rule-based and machine learning-based deidentification methods have been extensively implemented in practice, they remain limited in their versatility and effectiveness across diverse scenarios.”

  • The paper relies on this without support.

    Recent research has revealed a strong correlation between job opportunities and male job seekers, a correlation between negative emotions and the black race, and a correlation between positive emotions and the Asian race.

    “Recent research has revealed a strong correlation between job opportunities and male job seekers, a correlation between negative emotions and the black race, and a correlation between positive emotions and the Asian race.”
27 of 27

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs' ability to write in engaging and accessible ways will make dense medical language or medical services more accessible to the general public.

    “Notwithstanding existing reliability and inaccuracy concerns, it has been argued that LLMs' ability to write in engaging and accessible ways will make dense medical language or medical services more accessible to the general public (78–80).”

    From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [78]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • AKCM results exceed the metric's stated maximum

    Minor

    The stated AKCM formula divides the size of an intersection by the size of a union, so its value cannot exceed 1. Table 6 reports AKCM values of 1.3, 1.12, and 1.27 for the models being compared.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 54

  1. Arasu N et al. (2023) The survey on GPT-3 driven NLP approach for automatic medical documentation. AIP Conf Proc 10(1063/5):0152503
  2. Atallah S et al. (2023) How large language models including generative pre-trained transformer (GPT) 3 and 4 will impact medicine and surgery. Tech Coloproctol 23:1–6
  3. Chen M, Li G (2023) ChatGPT for mechanobiology and medicine: a perspective. Mech Biol Med 1(1):100005
  4. Chen S et al. (2020) Meddialog: a large-scale medical dialogue dataset. Preprint at https://arxiv.org/abs/3:03329
  5. da Mota SLA et al. (2023) Can GPT-4 be a viable alternative for discussing complex cases in digital oral radiology? a critical analysis. Excli J 22:749–751
  6. Davydova V, Tutubalina E (2022) Smm4h 2022 task 2: Dataset for stance and premise detection in tweets about health mandates related to covid-19. In: Proceedings of The seventh workshop on social media mining for health applications, workshop & shared task, pp 216–220
  7. Fatani B (2023) ChatGPT for future medical and dental research. Cureus 15(4):1–5
  8. Feng SY et al. (2022) CHARD: Clinical health-aware reasoning across dimensions for text generation models. Preprint at https://arxiv.org/abs/04191
  9. Fleming S L et al. (2023) Assessing the Potential of USMLE-Like Exam Questions Generated by GPT-4. Preprint at https://medRxiv.org/abs/23288588
  10. Francis S et al. (2023) Understanding the impact of label skewness and optimization on federated learning for text classification. Companion Proc of the ACM Web Conf 2023:1161–1166
  11. George AS et al. (2023) AI-Driven breakthroughs in healthcare: google health’s advances and the future of medical AI. Partn Univ Int Innov J 1(3):256–267
  12. Gilson A et al. (2023) How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 9(1):1–9
  13. Hadi MU et al. (2023) A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Prepr 2023:1–31
  14. Han T et al. (2023) MedAlpaca: an open-source collection of medical conversational AI models and training data. Preprint at https://arxiv.org/abs/08247
  15. Huang J, Tan M (2023) The role of ChatGPT in scientific communication: writing better scientific review articles. AJCR 13(4):1148
  16. Jing C et al. (2022) Supplementing domain knowledge to BERT with semi-structured information of documents. Expert Syst Appl 2023:121054
  17. Kalpakchi D, Boye J (2023) SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish. Preprint at https://arxiv.org/abs/13994
  18. Kaneda Y et al. (2023) Assessing the performance of GPT-3.5 and GPT-4 on the Japanese nursing examination. Cureus 15(8):1–7
  19. Klang E et al. (2023) Utilizing artificial intelligence for crafting medical examinations: a medical education study with GPT-4. Researchsquare 613:423
  20. Krishna K et al. (2020) Generating SOAP notes from doctor-patient conversations using modular summarization techniques. Preprint at https://arxiv.org/abs/01795
  21. Li X et al. (2022a) Is gpt-3 a psychopath? evaluating large language models from a psychological perspective. Preprint at https://arxiv.org/abs/10529
  22. Li J et al. (2023a) Huatuo-26M, a large-scale Chinese medical QA dataset. Preprint at https://arxiv.org/abs/01526
  23. Liévin V et al. (2022) Can large language models reason about medical questions?. Preprint at https://arxiv.org/abs/08143
  24. Liu Y et al. (2023a) Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. Preprint at https://arxiv.org/abs/01852
  25. Liu H et al. (2023b) Evaluating the logical reasoning ability of chatgpt and gpt-4. Preprint at https://arxiv.org/abs/03439
  26. Liu Z et al. (2023c) Deid-gpt: Zero-shot medical text de-identification by gpt-4. Preprint at https://arxiv.org/abs/11032
  27. Liu Y et al. (2023d) Trustworthy LLMs: a survey and guideline for evaluating large language models' alignment. Preprint at https://arxiv.org/abs/05374
  28. Lyu Q et al. (2023) Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: promising results, limitations, and potential. Vis Comput Ind Biomed 6(2023):1–10
  29. Nori H et al. (2023) Capabilities of gpt-4 on medical challenge problems. Preprint at https://arxiv.org/abs/13375
  30. Peng S et al. (2023) AI-ChatGPT/GPT-4: an booster for the development of physical medicine and rehabilitation in the new era! Ann Biomed Eng 62:1–5
  31. Pradeep R et al. (2020) Scientific claim verification with VerT5erini. Preprint at https://arxiv.org/abs/11930
  32. Roy K, Rawte V (2022) TDLR: top semantic-down syntactic language representation. In: NeurIPS'22 workshop on all things attention: bridging different perspectives on attention
  33. Sallam M (2023b) The utility of ChatGPT as an example of large language models in healthcare education, research and practice: systematic review on the future perspectives and potential limitations. Healthcare 11:887 · DOI · Paper page
  34. Schloss B, Konam S (2020) Towards an automated SOAP note: classifying utterances from medical conversations. In: machine learning for healthcare conference, pp 610–631
  35. Singhal K et al. (2023b) Towards expert-level medical question answering with large language models. Preprint at https://arxiv.org/abs/09617
  36. Thawkar O et al. (2023) Xraygpt: Chest radiographs summarization using medical vision-language models. Preprint at https://arxiv.org/abs/07971
  37. Thirunavukarasu AJ et al. (2023) Large language models in medicine. Nat Med 8:1–11
  38. Toma A et al. (2023) Clinical Camel: An open-source expert-level medical language model with dialogue-based knowledge encoding. Preprint at https://arxiv.org/abs/12031
  39. Waisberg E et al. (2023) GPT-4: a new era of artificial intelligence in medicine. Ir J Med Sci 51(2023):1645–1653
  40. Wang Y et al. (2023a) Are large language models ready for healthcare? a comparative study on clinical language understanding. Proc Mach Learn Res 219:1–24
  41. Wang Z et al. (2023b) Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today. Preprint at https://arxiv.org/abs/01499
  42. Wang H et al. (2023c) Huatuo: Tuning llama model with chinese medical knowledge. Preprint at https://arxiv.org/abs/06975
  43. Wang Y et al. (2023d) Are large language models ready for healthcare? A comparative study on clinical language understanding. Preprint at https://arxiv.org/abs/05368
  44. Wei C et al. (2023) An overview on language models: recent developments and outlook. Preprint at https://arxiv.org/abs/05759
  45. Wu C et al. (2023) Pmc-llama: further finetuning llama on medical papers. Preprint at https://arxiv.org/abs/14454
  46. Xiong H et al. (2023) Doctorglm: fine-tuning your chinese doctor is not a herculean task. Preprint at https://arxiv.org/abs/01097
  47. Yadav S et al. (2023) Towards understanding consumer healthcare questions on the web with semantically enhanced contrastive learning. Proc of the ACM Web Conf 2023:1773–1783
  48. Yadav S et al. (2022a) Chq-summ: a dataset for consumer healthcare question summarization. Preprint at https://arxiv.org/abs/06581
  49. Yang J et al. (2023) The Impact of ChatGPT and LLMs on medical imaging stakeholders: perspectives and use cases. MetaRadiology 1:100007
  50. Yang X et al. (2020) COVID-CT-dataset: a CT scan dataset about COVID-19. Preprint at https://arxiv.org/abs/13865
  51. Yang S et al. (2023a) Zhongjing: enhancing the Chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. Preprint at https://arxiv.org/abs/03549
  52. Zhang J et al. (2022) Fengshenbang 1.0: being the foundation of chinese cognitive intelligence. Preprint at https://arxiv.org/abs/02970
  53. Zhang J et al. (2023d) The potential and pitfalls of using a large language model such as ChatGPT or GPT-4 as a clinical assistant. Preprint at https://arxiv.org/abs/08152
  54. Zhu W et al. (2019) Panlp at mediqa 2019: Pre-trained language models, transfer learning and knowlsedge distillation. ACL Anthology. https://doi.org/10.18653/v1/W19-5039 · DOI

Cited by

Sorted by citations

Related

Authors
Wang, D., & Zhang, S.
Affiliations
Taizhou University (CN)
Corresponding author
Zhang, Shiqing
ORCID
Venue
Artificial Intelligence Review
Publisher
Springer Science and Business Media LLC
Published
2024-09-20
Volume
57
Issue
11
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
DOI
10.1007/s10462-024-10921-0
OpenAlex
W4402692877
Semantic Scholar
70b83014b3267ddcb04d65af0f7975920d46a87a
Cited by
246
References
122
Citations by year
2026: 124 · 2025: 119 · 2024: 3
FWCI
23.05
Type
journal article
Language
en
Open access
Licence
Locations
Artificial Intelligence Review (publishedVersion, cc-by-nc-nd)
Funders
  • Natural Science Foundation of Zhejiang Province (LQ24F020014)
  • The Project of Zhejiang Province's Educational Science Planning (2024SCG176)
  • National Natural Science Foundation of China (62276180)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, medical applications, medical question answering, medical education, clinical decision support, clinical reasoning, data security, algorithmic bias, de-identification, accountability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review