CoChat
Sign Up
Cited by 0Artificial Intelligence in Healthcare and EducationPharmacy and Medical PracticesElectronic Health Records Systems

Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations

Open access
Yakob, Reginald Amin, Bamgboje-Ayodele, Adeola, Collins, Jack C., Aslani, Parisa·2026·Research in Social and Administrative Pharmacy
DOI 10.1016/j.sapharm.2026.09.007
Open this paper in CoChat — free accountOpen PDF

Abstract

Introduction Large language models (LLMs) represent an advancement in artificial intelligence technology with the potential to support patient care tasks. However, reviews of their evaluations have often excluded pharmacy-specific applications. Objective To characterize published evaluations of LLMs for pharmacist-led direct patient care tasks, synthesize reported LLM performance, and identify opportunities and challenges for workflow integration. Methods Searches were conducted in Embase, MEDLINE, PubMed, CINAHL, Scopus, and Web of Science for studies published between January 2014 and April 2026 using the concepts "pharmacists," "pharmacy," and "LLMs." Original peer-reviewed studies evaluating LLMs for tasks aligned with pharmacist-led direct patient care were eligible. Covidence was used for screening studies; data were extracted and narratively synthesized. Results Of 4,925 identified records, 71 studies were included in the synthesis. Pharmacist-led direct patient care tasks were identified for which LLMs were evaluated; accuracy was the most assessed metric across studies (n=53), and ChatGPT variants were the most widely evaluated LLMs (n=63). Evaluation and reporting approaches varied across studies, limiting cross-study comparisons; however, studies generally reported higher performance for advanced LLM versions than for earlier ones. Challenges to workflow integration include confabulation, costs, and privacy risks. Conclusion Evaluations of LLMs largely focused on ChatGPT models and accuracy-based outcomes; however, heterogeneity in evaluation and reporting limited cross-study comparisons. Advanced LLMs generally outperformed earlier versions, suggesting increasing potential to support pharmacist-led direct patient care tasks. Future evaluations should adopt standardized approaches and include human factors outcomes relevant to workflow integration.

AI snapshot

Written from the paper’s full text.

TL;DR

This rapid review synthesized 71 published evaluations of large language models (LLMs) for pharmacist-led direct patient care tasks, finding that ChatGPT variants were most evaluated and accuracy was the most assessed metric. Advanced LLMs generally outperformed earlier versions, but heterogeneous evaluation and reporting limited cross-study comparisons, and real-world workflow integration evidence remained scarce.

Key findings

  • Of 4,925 identified records, 71 studies were included; all were published between 2023 and 2026. The largest proportion of studies came from authors affiliated with institutions in the United States (n = 24, 33.8%), followed by China (n = 5, 7.0%).
  • ChatGPT, including GPT variants, was the most widely evaluated LLM (n = 63), followed by Gemini (n = 18), Copilot (n = 11), Claude (n = 10), DeepSeek (n = 7), and Llama (n = 4). Accuracy was the most commonly evaluated performance variable (n = 53, 74.6%).
  • Thirteen pharmacist-led direct patient care task groups were identified. Advanced LLMs generally achieved higher reported performance than earlier versions; for drug-drug interaction prediction, one study reported Bing AI accuracy of 0.788, Bard 0.613, ChatGPT-4 0.592, and ChatGPT-3.5 0.469. For long-acting injectable antipsychotic dosing, one study reported descriptive accuracy of 95% for OpenEvidence, 85% for Perplexity, 62.5% for Copilot, and 40% for ChatGPT.
  • Only one study pilot-tested real-world integration of a GPT-4 LLM into clinical pharmacists’ workflow; no significant time savings were observed using electronic health record log data, but pharmacists reported perceived time savings, reduced cognitive task load, and lower work exhaustion, with a 44.4% mean utilization rate of LLM-generated drafts. Challenges to integration included confabulation, costs, and privacy risks.

Methods

  • Rapid review following Cochrane rapid review methods guidance, reported per PRISMA 2020, with a protocol published in PROSPERO (CRD42024607513). Searches were conducted in CINAHL, Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science for studies published between January 2014 and April 21, 2026, with additional reference-list and citation screening.
  • Eligible studies were original peer-reviewed articles evaluating AI technologies classified as LLMs and reporting outcomes for tasks aligned with pharmacist-led direct patient care responsibilities. Non-primary research, non-clinical LLM use, and technical development/engineering studies were excluded.
  • Covidence was used for screening; 30% of titles/abstracts were dual-screened with 100% reviewer agreement, then remaining records were single-screened. Full-text eligibility decisions were all checked by a second reviewer. A single reviewer extracted data, with three authors independently verifying separate 10% samples. Data were narratively synthesized, and task groups were mapped to Good Pharmacy Practice and ISCO-08 frameworks. No formal risk-of-bias assessment was conducted.

Limitations

  • The paper states that LLM technologies are rapidly advancing, so relevant studies published after the search date may have been missed, and only English-language publications were included.
  • The paper states that a formal risk-of-bias assessment was not conducted because the objective was to map and characterize the literature rather than estimate clinical effectiveness through meta-analysis; recommendations derive from narrative synthesis rather than pooled quantitative effect estimates.
  • Evident from the text: heterogeneity in LLM versions, prompting methods, evaluation methods, outcome measures, and reporting practices limited cross-study comparisons; accuracy was operationalized inconsistently across studies, and most evidence came from simulated evaluations rather than real-world workflow integration.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.sapharm.2026.09.007

Cites

113 of 113 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

60 of 94 cited sources checked · 15 supported · 8 flagged · accuracy not yet measured

  • Findingcitation

    Only one study pilot-tested the real-world implementation of a GPT-4 LLM integrated into clinical pharmacists' workflow to assist in drafting responses to incoming electronic patient inquiries.

    “Only one study pilot-tested the real-world implementation of a GPT-4 LLM integrated into clinical pharmacists' workflow to assist in drafting responses to incoming electronic patient inquiries.53”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Study design describes manually entering a predefined set of questions into ChatGPT, not integrating a model into pharmacists’ workflow to draft replies to incoming patient inquiries. The Discussion identifies the model as ChatGPT 3.5, not GPT-4, and says the question set may not reflect real-world practice. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Findingcitation

    In one real-world integration study, GPT-4 failed to generate draft responses for 5% of patient inquiries because of token and computational limits.

    “In one study that integrated an LLM into real-world clinician workflows, GPT-4 failed to generate draft responses for 5% of patient inquiries because of token and computational limits.53”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Study design describes manually entering predefined questions into ChatGPT 3.5, not integrating GPT-4 into clinician workflows. The Results report accuracy across 80 questions, rather than a 5% failure to generate drafts because of token or computational limits. (LLM, conf=0.89)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Findingcitation

    A prospective study found no statistically significant difference in mean accuracy between medication reviews performed by pharmacists assisted by LLMs and those performed by pharmacists unassisted by LLMs.

    “they found no statistically significant difference in mean accuracy between reviews performed by pharmacists assisted by LLMs and those performed by pharmacists unassisted by LLMs.68”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [68] Cornelison, 2025, Accuracy and safety of ChatGPT-3.5 in assessing over-the-counter medication use during pregnancy: a descriptive comparative study, Pharmacy source

      The Limitations section describes this as a cross-sectional assessment, not a prospective study. The Methods and Results describe reviewers scoring ChatGPT-3.5 responses against UpToDate; they do not compare medication reviews by LLM-assisted and unassisted pharmacists or report a statistical test of mean accuracy. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Findingcitation

    In the only study that integrated LLMs into real-world patient care, the models were reported to comply with HIPAA.

    “In the only study that integrated LLMs into realworld patient care, the models were reported to comply with the Health Insurance Portability and Accountability Act (HIPAA).53”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Study design section describes manually entering a standardized set of questions into ChatGPT, rather than integrating a model into real-world patient care. The Discussion likewise describes real-time patient data as a possible future improvement. No passage found reports HIPAA compliance. (LLM, conf=0.82)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    Previous reviews of LLM use within patient care have often overlooked pharmacy-specific applications.

    “However, previous reviews of LLM use within patient care have often overlooked pharmacyspecific applications.”
    • Also stated in: § 1. Introduction · plain
      “A systematic review of LLMs in patient care identified significant limitations but excluded pharmacy studies, 16 while another review on applications of LLMs in healthcare focused on medical practice specialties.14 A scoping review in pharmacy practice focused solely on ChatGPT, omitting other LLMs.20”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [16] Busch, 2025, Current applications and challenges in large language models for patient care: a systematic review, Commun Med source

      The Results (Screening results; page unavailable) say this review excluded a study focused solely on pharmacological treatment, supporting a narrower point about its scope. The paper does not establish that previous reviews often overlooked pharmacy-specific applications. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [14] Bedi, 2025, Testing and evaluation of health care applications of large Language models: a systematic review, JAMA source
    • cites [20] Lima, 2024, ChatGPT in pharmacy practice: disruptive or destructive innovation? A scoping review, Sci Pharm source

      The Introduction (page unavailable) says that no prior scoping review had synthesized studies of ChatGPT in pharmacy practice, supporting a gap in pharmacy-specific review coverage. It does not establish that reviews of LLM use in patient care often overlooked pharmacy applications. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • May combine two claims. 45
  • Classified as probably not a substantial claim. 31

  • May combine two claims.

    Findingcitation

    One study found that ChatGPT-3.5 with role-play prompts scored higher accuracy (86.67%) and provided more citations (n=144) than direct prompts (accuracy=77.78%, citations n=111).

    “One study using ChatGPT-3.5found that role-play prompts scored higher accuracy (86.67%) and provided more citations (n = 144) than direct prompts (accuracy = 77.78%,citations n = 111).70”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [70] Domián, 2025, Comparative evaluation of artificial intelligence platforms and drug interaction screening databases using real-world patient data, Explor Res Clin Soc Pharm source

      The Methods (Artificial intelligence platforms; Prompt development, page unavailable) say this study used GPT-4 and identical prompts across platforms, not ChatGPT-3.5 with role-play versus direct prompts. The Results (Performance evaluation of AI platforms, page unavailable) report drug-interaction screening metrics, not the claimed accuracy or citation counts. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    Single studies reported high accuracy for specialized reasoning models (OpenAI o3, OpenAI o4-mini) in pharmacotherapy management and a customized ChatGPT-4 model for patient education and counseling.

    “Furthermore, single studies reported high accuracy for specialized reasoning models, 100 OpenAI o3, 33 and OpenAI o4-mini, 60 in pharmacotherapy management and a customized ChatGPT-4 model for patient education and counseling.59”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [100] 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [33] Hsu, 2023, Examining real-world medication consultations and drug-herb interactions: ChatGPT performance evaluation, JMIR Med Educ source

      The citation is attached to the customized ChatGPT-4 claim, but the Limitations section states that this study tested GPT-3.5. The Results section reports 51 of 80 medication-consultation responses (64%) as appropriate, not high accuracy for a customized GPT-4 patient-education model. No page numbers are available in our copy. (LLM, conf=0.89)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [60] van Nuland, 2024, Assessing accuracy of ChatGPT in response to questions from day to day pharmaceutical care in hospitals, Explor Res Clin Soc Pharm source

      The Methods and Results examine ChatGPT responses to routine hospital-pharmacy questions, not OpenAI o3, o4-mini, or a customized ChatGPT-4 model. The Discussion mentions drug counselling and patient education but does not report the claimed high accuracy for those models (searched for the model names and the claimed applications). (LLM, conf=0.77)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    In the only real-world workflow study, GPT-3.5 was used to categorize patient inquiries and GPT-4 drafted responses for clinician review.

    “In that study, GPT-3.5 was used to categorize patient inquiries and GPT-4 drafted responses for clinician review.53”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Study design (page not provided) describes manually entering a predefined set of questions into ChatGPT and comparing its answers with standard responses. It says the model tested was ChatGPT 3.5; the Discussion likewise says the study focused on that version. This is contrary to the attributed GPT-3.5 categorization and GPT-4 drafting workflow for clinician review. (LLM, conf=0.89)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    Although ChatGPT was the most commonly evaluated LLM, some studies did not find a performance advantage over other LLMs.

    “Although ChatGPT was the most commonly evaluated LLM, some studies did not find a performance advantage over other LLMs.31,83”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [31] Al-Ashwal, 2023, Evaluating the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, bing AI, and bard against conventional drug-drug interactions clinical tools, Drug Healthc Patient Saf source

      The Discussion reports that ChatGPT models performed worse than the other AI tools on most evaluated metrics, supporting the claim that ChatGPT did not have a performance advantage in this study. The paper does not establish that ChatGPT was the most commonly evaluated LLM across studies. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [83] Yassin, 2025, Evaluating a generative artificial intelligence accuracy in providing medication instructions from smartphone images, J Am Pharm Assoc source

      The Discussion (page unavailable) says the authors did not know how other generative AI tools performed and could not complete their planned comparison with Google Bard. This paper therefore did not find that ChatGPT lacked a performance advantage over other LLMs; it made no such comparison. It also does not establish that ChatGPT was the most commonly evaluated LLM. (LLM, conf=0.78)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Search strategies were developed with an academic librarian and guided by the Peer Review of Electronic Search Strategies checklist.

    “The search strategies were developed in collaboration with an academic librarian and guided by the Peer Review of Electronic Search Strategies checklist, 24”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [24] McGowan, 2016, PRESS peer review of electronic search strategies: 2015 guideline statement, J Clin Epidemiol source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The review followed Cochrane rapid review methods guidance and reported results in accordance with PRISMA 2020 standards.

    “The review followed the Cochrane rapid review methods guidance, 22 with results reported in accordance with PRISMA 2020 standards.23”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [22] Garritty, 2024, Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness, bmj source
    • cites [23] Page, 2021, The PRISMA 2020 statement: an updated guideline for reporting systematic reviews, BMJ source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The review dual-screened 30% of titles and abstracts, with 100% reviewer agreement, before single-reviewer screening of remaining records.

    “we dual-screened 30% of titles and abstracts to assess reviewer agreement before proceeding with single-reviewer screening of the remaining records.”
    • Also stated in: § 2.4 Study selection and data extraction · plain
      “Reviewers agreed on 100% of the sampled records at the title and abstract screening stage”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [22] Garritty, 2024, Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness, bmj source

  • May combine two claims.

    Findingcitation

    ChatGPT-3.5 performed adequately in providing medication information but had poor performance in recognizing adverse drug reactions and on high-complexity tasks.

    “In other studies, ChatGPT-3.5 was reported to have performed adequately in providing medication information, 32,55 but had poor performance in J o u r n a l P r e -p r o o f recognizing adverse drug reactions and similar poor performance on high-complexity tasks.55,91,92”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [32] Al-Dujaili, 2023, Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: a preliminary analysis with clinical pharmacy experts worldwide, Res Soc Adm Pharm RSAP source
    • cites [55] Sheikh, 2024, Identification of kidney-related medications using AI from self-captured pill images, Ren Fail source

      The Methods and Results assess ChatGPT-4’s identification of medications from pill images, not ChatGPT-3.5’s provision of medication information. A search for adverse drug reactions and high-complexity tasks found no passage assessing either; the Discussion’s comparison concerns ChatGPT-3 and ChatGPT-4 instead. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites [91] Mohammed, 2026, Comparative performance of large language models and Drugs.com versus Lexicomp for antiseizure medication drug–drug interactions: a cross-sectional study with iterative prompting analysis, Explor Res Clin Soc Pharm source

      The Methods (Study design and Data collection procedure; page unavailable) specify that the study evaluated ChatGPT-4, not ChatGPT-3.5. The Results report performance on antiseizure-medication drug interactions, but no passage found assesses ChatGPT-3.5’s provision of medication information, recognition of adverse drug reactions, or performance on high-complexity tasks. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    In comparing drug-drug interaction prediction, Bing AI had the highest accuracy (0.788), followed by Bard (0.613), ChatGPT-4 (0.592), and ChatGPT-3.5 (0.469).

    “When comparing different LLMs' ability to predict drug-drug interactions, one study found that Bing AI had the highest accuracy (0.788), followed by Bard (0.613), ChatGPT-4 (0.592), and ChatGPT-3.5 (0.469).31”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [31] Al-Ashwal, 2023, Evaluating the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, bing AI, and bard against conventional drug-drug interactions clinical tools, Drug Healthc Patient Saf source

  • May combine two claims.

    Findingcitation

    In that real-world study, pharmacists reported perceived time savings, reduced cognitive task load, and lower work exhaustion after the pilot.

    “pharmacists reported perceived time savings, reduced cognitive task load, and lower work exhaustion after the pilot.53”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Study design describes testing ChatGPT on pharmacy questions, and the Results report response accuracy and a correlation between perceived benefits and concerns (page numbers unavailable). Neither section reports a pilot in which pharmacists assessed time savings, cognitive task load, or work exhaustion. Searches for those outcomes and for a pharmacist pilot found no supporting passage. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    Another study noted that ChatGPT-4 was more accurate than ChatGPT-3.5 but required paid access and had token limitations, whereas ChatGPT-3.5 was available free of charge at the time of evaluation.

    “Another study noted that although ChatGPT-4 was more accurate than ChatGPT-3.5, it required paid access and had token limitations, whereas ChatGPT-3.5 was available free of charge at the time of evaluation.38”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [38] Albogami, 2024, Safety and quality of AI chatbots for drug-related inquiries: a real-world comparison with licensed pharmacists, Digit Health source

      The Results report that GPT-4 gave completely accurate answers to 64.3% of drug-related inquiries, versus 45% for GPT-3.5. The cited paper's Methods describe the chatbots used, but the passages found do not substantiate paid access or token limitations for GPT-4, or free access to GPT-3.5 at the time of evaluation. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLMs process text as tokens, and token use can affect both usage costs and computational requirements.

    “LLMs process text as tokens, and token use can affect both usage costs and computational requirements.[103][104][105]”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [103] Bergemann, 2025, The economics of large language models: token allocation, fine-tuning, and optimal pricing source
    • cites [104]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [105]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Specialized proprietary LLMs (e.g., OpenEvidence and UpToDate Expert AI) are designed to provide responses grounded in medical literature and are seeing increasing adoption in medical practice among health professionals and medical students.

    “Also, specialized proprietary LLMs (e.g., OpenEvidence and UpToDate Expert AI) are designed to provide responses grounded in medical literature, 115 and are seeing increasing adoption in medical practice, 9 among health professionals and medical students.115,116”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [115] Bergsbaken, 2026, Prescribing caution: a critique of OpenEvidence to answer medication‐related questions, JACCP J Am Coll Clin Pharm source
    • cites [9] Vishwanath, 2026, General-purpose large language models outperform specialized clinical AI tools on medical benchmarks, Nat Med source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [116] Barr, 2026, Medical students' use of large language models: a national survey, Int J Med Inf source

  • May combine two claims.

    Premisecitation

    Some general-purpose LLMs, including GPT-4-based models, can be customized through methods such as prompt engineering and in-context learning and augmented with domain-specific external knowledge sources, enabling responses grounded in relevant databases or documents.

    “Some general-purpose LLMs, including GPT-4-based models, can be customized through methods such as prompt engineering and in-context learning (e.g., PharmacyGPT), 10 and augmented with domain-specific external knowledge sources, enabling responses grounded in relevant databases or documents.114”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [10] Liu, 2025, PharmacyGPT: exploration of artificial intelligence for medication management in the intensive care unit, BMC Med Inform Decis Mak source

      Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [114] Amugongo, 2025, Retrieval augmented generation for large language models in healthcare: a systematic review, PLOS Digit Health source

  • Classified as probably not a substantial claim.

    Interpretationcitation

    Omitting details on LLM use limits experimental transparency, replicability, and interpretation of evaluation findings.

    “Omitting details on LLM use also limits experimental transparency, replicability, and interpretation of evaluation findings.111”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [111] Gallifant, 2025, The TRIPOD-LLM reporting guideline for studies using large language models, Nat Med source

  • May combine two claims.

    Findingcitation

    Despite LLMs' potential, notable challenges include hallucination risk, response sensitivity to prompting method, cost, and data privacy.

    “Despite LLMs' potential to support pharmacist-led direct patient care tasks, notable challenges include hallucination risk, response sensitivity to prompting method, cost, and data privacy.16,53,92”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [16] Busch, 2025, Current applications and challenges in large language models for patient care: a systematic review, Commun Med source

      The Results (Limitations of large language models; page unavailable) report hallucinations, dependence of output quality on prompting/input, and limitations involving storage or processing of sensitive health information. The Discussion (page unavailable) also identifies privacy breaches as a concern. Our copy does not identify cost as a challenge, so it supports only part of the attributed list. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Results (page not provided) document incorrect ChatGPT answers, and the Discussion warns that responses may be misleading or harmful, supporting a concern about answer reliability. The Study design (page not provided) repeats identical questions rather than comparing prompting methods; the passages do not substantiate sensitivity to prompting method, cost, or data privacy as challenges. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [92] Ready, 2026, Assessing ChatGPT's capability in understanding and reporting antiretroviral therapy drug–drug interactions (ACCURATE-DDI), AIDS source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    ChatGPT's popularity may stem from its early public availability and the substantial media attention it received.

    “ChatGPT's popularity may stem from its early public availability and the substantial media attention it received.112,113”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [112] Ryazanov, 2025, How ChatGPT changed the media's narratives on AI: a semi-automated narrative analysis through frame semantics, Minds Mach source

      The Introduction (page unavailable) describes a surge in public interest immediately after ChatGPT’s launch, and §4.1 General Statistics (page unavailable) documents substantially increased media coverage and frequent mentions of ChatGPT. The paper does not establish that early public availability or media attention caused ChatGPT’s popularity; it suggests that rising public interest may have been reflected in media coverage. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites [113] Leiter, 2024, Chatgpt: a meta-analysis after 2.5 months, Mach Learn Appl source

  • May combine two claims.

    Premisecitation

    LLM version, prompting approach, session handling, and parameter settings can influence model outputs.

    “These details are important because LLM version, prompting approach, session handling, and parameter settings can influence model outputs.106,111”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [106] 2025, Reporting guideline for chatbot health advice studies: the Chatbot Assessment reporting tool (CHART) statement, BMJ Med

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [111] Gallifant, 2025, The TRIPOD-LLM reporting guideline for studies using large language models, Nat Med source

      The Discussion says prompting approaches can considerably influence LLM performance and calls for reporting model names and versions. The Rationale for TRIPOD-LLM says choices of fine-tuning hyperparameters and prompting affect model reliability. These passages support much of the claim, but our copy does not establish that session handling influences outputs. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A survey of pharmacists in Jordan found that 77.2% agreed that pharmacists could benefit from using ChatGPT.

    “a survey of pharmacists in Jordan found that 77.2% agreed that pharmacists could benefit from using ChatGPT.118”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [118] Abu Hammour, 2023, ChatGPT in pharmacy practice: a cross-sectional exploration of Jordanian pharmacists' perception, practice, and concerns, J Pharm Policy Pract source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    This finding aligns with a previous review of ChatGPT in pharmacy practice, which highlighted a lack of standardized evaluation methods, and with broader healthcare reviews calling for standardized LLM evaluation approaches.

    “This finding aligns with a previous review of ChatGPT in pharmacy practice, which highlighted a lack of standardized evaluation methods, 20 and with broader healthcare reviews calling for standardized LLM evaluation approaches.14,15”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [20] Lima, 2024, ChatGPT in pharmacy practice: disruptive or destructive innovation? A scoping review, Sci Pharm source
    • cites [14] Bedi, 2025, Testing and evaluation of health care applications of large Language models: a systematic review, JAMA source
    • cites [15] Hobensack, 2024, A rapid review on current and potential uses of large language models in nursing, Int J Nurs Stud source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Some studies evaluated LLMs in multiple languages, highlighting a potential role for multilingual LLMs in supporting knowledge transfer and improving access to healthcare information across diverse linguistic and cultural contexts.

    “some studies evaluated LLMs in multiple languages, highlighting a potential role for multilingual LLMs in supporting J o u r n a l P r e -p r o o f knowledge transfer and improving access to healthcare information across diverse linguistic and cultural contexts.2”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [2] Rodriguez, 2024, Leveraging large language models to foster equity in healthcare, J Am Med Inform Assoc source

      The Community health section describes potential LLM support for patient messages in multiple languages, and the section on retrieval augmentation and fine-tuning discusses culturally and linguistically diverse health information. These passages support the proposed role in improving access, but the paper does not report that studies evaluated LLMs in multiple languages or establish the claimed knowledge-transfer role. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLM response reproducibility refers to the consistency of its outputs to the same query and has been shown to differ on repeat assessment.

    “LLM response reproducibility refers to the consistency of its outputs to the same query and has been shown to differ on repeat assessment.58,59,66”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [58] Stanceski, 2024, The quality and safety of using generative AI to produce patient-centred discharge instructions, Npj Digit Med source

      The Methods section (Analysis and outcome measures; page unavailable) reports agreement between investigators scoring responses, not consistency between LLM outputs to the same query. Searches for reproducibility, repeated queries, and repeated generation found no passage assessing whether outputs differed on repeat assessment. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites [59] van Nuland, 2024, Performance of ChatGPT on factual knowledge questions regarding clinical pharmacy, J Clin Pharmacol source
    • cites [66] Chase, 2025, Evaluation of large language models' ability to identify clinically relevant drug-drug interactions and generate high-quality clinical pharmacotherapy recommendations, Am J Health Syst Pharm source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    These findings are consistent with broader trends in LLM development, where model performance has improved with increases in model scale, quality of training data, or information availability.

    “These findings are consistent with broader trends in LLM development, where model performance has improved with increases in model scale, quality of training data, or information availability.117”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [117] Xiao, 2025, Densing law of LLMs, Nat Mach Intell source

      The Abstract says LLM performance improves as model parameters and the amount of training data increase. It does not establish that improvements result from higher-quality training data or greater information availability; those are distinct from data volume. Page numbers are not available in our copy. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    The prompts used to query the LLM significantly affect response quality and LLM usage costs.

    “The prompts used to query the LLM also significantly affect response quality and LLM usage costs.53,103,119”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [53] Salama, 2024, The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy, Pharmacia source

      The Results (page unavailable) report different ChatGPT accuracy rates across categories of pharmacy questions, supporting the narrower point that response quality varies with the questions asked. The Study design says the questions used a standardized format; the paper does not establish a significant effect of prompt wording or assess LLM usage costs. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
    • cites [103] Bergemann, 2025, The economics of large language models: token allocation, fine-tuning, and optimal pricing source
    • cites [119] Gao, 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Prompt engineering has emerged as a key strategy for engaging LLMs and has been shown to improve LLM responses.

    “Prompt engineering has emerged as a key strategy for engaging LLMs, involving structuring queries with clear context, instructions, and expected outputs, 119 and has been shown to improve LLM responses.120”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [119] Gao, 2023 source
    • cites [120] Yan, 2024, Prompt engineering on leveraging large language models in generating response to InBasket messages, J Am Med Inform Assoc source

  • May combine two claims.

    Findingown data

    The date of LLM querying was reported in 39 studies (54.9%), session handling in 37 studies (52.1%), and response generation parameter settings in 10 studies (14.1%).

    “The date of LLM querying was reported in 39 studies (54.9%), how separate J o u r n a l P r e -p r o o f query sessions were handled, such as disabling the LLM memory or starting new chat sessions for new queries, was reported in 37 studies (52.1%), and the LLM's response generation parameter settings were reported in 10 studies (14.1%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The database search yielded 4,925 records.

    “The database search yielded 4,925 records.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A smaller number employed role-play prompts (n=9, 12.7%).

    “A smaller number employed role-play prompts (n = 9, 12.7%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Challenges to workflow integration include confabulation, costs, and privacy risks.

    “Challenges to workflow integration include confabulation, costs, and privacy risks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Other commonly evaluated LLMs included Gemini (n=18), Copilot (n=11), Claude (n=10), DeepSeek (n=7), and Llama (n=4).

    “Other commonly evaluated LLMs included Gemini (n = 18), 34,[40][41][42][43]48,50,52,63,68,69,71,80,81,83,84,93,95 Copilot (n = 11), 34,35,40,42,48,50,71,80,87,93,95 Claude (n = 10), [40][41][42][43][44]63,68,80,81,83 DeepSeek (n = 7), 49,63,64,69,96,97,99 and Llama (n = 4), 44,80,83,93”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Jordan, the Netherlands, and Saudi Arabia each had four studies (5.6%).

    “Other represented countries included Jordan, the Netherlands, and Saudi Arabia, with four studies each (5.6%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    An additional 4 records were identified through screening reference lists and citations of eligible studies and related reviews.

    “An additional 4 records were identified through screening the reference lists and citations of eligible studies and related reviews.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Publication counts increased from 7 studies (9.9%) in 2023 to 24 (33.8%) in 2024 and 24 (33.8%) in 2025, with 16 (22.5%) identified by the April 2026 search end date.

    “Publication counts increased from 7 studies (9.9%) in 2023, 31,32,54,56,65,66,70 to 24 (33.8%) in 2024, 30,36,38,39,45,47,51,53,55,58,61,62,67,72,74,75,[77][78][79]83,85,[90][91][92] and 24 (33.8%) in 2025, 29,33,35,40,42- 44,46,48,50,57,59,63,68,69,71,81,82,84,86,89,93,94,98 with 16 (22.5%)studies, 34,37,41,49,52,60,64,73,76,80,87,88,[95][96][97]99 identified by the April 2026 search end date.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Evaluation metrics varied across task types; factual knowledge questions were commonly assessed using percentage accuracy scores, whereas drug-drug interaction checking was more often evaluated using binary classification metrics.

    “Evaluation metrics varied across task types.For example, factual knowledge questions were commonly assessed using percentage accuracy scores, 78,90 whereas checking for drug-drug interactions was more often evaluated using binary classification metrics.31,37,61”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Evaluation and reporting approaches varied across studies, limiting cross-study comparisons.

    “Evaluation and reporting approaches varied across studies, limiting crossstudy comparisons”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The largest proportion of included studies originated from the United States (n=24, 33.8%), followed by China (n=5, 7.0%).

    “the largest proportion originated from the United States, based on the countries of the authors' affiliated institutions (n = 24, 33.8%), followed by China (n = 5, 7.0%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Other reported limitations included restricted evaluation of LLM capabilities, such as querying in a single language, and limited generalizability due to study setting.

    “Other reported limitations included restricted evaluation of LLM capabilities, such as querying in a single language, and limited generalizability due to the study setting”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    One study reported poor performance for GPT-4 in adverse drug causality tasks, with acceptable performance in providing medication information.

    “One study reported poor performance for GPT-4 in adverse drug causality tasks, with acceptable performance in providing medication information.65”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [65] Campbell, 2025, Accuracy and teachability of artificial intelligence chatbots in solving pharmaceutical calculations: a descriptive study, Int J Clin Pharm source

  • May combine two claims.

    Findingcitation

    One study tested GPT-3.5 on 3 occasions over 5 weeks and reported reproducibility rates of 79.2%, 87.5%, and 83.3%, with statistically significant agreement (P < 0.05).

    “One study tested GPT-3.5 on 3 occasions over 5 weeks and reported reproducibility rates of 79.2%, 87.5%, and 83.3%, with statistically significant agreement (P < 0.05).32”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] Al-Dujaili, 2023, Assessing the accuracy and consistency of ChatGPT in clinical pharmacy management: a preliminary analysis with clinical pharmacy experts worldwide, Res Soc Adm Pharm RSAP source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    ChatGPT-4 reportedly provided advice that was concordant with guidelines that were publicly available.

    “Similarly, ChatGPT-4 reportedly provided advice that was concordant with guidelines that were publicly available, 79”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [79] Sridharan, 2025, Investigating the capabilities of advanced large language models in generating patient instructions and patient educational material, Eur J Hosp Pharm source

  • May combine two claims.

    Findingown data

    Within this review, despite a 12-year search window, all eligible studies were published after 2022, highlighting the recency and growing interest in applying LLMs within pharmacy practice.

    “Also, within this review, despite a 12-year search window, all eligible studies were published after 2022, highlighting the recency and growing interest in applying LLMs within pharmacy practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    A rapid increase in studies evaluating LLMs followed the public release of ChatGPT in November 2022.

    “A rapid increase in studies evaluating LLMs followed the public release of ChatGPT in November 2022, 106”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [106] 2025, Reporting guideline for chatbot health advice studies: the Chatbot Assessment reporting tool (CHART) statement, BMJ Med

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    At full-text screening, one reviewer assessed all articles and two additional reviewers each verified 50% of eligibility decisions, with one disagreement resolved by consensus.

    “At the full-text screening stage, one reviewer assessed all full-text articles for eligibility, and two additional reviewers each verified 50% of eligibility decisions, such that all full-text screening decisions were checked by a second reviewer.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Advanced LLMs generally outperformed earlier versions, suggesting increasing potential to support pharmacist-led direct patient care tasks.

    “Advanced LLMs generally outperformed earlier versions, suggesting increasing potential to support pharmacist-led direct patient care tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: LLM performance in simulated evaluations is indicative of potential performance in real-world pharmacist-led direct patient care.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The protocol was published in PROSPERO on 08 November 2024 (ID: CRD42024607513).

    “The protocol was published in PROSPERO on 08 November 2024 (ID: CRD42024607513).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    A single reviewer extracted data, and three authors independently verified a 10% sample, with no discrepancies identified.

    “a single reviewer extracted data from the final eligible studies.Three authors independently verified the data extraction by each reviewing a separate 10% sample of the included studies; no discrepancies were identified.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The diverse range of represented countries suggests widespread international interest in evaluating LLMs for pharmacist-led direct patient care tasks.

    “The diverse range of represented countries suggests widespread international interest in evaluating LLMs for pharmacist-led direct patient care tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Simpler descriptive counts were generally used to summarize performance on factual recall tasks, whereas classification metrics were more commonly used for detection-based tasks.

    “This suggests that simpler descriptive counts were generally used to summarize performance on factual recall tasks, whereas classification metrics were more commonly used for detection-based tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Reported LLM reproducibility varied across studies and was generally higher for constrained or detection-based tasks than for open-ended text-generation tasks.

    “Overall, reported LLM reproducibility varied across studies and was generally higher for constrained or detection-based tasks, such as medication (pill) identification, 78 than for open-ended tasks related to text-generation such as medication information.58”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Overall findings suggest that LLM type, question context, and prompting approaches may influence hallucination risk in LLMs.

    “Nonetheless, the overall findings suggest that both the LLM type, question context, and prompting approaches may have an influence on the hallucination risk in LLMs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The high number of published evaluations of LLMs for diverse pharmacist-led patient care tasks suggests existing interest in applications relevant to pharmacy practice.

    “The high number of published evaluations of LLMs for diverse pharmacist-led patient care tasks suggests existing interest in applications relevant to pharmacy practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Categorizing and mapping tasks into groups clarified their relevance despite differences in terminology across the studies.

    “However, categorizing and J o u r n a l P r e -p r o o f mapping tasks into groups clarified their relevance despite differences in terminology across the studies.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Newer LLM versions may offer improved performance but may also entail higher costs and computational limitations that could pose challenges for implementation in clinical settings.

    “Therefore, although newer LLM versions may offer improved performance, they may also entail higher costs and computational limitations that could pose challenges for implementation in clinical settings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    This is the first review to examine published evaluations of LLMs for pharmacist-led direct patient care tasks, categorizing 13 task groups mapped to roles and tasks within the GPP and ISCO-08 frameworks.

    “To the best of our knowledge, this is the first review to examine published evaluations of LLMs for pharmacist-led direct patient care tasks, categorizing 13 task groups mapped to roles and tasks within the GPP and ISCO-08 frameworks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Access to domain-specific, current, and context-relevant clinical data affects LLM outputs; such access may be constrained by privacy, governance, and proprietary restrictions.

    “Together, these findings suggest that access to domain-specific, current, and context-relevant clinical data affects LLM outputs; however, such access may be constrained by privacy, governance, and proprietary restrictions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Premisecitation

    Enthusiasm for new technologies can lead to unfocused experimentation, which may impede the full realization of the technology's potential.

    “However, enthusiasm for new technologies can lead to unfocused experimentation, 14 which may impede the full realization of the technology's potential.107,108”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [14] Bedi, 2025, Testing and evaluation of health care applications of large Language models: a systematic review, JAMA source
    • cites [107] Haigh, 2024, Between the booms: AI in Winter, Commun ACM source
    • cites [108] 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    This criticism of unfocused LLM experimentation seems consistent with the heterogeneity identified in the methods and reporting of LLM evaluations.

    “This criticism of unfocused LLM experimentation seems consistent with the heterogeneity identified in the methods and reporting of LLM evaluations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    From a human factors perspective, variability in LLM outputs, sensitivity to prompting, and the need for verification may influence pharmacists' trust in, and reliance on, these systems.

    “From a human factors perspective, variability in LLM outputs, sensitivity to prompting, and the need for verification may influence pharmacists' trust in, and reliance on, these systems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Multidisciplinary collaboration will be required to address the multifaceted challenges associated with integrating LLMs into pharmacist-led direct patient care workflows.

    “Multidisciplinary collaboration will be required to address the multifaceted challenges associated with integrating LLMs into pharmacist-led direct patient care workflows.”
    • Also stated in: § J o u r n a l P r e -p r o o f · plain
      “Successful LLM integration into clinical workflows will require a multidisciplinary approach that includes clinicians, developers, and regulatory experts to address the multifaceted challenges posed, such as safety, effectiveness, and legal compliance.”
    • Also stated in: § J o u r n a l P r e -p r o o f · plain
      “• Workflow integration requires multidisciplinary collaboration to resolve challenges.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Standardized approaches may improve transparency and comparability across studies.

    “• Standardized approaches may improve transparency and comparability across studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Relevant studies published after the search date may have been missed.

    “LLM technologies are rapidly advancing, and relevant studies published after the search date may have been missed.”
    Quote in the paperNot a substantial claim
61 of 61

  • The paper relies on this without support.

    LLM performance in simulated evaluations is indicative of potential performance in real-world pharmacist-led direct patient care.

    “Advanced LLMs generally outperformed earlier versions, suggesting increasing potential to support pharmacist-led direct patient care tasks.”

  • The paper relies on this without support.

    Version labels of LLMs (e.g., GPT-4 versus GPT-3.5) correspond to meaningful generational improvements in capability.

    “studies generally reported higher performance for advanced J o u r n a l P r e -p r o o f LLM versions than for earlier ones.”
    “Overall, the heatmap suggests that GPT-4 generally achieved higher reported percentage accuracy than its predecessors, GPT-3 and GPT-3.5, with higher scores across the included task categories.”
2 of 2

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Citation for the workflow pilot points to a different study

    Minor

    The text cites reference 53 for the study of AI-generated draft replies in pharmacists' workflow, but the reference list assigns 53 to a study of medication consultations. The draft-replies study is listed as reference 52.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Shareef, 2026, Gene therapies administration utilizing ChatGPT-4 as a health biotechnology tool: a mix methods study comparing precision and accuracy with reference protocols, Health Biotechnol Biopharma HBB, 9, 138
  2. 2025, Reporting guideline for chatbot health advice studies: the Chatbot Assessment reporting tool (CHART) statement, BMJ Med, 4

Authors
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P.
Affiliations
The University of Sydney (AU)
Corresponding author
Aslani, Parisa
ORCID
Venue
Research in Social and Administrative Pharmacy
Publisher
Elsevier BV
Published
2026-10
ISSN
1551-7411 (ISSN-L)
DOI
10.1016/j.sapharm.2026.09.007
OpenAlex
W7215841520
Semantic Scholar
928ea2dc0561af896bb17cfd56207598dfa3b792
Cited by
0
References
89
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
Research in Social and Administrative Pharmacy (publishedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
large language models, ChatGPT, LLM evaluation, confabulation, LLM accuracy, privacy risks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review