CoChat
Sign Up
Cited by 9Artificial Intelligence in Healthcare and EducationPharmacy and Medical PracticesPharmaceutical Practices and Patient Outcomes

The promise and challenges of ChatGPT in community pharmacy: a comparative analysis of response accuracy

Open access
Salama, Ali H.·2024·Pharmacia
DOI 10.3897/pharmacia.71.e116927
Open this paper in CoChat — free accountOpen PDF

Abstract

This study evaluates ChatGPT, an AI-based language model, in addressing common pharmacist inquiries in community pharmacies. The assessment encompasses Drug-Drug Interactions, Adverse Drug Effects, Drug Dosage, and Alternative Therapies, each comprising 20 questions, totaling 80 questions. Responses from ChatGPT were compared against standard answers, generating textual and chart scores. Textual score was computed by relating correct answers to the total questions within each category, while chart score involved the total correct answers multiplied by the chart-type questions. ChatGPT exhibited distinct performance rates: 30% for Drug-Drug Interactions, 65% for Adverse Drug Effects, 35% for Drug Dosage, and an impressive 85% for Alternative Therapies. While Alternative Therapies displayed high accuracy, challenges arose in accurately addressing Drug Dosage and Drug-Drug Interactions. Conclusion: The study underscores the complexity of pharmacy-related inquiries and the necessity for AI model enhancement. Despite promising accuracy in certain categories, like Alternative Therapies, improvements are crucial for Drug Dosage and Drug-Drug Interactions. The findings emphasize the need for ongoing AI model development to optimize integration into community pharmacy settings.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3897/pharmacia.71.e116927

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Only one study pilot-tested the real-world implementation of a GPT-4 LLM integrated into clinical pharmacists' workflow to assist in drafting responses to incoming electronic patient inquiries.

    “Only one study pilot-tested the real-world implementation of a GPT-4 LLM integrated into clinical pharmacists' workflow to assist in drafting responses to incoming electronic patient inquiries.53”

    From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [53]

    The Study design describes manually entering a predefined set of questions into ChatGPT, not integrating a model into pharmacists’ workflow to draft replies to incoming patient inquiries. The Discussion identifies the model as ChatGPT 3.5, not GPT-4, and says the question set may not reflect real-world practice. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • In one real-world integration study, GPT-4 failed to generate draft responses for 5% of patient inquiries because of token and computational limits.

    “In one study that integrated an LLM into real-world clinician workflows, GPT-4 failed to generate draft responses for 5% of patient inquiries because of token and computational limits.53”

    From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [53]

    The Study design describes manually entering predefined questions into ChatGPT 3.5, not integrating GPT-4 into clinician workflows. The Results report accuracy across 80 questions, rather than a 5% failure to generate drafts because of token or computational limits. (LLM, conf=0.89)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • In the only real-world workflow study, GPT-3.5 was used to categorize patient inquiries and GPT-4 drafted responses for clinician review.

    “In that study, GPT-3.5 was used to categorize patient inquiries and GPT-4 drafted responses for clinician review.53”

    From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [53]

    The Study design (page not provided) describes manually entering a predefined set of questions into ChatGPT and comparing its answers with standard responses. It says the model tested was ChatGPT 3.5; the Discussion likewise says the study focused on that version. This is contrary to the attributed GPT-3.5 categorization and GPT-4 drafting workflow for clinician review. (LLM, conf=0.89)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • In the only study that integrated LLMs into real-world patient care, the models were reported to comply with HIPAA.

    “In the only study that integrated LLMs into realworld patient care, the models were reported to comply with the Health Insurance Portability and Accountability Act (HIPAA).53”

    From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [53]

    The Study design section describes manually entering a standardized set of questions into ChatGPT, rather than integrating a model into real-world patient care. The Discussion likewise describes real-time patient data as a possible future improvement. No passage found reports HIPAA compliance. (LLM, conf=0.82)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Despite LLMs' potential, notable challenges include hallucination risk, response sensitivity to prompting method, cost, and data privacy.

    “Despite LLMs' potential to support pharmacist-led direct patient care tasks, notable challenges include hallucination risk, response sensitivity to prompting method, cost, and data privacy.16,53,92”

    From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [53]

    The Results (page not provided) document incorrect ChatGPT answers, and the Discussion warns that responses may be misleading or harmful, supporting a concern about answer reliability. The Study design (page not provided) repeats identical questions rather than comparing prompting methods; the passages do not substantiate sensitivity to prompting method, cost, or data privacy as challenges. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Salama, A. H.
Affiliations
Middle East University (JO)
Corresponding author
Salama, Ali H.
ORCID
Salama, Ali H.
Venue
Pharmacia
Publisher
Pensoft Publishers
Published
2024-01-30
Volume
71
Pages
1-5
ISSN
  • 0428-0296 (ISSN-L)
  • 2603-557X
DOI
10.3897/pharmacia.71.e116927
OpenAlex
W4391345367
Cited by
9
References
18
Citations by year
2026: 2 · 2025: 5 · 2024: 2
FWCI
0.84
Type
journal article
Language
en
Open access
Licence
Locations
  • Pharmacia (publishedVersion, cc-by)
  • ZooKeys (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
  • Health Sciences › Medicine › Geriatrics and Gerontology › Pharmaceutical Practices and Patient Outcomes
Keywords
ChatGPT, large language models, community pharmacy, drug-drug interactions, adverse drug reactions, drug dosing, complementary and alternative medicine, response accuracy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review