Accuracy and safety of ChatGPT-3.5 in assessing over-the-counter medication use during pregnancy: a descriptive comparative study
Open accessAbstract
As artificial intelligence (AI) becomes increasingly utilized to perform tasks requiring human intelligence, patients who are pregnant may turn to AI for advice on over-the-counter (OTC) medications. However, medications used in pregnancy may pose profound safety concerns limited by data availability. This study focuses on a chatbot’s ability to accurately provide information regarding OTC medications as it relates to patients that are pregnant. A prospective, descriptive design was used to compare the responses generated by the Chat Generative Pre-Trained Transformer 3.5 (ChatGPT-3.5) to the information provided by UpToDate®. Eighty-seven of the top pharmacist-recommended OTC drugs in the United States (U.S.) as identified by Pharmacy Times were assessed for safe use in pregnancy using ChatGPT-3.5. A piloted, standard prompt was input into ChatGPT-3.5, and the responses were recorded. Two groups independently rated the responses compared to UpToDate on their correctness, completeness, and safety using a 5-point Likert scale. After independent evaluations, the groups discussed the findings to reach a consensus, with a third independent investigator giving final ratings. For correctness, the median score was 5 (interquartile range [IQR]: 5–5). For completeness, the median score was 4 (IQR: 4–5). For safety, the median score was 5 (IQR: 5–5). Despite high overall scores, the safety errors in 9% of the evaluations (n = 8), including omissions that pose a risk of serious complications, currently renders the chatbot an unsafe standalone resource for this purpose.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A prospective study found no statistically significant difference in mean accuracy between medication reviews performed by pharmacists assisted by LLMs and those performed by pharmacists unassisted by LLMs.
“they found no statistically significant difference in mean accuracy between reviews performed by pharmacists assisted by LLMs and those performed by pharmacists unassisted by LLMs.68”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [68]
The Limitations section describes this as a cross-sectional assessment, not a prospective study. The Methods and Results describe reviewers scoring ChatGPT-3.5 responses against UpToDate; they do not compare medication reviews by LLM-assisted and unassisted pharmacists or report a statistical test of mean accuracy. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Xu, L., Sanders, L., Li, K., & Chow, J. C. L. (2021) · JMIR Cancer · doi:10.2196/27850 · Cited by 571 · see citing works
Internet use by pregnant women seeking pregnancy-related information: a systematic review
Sayakhot, P., & Carolan, M. C. (2016) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-016-0856-5 · Cited by 429 · see citing works
Over-the-counter medications in pregnancy.
Black, R. A., & Hill, D. A. (2003) · PubMed · Cited by 186 · see citing works
Prescription and Other Medication Use in Pregnancy
Haas, D. M., Marsh, D. J., Dang, D. T., Parker, C. B., Wing, D. A., Simhan, H. N., Grobman, W. A., Mercer, B. M., Silver, R. M., Hoffman, M., et al. (2018) · Obstetrics and Gynecology · doi:10.1097/aog.0000000000002579 · Cited by 176 · see citing works
Use of UpToDate and outcomes in US hospitals
Isaac, T., Zheng, J., & Jha, A. K. (2011) · Journal of Hospital Medicine · doi:10.1002/jhm.944 · Cited by 122 · see citing works
Hatia, A., Doldo, T., Parrini, S., Chisci, E., Cipriani, L., Montagna, L., Laganà, G., Guenza, G., Agosta, E., Vinjolli, F., et al. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13030735 · Cited by 97 · see citing works
Risk Management and Patient Safety in the Artificial Intelligence Era: A Systematic Review
Ferrara, M., Bertozzi, G., Fazio, N. D., Aquila, I., Fazio, A. D., Maiese, A., Volonnino, G., Frati, P., & Russa, R. L. (2024) · Healthcare · doi:10.3390/healthcare12050549 · Cited by 62 · see citing works
Future of ADHD Care: Evaluating the Efficacy of ChatGPT in Therapy Enhancement
Berrezueta-Guzmán, J., Kandil, M., Martín-Ruiz, M.-L., Pau, I., & Krusche, S. (2024) · Healthcare · doi:10.3390/healthcare12060683 · Cited by 53 · see citing works
Bradley-Ridout, G., Nekolaichuk, E., Jamieson, T., Jones, C. A., Morson, N., Chuang, R., & Springall, E. (2021) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2021.1176 · Cited by 40 · see citing works
Gomez-Cabello, C. A., Borna, S., Pressman, S. M., Haider, S. A., Sehgal, A., Leibovich, B. C., & Forte, A. J. (2024) · Healthcare · doi:10.3390/healthcare12111083 · Cited by 36 · see citing works
Ahmadi, S.-F., Faghankhani, M., Javanbakht, A., Akbarshahi, M., Mirghorbani, M., Safarnejad, B., & Baradaran, H. R. (2011) · Medical Teacher · doi:10.3109/0142159x.2010.531155 · Cited by 31 · see citing works
Baxter, S. L., Lander, L. I., Clay, B. J., Bell, J., Hansen, K., Walker, A., & Tai‐Seale, M. (2022) · Applied Clinical Informatics · doi:10.1055/s-0041-1742216 · Cited by 31 · see citing works
Alsanosi, S., & Padmanabhan, S. (2024) · Healthcare · doi:10.3390/healthcare12070788 · Cited by 31 · see citing works
Over the Counter Pain Medications Used by Adults: A Need for Pharmacist Intervention
Karłowicz-Bodalska, K., Sauer, N. J., Jonderko, L., & Wiela−Hojeńska, A. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20054505 · Cited by 22 · see citing works
Accuracy of a chatbot in answering questions that patients should ask before taking a new medication
Cornelison, B. R., Erstad, B. L., & Edwards, C. J. (2024) · Journal of the American Pharmacists Association · doi:10.1016/j.japh.2024.102110 · Cited by 20 · see citing works
Siebielec, J., Ordak, M., Oskroba, A., Dworakowska, A. M., & Bujalska‐Zadrożny, M. (2024) · Healthcare · doi:10.3390/healthcare12161637 · Cited by 19 · see citing works
Doctors' perceptions of laboratory monitoring in office practice
Goldman, R. E., Soran, C. S., Hayward, G. L., et al. (2010) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2009.01282.x · Cited by 19 · see citing works
The Risks of “Getting High” on Over-the-Counter Drugs during Pregnancy
Ősz, B.-E., Ștefănescu, R., Sălcudean, A., Jîtcă, G., & Vari, C.-E. (2024) · Scientia Pharmaceutica · doi:10.3390/scipharm92010007 · Cited by 3 · see citing works
Zhou, T., Shen, Y., Lyu, J., Li, Y., Wang, H., Hong, S., & Ji, Y. (2024) · Healthcare · doi:10.3390/healthcare12070713 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Koluman, A., Yiğitbay, A., Çiftçi, E. A., Çiftçi, M. U., Ziroğlu, N., & Kural, C. (2026) · Journal of Medicine and Palliative Care · doi:10.47582/jompac.1884791 · Cited by 2 · see citing works
Wang, R., Li, Y., Feng, X., & Feng, X. (2026) · International Journal of Clinical Pharmacy · doi:10.1007/s11096-026-02138-8 · Cited by 1 · see citing works
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007
A 20-Year Analysis of Analgesic Enquiries to an Obstetric Medicines Information Service
Mukadam, N., Emmerton, L., Czarniak, P., Burford, O., Teoh, S. W. K., & Lebedevs, T. (2026) · Anesthesia Research · doi:10.3390/anesthres3020009
- Authors
- Cornelison, B., Axon, D. R., Abbott, B., Bishop, C., Jebara, C., Kumar, A., & Root, K. A.
- Affiliations
- University of Arizona (US)
- Corresponding author
- Cornelison, Bernadette
- Venue
- Pharmacy
- Publisher
- MDPI AG
- Published
- 2025-07-30
- Volume
- 13
- Issue
- 4
- Pages
- 104
- ISSN
- 2226-4787 (ISSN-L)
- PMC
- PMC12389367
- OpenAlex
- W4412761284
- Cited by
- 4
- References
- 18
- Citations by year
- 2026: 4
- FWCI
- 0.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Pharmacy (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Pregnancy and Medication Impact
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmacy and Medical Practices
- Keywords
- ChatGPT-3.5, large language models, over-the-counter drugs, pregnancy, drug safety, chatbots, UpToDate, AI-generated health information, accuracy assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
