The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Opportunities and risks of artificial intelligence in patient portal messaging in primary care
Open accessAbstract
The rapid increase in patient portal messaging has heightened the workload for primary care physicians (PCPs), contributing to burnout. The use of generative artificial intelligence (AI) to draft responses to patient messages has shown promise in reducing cognitive burden, yet there is still much unknown about the safety and perceptions of using AI drafts. This cross-sectional simulation study assessed whether PCPs could identify and correct errors in AI-generated draft responses to patient portal messages. Twenty practicing PCPs reviewed 18 patient portal messages, four of which contained errors categorized as objective inaccuracies or potentially harmful omissions. Each error was insufficiently addressed by 13-15 participants, and 35-45% of erroneous drafts were submitted entirely unedited. While 80% of participants agreed AI drafts reduced cognitive workload and 75% found them safe, uncorrected errors highlight patient safety risks, underscoring the need for improved design, training, and error-detection mechanisms for AI tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a cross-sectional simulation study evaluating provider editing of AI-generated draft responses to patient portal messages, participants frequently failed to recognize clinically relevant note errors.15”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Although clinician proofreading remains the de jure safeguard against ADS-generated errors, early simulation studies suggest this may not be an effective safeguard.15”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“Fourth, providers may have succumbed to automation bias 23 , the tendency to over-rely on automation, which can be exacerbated under increasedworkload 24 .”
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 1×
“This increase in patient portal messaging has been speci fi cally burdensome on primary care physicians (PCPs) 4 , who spend over half their workday on EHR-related tasks, 1.4 h of which occurs after clinical hours 5 .”
InfluentialPerformance Consequences of Automation-Induced 'Complacency'
Parasuraman, R., Molloy, R., & Singh, I. L. (1993) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0301_1 · Cited by 875 · cited 1×
“Third, automation complacency 21 , the tendency to monitor automated systems poorly, may have played a role.”
Varieties of Confirmation Bias
Klayman, J. (1995) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60315-1 · Cited by 755
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 1×
“Fourth, providers may have succumbed to automation bias 23 , the tendency to over-rely on automation, which can be exacerbated under increasedworkload 24 .”
Shanafelt, T. D., West, C. P., Sinsky, C., Trockel, M., Tutty, M., Wang, H., Carlasare, L. E., & Dyrbye, L. N. (2022) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2021.11.021 · Cited by 434
Adler-Milstein, J., Zhao, W., Willard-Grace, R., Knox, M., & Grumbach, K. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz220 · Cited by 356
Monitoring an Automated System for a Single Failure: Vigilance and Task Complexity Effects
MOLLOY, R., & PARASURAMAN, R. (1996) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/001872089606380211 · Cited by 321
Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages
Garcia, P., Ma, S. P., Shah, S., Smith, M., Jeong, Y., Devon-Sand, A., Tai-Seale, M., Takazawa, K., Clutter, D., Vogt, K., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.3201 · Cited by 266
Functional fixedness as related to problem solving: a repetition of three experiments.
Adamson, R. E. (1952) · Journal of Experimental Psychology · doi:10.1037/h0062487 · Cited by 250
Cited by
AI, Health, and Health Care Today and Tomorrow: The JAMA Summit Report on Artificial Intelligence
Angus, D. C., Khera, R., Lieu, T., Liu, V., Ahmad, F. S., Anderson, B., Bhavani, S. V., Bindman, A., Brennan, T., Celi, L. A., et al. (2025) · JAMA · doi:10.1001/jama.2025.18490 · Cited by 195
Ethical Obligations to Inform Patients About Use of AI Tools
Mello, M. M., Char, D., & Xu, S. H. (2025) · JAMA · doi:10.1001/jama.2025.11417 · Cited by 57
Brewster, R., Tse, G., Fan, A. L., Elborki, M., Newell, M., Gonzalez, P., Hoq, A., Chang, C. T., Chowdhury, M., Geeti, A., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02055-6 · Cited by 24
A Dataset for Addressing Patient’s Information Needs related to Clinical Course of Hospitalization
Soni, S., & Fushman, D. D. (2026) · Scientific Data · doi:10.1038/s41597-026-06639-z · Cited by 19
Anderson, T. N., Mohan, V., Dorr, D. A., Ratwani, R. M., Biro, J. M., & Gold, J. A. (2025) · Mayo Clinic proceedings. Digital health · doi:10.1016/j.mcpdig.2025.100292 · Cited by 17
“15 Because of ADS licensing costs, providers may experience explicit or implicit organizational pressure to increase their clinical output, which could further compound this problem.”
Self-citation
Metadata
- Authors
- Biro, J. M., Handley, J. L., Malcolm McCurry, J., Visconti, A., Weinfeld, J., Gregory Trafton, J., & Ratwani, R. M.
- Affiliations
- MedStar Health (US)
- Georgetown University Medical Center (US)
- Georgetown University (US)
- Corresponding author
- Biro, Joshua M.
- Venue
- npj Digital Medicine
- Published
- 2025-04-24
- Volume
- 8
- Issue
- 1
- Pages
- 222-222
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC12022076
- OpenAlex
- W4409772336
- Cited by
- 35
- References
- 22
- Citations by year
- 2026: 27 · 2025: 8
- FWCI
- 8.42
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Agency for Healthcare Research and Quality (1r03hs030307-01, R03 HS030307)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- generative AI, patient portal messaging, primary care physicians, error detection, patient safety, physician burnout, mental workload, physician perceptions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Opportunities and risks of artificial intelligence in patient portal messaging in primary care. https://app.cochat.ai/papers/10.1038/s41746-025-01586-2
- The paper
- Biro, J. M., Handley, J. L., Malcolm McCurry, J., Visconti, A., Weinfeld, J., Gregory Trafton, J., & Ratwani, R. M. (2025). Opportunities and risks of artificial intelligence in patient portal messaging in primary care. npj Digital Medicine. https://doi.org/10.1038/s41746-025-01586-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.