AI-Generated Draft Replies Integrated Into Health Records and Physicians’ Electronic Communication
Open accessAbstract
Importance: Timely tests are warranted to assess the association between generative artificial intelligence (GenAI) use and physicians’ work efforts. Objective: To investigate the association between GenAI-drafted replies for patient messages and physician time spent on answering messages and the length of replies. Design, Setting, and Participants: Randomized waiting list quality improvement (QI) study from June to August 2023 in an academic health system. Primary care physicians were randomized to an immediate activation group and a delayed activation group. Data were analyzed from August to November 2023. Exposure: Access to GenAI-drafted replies for patient messages. Main Outcomes and Measures: Time spent (1) reading messages, (2) replying to messages, (3) length of replies, and (4) physician likelihood to recommend GenAI drafts. The a priori hypothesis was that GenAI drafts would be associated with less physician time spent reading and replying to messages. A mixed-effects model was used. Results: Fifty-two physicians participated in this QI study, with 25 randomized to the immediate activation group and 27 randomized to the delayed activation group. A contemporary control group included 70 physicians. There were 18 female participants (72.0%) in the immediate group and 17 female participants (63.0%) in the delayed group; the median age range was 35-44 years in the immediate group and 45-54 years in the delayed group. The median (IQR) time spent reading messages in the immediate group was 26 (11-69) seconds at baseline, 31 (15-70) seconds 3 weeks after entry to the intervention, and 31 (14-70) seconds 6 weeks after entry. The delayed group’s median (IQR) read time was 25 (10-67) seconds at baseline, 29 (11-77) seconds during the 3-week waiting period, and 32 (15-72) seconds 3 weeks after entry to the intervention. The contemporary control group’s median (IQR) read times were 21 (9-54), 22 (9-63), and 23 (9-60) seconds in corresponding periods. The estimated association of GenAI was a 21.8% increase in read time (95% CI, 5.2% to 41.0%; P = .008), a −5.9% change in reply time (95% CI, −16.6% to 6.2%; P = .33), and a 17.9% increase in reply length (95% CI, 10.1% to 26.2%; P < .001). Participants recognized GenAI’s value and suggested areas for improvement. Conclusions and Relevance: In this QI study, GenAI-drafted replies were associated with significantly increased read time, no change in reply time, significantly increased reply length, and some perceived benefits. Rigorous empirical tests are necessary to further examine GenAI’s performance. Future studies should examine patient experience and compare multiple GenAIs, including those with medical training.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This study indicates that LLMs may influence physicians to better consider human factors in their management reasoning.
“Similar to studies showing increased empathetic communication phrasing from LLMs to patient queries, this study indicates that LLMs may influence physicians to better consider human factors in their management reasoning.26–28”
From Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial (2024) · cites this paper as [27]
The Results (Likelihood to Recommend; page not indicated) report that physicians valued the empathy in AI-drafted patient replies, and the Discussion says such drafts may reduce cognitive burden. The paper does not establish that the drafts led physicians to better consider human factors in their clinical management reasoning; its reply examples show physicians making their own substantive clinical changes. (LLM, conf=0.82)
Experimental — accuracy not yet measuredUsing artificial intelligence to perform tasks currently handled by NPs, such as sending personalized feedback, reminders to submit BP records, nudging patients to take medications, and scheduling appointments for elevated BP, can improve patient adherence and compliance in remote monitoring.
“For example, using artificial intelligence (AI) to perform tasks currently handled by NPs,29 such as sending personalized feedback, reminders to submit BP records, nudging patients to take medications, and scheduling appointments for elevated BP, can improve patient adherence and compliance in remote monitoring.30”
From Program cost and return on investment analysis of remote patient monitoring for hypertension management in the cardiology department of a large healthcare system (2026) · cites this paper as [30]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredImplementing personalized care strategies-such as tailored feedback, reminders, and incentives-can significantly boost patient engagement.
“Studies have shown that implementing personalized care strategies—such as tailored feedback, reminders, and incentives—can significantly boost patient engagement.30,33,34”
From Program cost and return on investment analysis of remote patient monitoring for hypertension management in the cardiology department of a large healthcare system (2026) · cites this paper as [30]
The Methods > Outcomes Assessment section measures physicians’ message-reading time, reply time, reply length, and likelihood of recommending AI drafts—not patient engagement. The Discussion mentions personalization as an opportunity for improvement, but does not report that tailored feedback, reminders, or incentives increase patient engagement. Searches for those strategies and for increased patient engagement found no supporting passage in our copy; page numbers were not provided. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 16 · Cited by 158
- Authors
- Tai-Seale, M., Baxter, S. L., Vaida, F., Walker, A., Sitapati, A. M., Osborne, C., Diaz, J., Desai, N., Webb, S., Polston, G., Helsten, T., Gross, E., Thackaberry, J., Mandvi, A., Lillie, D., Li, S., Gin, G., Achar, S., Hofflich, H., Sharp, C., Millen, M., & Longhurst, C. A.
- Affiliations
- Department of Family Medicine, University of California San Diego School of Medicine, La Jolla
- University of California San Diego (US)
- Stanford Medicine (US)
- Department of Medicine, University of California San Diego School of Medicine, La Jolla
- ORCID
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2024-04-15
- Volume
- 7
- Issue
- 4
- Pages
- e246565
- ISSN
- 2574-3805 (ISSN-L)
- PMC
- PMC11019394
- OpenAlex
- W4394807113
- Cited by
- 158
- References
- 16
- Citations by year
- 2026: 56 · 2025: 74 · 2024: 27 · 2023: 1
- FWCI
- 14.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JAMA Network Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Agency for Healthcare Research and Quality (P30 HS029770)
- University of California, San Diego
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Healthcare Systems and Technology
- Keywords
- generative AI, patient messaging, physician workload, electronic health records, primary care physicians, physician time use, physician attitudes toward AI, physician-patient communication, quality improvement intervention, delayed activation, academic health system, clinical implementation of AI
- MeSH terms
- Adult, Artificial Intelligence, Communication, Electronics, Female, Humans, Male, Middle Aged, Physicians, Medical Records Systems, Computerized
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
