The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Artificial Intelligence–Generated Draft Replies to Patient Inbox Messages
Open accessAbstract
Importance: The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making. Objective: To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox. Design, Setting, and Participants: A 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot. Intervention: Draft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act–compliant electronic health record–integrated large language model. Main Outcomes and Measures: The primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey. Results: A total of 197 clinicians were enrolled in the pilot; 35 clinicians who were prepilot beta users, out of office, or not tied to a specific ambulatory clinic were excluded, leaving 162 clinicians included in the analysis. The survey analysis cohort consisted of 73 participants (45.1%) who completed both the presurvey and postsurvey. In gastroenterology and hepatology, there were 58 physicians and APPs and 10 nurses. In primary care, there were 83 physicians and APPs, 4 nurses, and 8 clinical pharmacists. The mean AI-generated draft response utilization rate across clinicians was 20%. There was no change in reply action time, write time, or read time between the prepilot and pilot periods. There were statistically significant reductions in the 4-item physician task load score derivative (mean [SD], 61.31 [17.23] presurvey vs 47.26 [17.11] postsurvey; paired difference, −13.87; 95% CI, −17.38 to −9.50; P < .001) and work exhaustion scores (mean [SD], 1.95 [0.79] presurvey vs 1.62 [0.68] postsurvey; paired difference, −0.33; 95% CI, −0.50 to −0.17; P < .001). Conclusions and Relevance: In this quality improvement study of an early implementation of generative AI, there was notable adoption, usability, and improvement in assessments of burden and burnout. There was no improvement in time. Further code-to-bedside testing is needed to guide future development and organizational strategy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613
Trockel, M., Bohman, B., Lesure, E., Hamidi, M. S., Welle, D., Roberts, L., & Shanafelt, T. (2018) · Academic Psychiatry · doi:10.1007/s40596-017-0849-3 · Cited by 602
Self-citationShanafelt, T. D., West, C. P., Dyrbye, L. N., Trockel, M., Tutty, M., Wang, H., Carlasare, L. E., & Sinsky, C. (2022) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2022.09.002 · Cited by 558
Self-citationCreation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435
Self-citationProfessional Satisfaction and the Career Plans of US Physicians
Sinsky, C. A., Dyrbye, L. N., West, C. P., Satele, D., Tutty, M., & Shanafelt, T. D. (2017) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2017.08.017 · Cited by 311
Self-citationHamidi, M. S., Bohman, B., Sandborg, C., Smith-Coggins, R., de Vries, P., Albert, M. S., Murphy, M. L., Welle, D., & Trockel, M. T. (2018) · BMC Health Services Research · doi:10.1186/s12913-018-3663-z · Cited by 278
Tai-Seale, M., Dillon, E. C., Yang, Y., Nordgren, R., Steinberg, R. L., Nauenberg, T., Lee, T. C., Meehan, A., Li, J., Chan, A. S., et al. (2019) · Health Affairs · doi:10.1377/hlthaff.2018.05509 · Cited by 226
Self-citationPhysician Task Load and the Risk of Burnout Among US Physicians in a National Survey
Harry, E. M., Sinsky, C. A., Dyrbye, L. N., Makowski, M. S., Trockel, M., Tutty, M., Carlasare, L. E., West, C. P., & Shanafelt, T. D. (2020) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/j.jcjq.2020.09.011 · Cited by 192
Self-citation
Cited by
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024) · doi:10.1101/2024.04.15.24305869 · Cited by 738
Self-citationTesting and Evaluation of Health Care Applications of Large Language Models: A Systematic Review
Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2025) · JAMA · doi:10.1001/jama.2024.21700 · Cited by 662
Self-citationShah, S. J., Devon-Sand, A., Ma, S. P., Jeong, Y., Crowell, T., Smith, M., Liang, A. S., Delahaie, C., Hsia, C., Shanafelt, T., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae295 · Cited by 224
Self-citationBhuyan, S. S., Sateesh, V., Mukul, N., Galvankar, A., Mahmood, A., Nauman, M., Rai, A., Bordoloi, K., Basu, U., & Samuel, J. (2025) · Journal of Medical Systems · doi:10.1007/s10916-024-02136-1 · Cited by 145
Ng, K. K. Y., Matsuba, I., & Zhang, P. C. (2025) · NEJM AI · doi:10.1056/aira2400380 · Cited by 107
Large language models in biomedicine and health: current research landscape and future directions
Lu, Z., Peng, Y., Cohen, T., Ghassemi, M., Weng, C., & Tian, S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae202 · Cited by 92
Triola, M. M., & Rodman, A. (2025) · Academic Medicine · doi:10.1097/acm.0000000000005963 · Cited by 92
Performance of Large Language Models on Medical Oncology Examination Questions
Longwell, J. B., Hirsch, I., Binder, F., Gonzalez Conchas, G. A., Mau, D., Jang, R., Krishnan, R. G., & Grant, R. C. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.17641 · Cited by 81
Does AI-powered clinical documentation enhance clinician efficiency: a longitudinal study.
Liu, T.-L., Hetherington, T. C., Dharod, A., Carroll, T., Bundy, R., Nguyen, H., Bundy, H. E., Isreal, M., McWilliams, A., & Cleveland, J. A. (2024) · NEJM AI · doi:10.1056/aioa2400659 · Cited by 65
You, J. G., Hernandez-Boussard, T., Pfeffer, M. A., Landman, A., & Mishuris, R. G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01506-4 · Cited by 59
Self-citationThe effect of ambient artificial intelligence notes on provider burnout.
Misurac, J., Knake, L. A., & Blum, J. M. (2025) · Applied Clinical Informatics · doi:10.1055/a-2461-4576 · Cited by 58
“21,22 Tierney et al report that with a large-scale deployment of an ambient AI documentation tool, there was a reduction of after-hours time in primary care users, though burnout was not assessed.”
Performance of multimodal artificial intelligence chatbots evaluated on clinical oncology cases
Chen, D., Huang, R. S., Jomy, J., Wong, P., Yan, M., Croke, J., Tong, D., Hope, A., Eng, L., & Raman, S. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.37711 · Cited by 49
Prompt engineering on leveraging large language models in generating response to InBasket messages
Yan, S., Knapp, W., Leong, A., Kadkhodazadeh, S., Das, S., Jones, V. G., Clark, R., Grattendick, D., Chen, K., Hladik, L., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae172 · Cited by 47
Guo, Y., Wang, J., Hu, D., Tam, S., Gilman, C., Chow, E., Perret, D., Pandita, D., & Zheng, K. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf180 · Cited by 43
Sarraf, B., & Ghasempour, A. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1628831 · Cited by 40
Opportunities and risks of artificial intelligence in patient portal messaging in primary care
Biro, J. M., Handley, J. L., Malcolm McCurry, J., Visconti, A., Weinfeld, J., Gregory Trafton, J., & Ratwani, R. M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01586-2 · Cited by 35
Ethics dumping in artificial intelligence
Bélisle-Pipon, J.-C., & Victor, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1426761 · Cited by 26
“…predictive and diagnostic tools, clinical decision support systems (CDSS), AI-generated in-basket responses, or patient monitoring tools tout promises of increased efficiency and optimized resource allocation (Chen et al., 2021; Elhaddad and Hamam, 2024; Garcia et al., 2024; Khosravi et al., 2024).”
Croxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M., Wills, G., Chen, G., Dligach, D., et al. (2025) · doi:10.1101/2025.04.22.25326219 · Cited by 25
Workum, J. D., van de Sande, D., Gommers, D., & van Genderen, M. E. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1504805 · Cited by 25
Pavlik, E. J., Land Woodward, J., Lawton, F., Swiecki-Sikora, A. L., Ramaiah, D. D., & Rives, T. A. (2025) · doi:10.20944/preprints202502.0521.v1 · Cited by 8
Suenghataiphorn, T., Danpanichkul, P., Tribuddharat, N., & Kulthamrongsri, N. (2025) · Journal of Clinical and Experimental Hepatology · doi:10.1016/j.jceh.2025.102627 · Cited by 6
Büyükceran, E. U., Seyfettin, A., Babatürk, A., Özkan, M. B., Çolak, D., Ünal, İ., Kaymaz, E., Ergün, E., Emer, M. Ö., & Mersin, H. H. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02314-9 · Cited by 6
Improving Patient Engagement: Is There a Role for Large Language Models?
Kouzy, R., & Bitterman, D. S. (2024) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.06.001 · Cited by 3
Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations
Yakob, R. A., Bamgboje-Ayodele, A., Collins, J. C., & Aslani, P. (2026) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2026.09.007 · Cited by 1
Metadata
- Authors
- Garcia, P., Ma, S. P., Shah, S., Smith, M., Jeong, Y., Devon-Sand, A., Tai-Seale, M., Takazawa, K., Clutter, D., Vogt, K., Lugtu, C., Rojo, M., Lin, S., Shanafelt, T., Pfeffer, M. A., & Sharp, C.
- Venue
- JAMA Network Open
- Publisher
- American Medical Association (AMA)
- Published
- 2024-03-20
- Volume
- 7
- Issue
- 3
- Pages
- e243201
- ISSN
- 2574-3805 (ISSN-L)
- OpenAlex
- W4392986561
- Cited by
- 266
- References
- 28
- FWCI
- 21.33
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.