A Pragmatic Randomized Controlled Trial of Ambient Artificial Intelligence to Improve Health Practitioner Well-Being
Open accessAbstract
BACKGROUND: Electronic health record (EHR) documentation is a major contributor to work-related practitioner exhaustion and the interpersonal disengagement known as burnout. Generative artificial intelligence (AI) scribes that passively capture clinical conversations and draft visit notes may alleviate this burden, but evidence remains limited. METHODS: A 24-week, stepped-wedge, individually randomized pragmatic trial was conducted across ambulatory clinics in two states. Sixty-six health care practitioners were randomly assigned to three 6-week sequences of ambient AI. The coprimary outcomes were professional fulfillment and work exhaustion/interpersonal disengagement from the Stanford Professional Fulfillment Index. Secondary measures included time spent on notes, work outside work (WoW), documentation quality with the Provider Documentation Summarization Quality Instrument 9 (PDSQI-9), and billing diagnostic codes reviewed by professional staff coders. Linear mixed models were used for intention-to-treat (ITT) analyses. RESULTS: A total of 71,487 notes were authored, of which 27,092 (38%) were generated using ambient AI. Ambient AI use had a significant reduction in work exhaustion/interpersonal disengagement (-0.44 points; 95% confidence interval [CI], -0.62 to -0.25; P<0.001), and a nonsignificant increase in professional fulfillment (+0.14 points; 95% CI, 0.004 to 0.28; P=0.04) on a five-point Likert scale. Among secondary measures, time spent on notes decreased (-0.36 hours per day; 95% CI, -0.55 to -0.17). The reduction in WoW (-0.50 hours per day; 95% CI, -0.90 to -0.09) was sensitive to exclusion of extreme values and was no longer significant after removing the top 3% of daily observations. Diagnostic billing codes improved with ambient AI use (P<0.001). Documentation quality, assessed with the PDSQI-9, demonstrated mean scores ranging from 3.97 to 4.99 across domains on a five-point scale. No drift in software performance was detected. CONCLUSIONS: In a real-world randomized implementation, ambient AI reduced health care practitioners' work exhaustion/interpersonal disengagement but did not significantly increase professional fulfillment. Documentation time decreased without compromising diagnosis, billing compliance, or note quality. (Funded by the University of Wisconsin Hospital and Clinics and the National Institutes of Health Clinical and Translational Science Award; ClinicalTrials.gov number, NCT06517082.).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several elements of the existing outpatient provider-focused Pragmatic Trial Operations playbook translated to nursing, including an LHS approach, core technical integration strategies, governance, and processes for iterative evaluation and alignment between operational and research goals.
“Several elements of existing outpatient provider -focused Pragmatic Trial Operations (PTOps) playbook translated to nursing, including an LHS approach, core technical integration strategies, governance, and processes for iterative evaluation and alignment between operational and research goals [19].”
From Protocol for an EHR-embedded pragmatic randomized control trial of Ambient AI to Reduce Nursing Staff Documentation Time (2026) · cites this paper as [19]
The Discussion (page not available) describes pragmatic rollout, Epic API integration, and real-time monitoring in an outpatient practitioner trial; Methods > Recruitment and Data Monitoring describes clinical-operations oversight. Our copy does not describe a PTOps playbook, its translation to nursing, or a process for aligning operational and research goals. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTo monitor for unexpected intercurrent events, the study biostatistician will perform moving-window difference-in-differences analyses using one-month observation windows updated bimonthly.
“To monitor for unexpected intercurrent events (e.g. software or EHR updates), the study biostatistician will perform a moving-window difference-in-differences analyses, as described in [19] using one-month observation windows updated bimonthly.”
From Protocol for an EHR-embedded pragmatic randomized control trial of Ambient AI to Reduce Nursing Staff Documentation Time (2026) · cites this paper as [19]
The Methods (Recruitment and Data Monitoring; page unavailable) say a difference-in-differences analysis monitored for sudden changes in documentation efficiency. The Results (Drift Monitoring; page unavailable) report 15 sequential one-month rolling windows and assess a software update. Our copy does not say that a biostatistician performed the analysis or that the one-month windows were updated bimonthly; the Discussion instead describes monitoring in two-week rolling windows. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Darzi, A., Holmes, C., Yau, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1037-7 · Cited by 748 · see citing works
Trockel, M., Bohman, B., Lesure, E., Hamidi, M. S., Welle, D., Roberts, L., & Shanafelt, T. (2018) · Academic Psychiatry · doi:10.1007/s40596-017-0849-3 · Cited by 602 · see citing works
Pustejovsky, J. E., & Tipton, E. (2018) · Journal of Business & Economic Statistics · doi:10.1080/07350015.2016.1247004 · Cited by 492 · see citing works
Adjustments for Center in Multicenter Studies: An Overview
Localio, A. R., Berlin, J. A., Have, T. R. T., & Kimmel, S. E. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-135-2-200107170-00012 · Cited by 442 · see citing works
Ambient artificial intelligence scribes to alleviate the burden of clinical documentation.
Tierney, A. A., Gayre, G., Hoberman, B., Mattern, B., Ballesca, M., Kipnis, P., Liu, V., & Lee, K. (2024) · NEJM Catalyst · doi:10.1056/cat.23.0404 · Cited by 400 · see citing works
Adler‐Milstein, J., Zhao, W., Willard‐Grace, R., Knox, M. J., & Grumbach, K. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz220 · Cited by 363 · see citing works
Duggan, M. J., Gervase, J., Schoenbaum, A. E., Hanson, W., Howell, J. T., Sheinberg, M., & Johnson, K. B. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.60637 · Cited by 266 · see citing works
Shah, S. J., Devon-Sand, A., Ma, S. P., Jeong, Y., Crowell, T., Smith, M., Liang, A. S., Delahaie, C., Hsia, C., Shanafelt, T., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae295 · Cited by 224 · see citing works
Calculating Confidence Intervals for the Number Needed to Treat
Bender, R. (2001) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00134-3 · Cited by 214 · see citing works
The impact of nuance DAX ambient listening AI documentation: a cohort study.
Haberle, T., Cleveland, C., Snow, G. L., Barber, C., Stookey, N., Thornock, C., Younger, L., Mullahkhel, B., & Ize-Ludlow, D. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae022 · Cited by 171 · see citing works
Physician Perspectives on Ambient AI Scribes
Shah, S., Crowell, T., Jeong, Y., Devon-Sand, A., Smith, M. A., Yang, B., Ma, S. P., Liang, A. S., Delahaie, C., Hsia, C., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.1904 · Cited by 163 · see citing works
Metrics for assessing physician activity using electronic health record log data
Sinsky, C. A., Rule, A., Cohen, G., Arndt, B. G., Shanafelt, T. D., Sharp, C. D., Baxter, S. L., Tai-Seale, M., Yan, S., Chen, Y., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz223 · Cited by 162 · see citing works
Ambient artificial intelligence scribes: utilization and impact on documentation time
Ma, S. P., Liang, A. S., Shah, S. J., Smith, M., Jeong, Y., Devon-Sand, A., Crowell, T., Delahaie, C., Hsia, C., Lin, S., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae304 · Cited by 155 · see citing works
Evaluation of an Ambient Artificial Intelligence Documentation Platform for Clinicians
Stults, C. D., Deng, S., Martinez, M. C., Wilcox, J., Szwerinski, N., Chen, K. H., Driscoll, S., Washburn, J., & Jones, V. G. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2025.8614 · Cited by 147 · see citing works
Ambient Artificial Intelligence Scribes: Learnings after 1 Year and over 2.5 Million Uses
Tierney, A. A., Gayre, G. S., Hoberman, B., Mattern, B., Ballesca, M. A., Hannay, S. B. W., Castilla, K., Lau, C. S., Kipnis, P., Liu, V. X., et al. (2025) · NEJM Catalyst · doi:10.1056/cat.25.0040 · Cited by 134 · see citing works
Albrecht, M., Shanks, D., Shah, T., Hudson, T., Thompson, J., Filardi, T., Wright, K., Ator, G. A., & Smith, T. R. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooaf013 · Cited by 103 · see citing works
AI-Powered Clinical Documentation and Clinicians’ Electronic Health Record Experience
Liu, T., Hetherington, T. C., Stephens, C., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.32460 · Cited by 92 · see citing works
McPeek-Hinz, E., Boazak, M., Sexton, J. B., Adair, K. C., West, V., Goldstein, B. A., Alphin, R. S., Idris, S., Hammond, W. E., Hwang, S. E., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.5686 · Cited by 80 · see citing works
Virtual Scribes and Physician Time Spent on Electronic Health Records
Rotenstein, L. S., Melnick, E. R., Iannaccone, C. K., Zhang, J., Mugal, A., Lipsitz, S. R., Healey, M. J., Holland, C. J., Snyder, R. H., Sinsky, C. A., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.13140 · Cited by 67 · see citing works
Evaluating clinical AI summaries with large language models as judges
Croxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M. K., Wills, G. J., Chen, G., Dligach, D., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02005-2 · Cited by 51 · see citing works
Self-citationCroxford, E., Gao, Y., Pellegrino, N., Wong, K., Wills, G. J., First, E., Schnier, M., Burton, K. A., Ebby, C. G., Gorski, J. K., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf068 · Cited by 40 · see citing works
Self-citationAfshar, M., Resnik, F., Ryan, M. M., Hintzke, J., Lemmon, K. K., Sullivan, A. G., Shah, T., Stordalen, A., Oberst, M., Dambach, J., et al. (2025) · NEJM AI · doi:10.1056/aidbp2401267 · Cited by 40 · see citing works
Self-citationNumber Needed to Treat ( NNT )
Bender, R. (2005) · Encyclopedia of Biostatistics · doi:10.1002/0470011815.b2a04032 · Cited by 23 · see citing works
Empirical Assessment of Well-Being: The Stanford Model of Occupational Well-Being
Bohman, B. D., Makowski, M. S., Wang, H., Menon, N. K., Shanafelt, T. D., & Trockel, M. (2025) · Academic Medicine · doi:10.1097/acm.0000000000006025 · Cited by 17 · see citing works
Lukac, P. J., Turner, W., Vangala, S., Chin, A. T., Khalili, J., Shih, Y. T., Sarkisian, C. A., Cheng, E. M., & Mafi, J. N. (2025) · medRxiv · doi:10.1101/2025.07.10.25331333 · Cited by 17 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [22]Bell RM, McCaffrey DF. Bias reduction in standard errors for linear regression with multi-stage samples. Surv Methodol 2002; 28: 169– 181 (https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2002002/article/9058-eng.pdf?st=X957j0aS).
Cited by
Sorted by citations
Accelerating Science with Human+AI Review
Manrai, A. K., Ouyang, D., Hogan, J. W., & Kohane, I. S. (2025) · NEJM AI · doi:10.1056/aie2501175 · Cited by 17 · see citing works
Wieben, A., Pfaff, J., Baumann, M. R., Resnik, F., Brzozowski, S., Langer, C., Stine, K., Gillis, C., Sullivan, A. G., Voegele, C., et al. (2026) · medRxiv · doi:10.64898/2026.07.09.26357653
Self-citation
- Authors
- Afshar, M., Ryan Baumann, M., Resnik, F., Hintzke, J., Gravel Sullivan, A., Wills, G., Lemmon, K., Dambach, J., Mrotek, L. A., Quinn, M., Abramson, K., Kleinschmidt, P., Brazelton, T. B., Leaf, M. A., Twedt, H., Kunstman, D., Patterson, B., Liao, F., Rasmussen, S., Burnside, E. S., Goswami, C., & Gordon, J.
- Affiliations
- University of Wisconsin–Madison (US)
- Institute for Clinical and Translational Research, School of Medicine and Public Health, University of Wisconsin, Madison, WI, USA
- University of Wisconsin Hospitals and Clinics, Madison, WI, USA
- ORCID
- Afshar, Majid
- Ryan Baumann, Mary
- Resnik, Felice
- Hintzke, Josie
- Gravel Sullivan, Anne
- Wills, Graham
- Lemmon, Kayla
- Dambach, Jason
- Mrotek, Leigh Ann
- Quinn, Mariah
- Abramson, Kirsten
- Kleinschmidt, Peter
- Brazelton, Thomas B.
- Leaf, Margaret A.
- Twedt, Heidi
- Kunstman, David
- Patterson, Brian
- Liao, Frank
- Rasmussen, Stacy
- Burnside, Elizabeth S.
- Goswami, Cherodeep
- Gordon, Joel
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-11-26
- Volume
- 2
- Issue
- 12
- ISSN
- 2836-9386 (ISSN-L)
- PMC
- PMC12858090
- OpenAlex
- W4416715849
- Cited by
- 59
- References
- 25
- Citations by year
- 2026: 57 · 2025: 2
- FWCI
- 46.25
- Type
- journal article
- Language
- en
- Open access
- green
- pmc.ncbi.nlm.nih.gov
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- Funders
- National Institutes of Health (KL2TR002737, UL1TR002373)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › General Health Professions › Healthcare professionals’ stress and burnout
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- electronic health record documentation, burnout, work exhaustion, practitioner well-being, professional fulfillment, documentation time, documentation quality, ambulatory care clinics, billing compliance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
