Predicting physician burnout using clinical activity logs: Model performance and lessons learned
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Background: Burnout is a significant public health concern affecting more than half of the healthcare workforce; however, passive screening tools to detect burnout are lacking. We investigated the ability of machine learning (ML) techniques to identify burnout using passively collected electronic health record (EHR)-based audit log data. Method: Physician trainees participated in a longitudinal study where they completed monthly burnout surveys and provided access to their EHR-based audit logs. Using the monthly burnout scores as the target outcome, we trained ML models using combinations of features derived from audit log data—aggregate measures of clinical workload, time series-based temporal measures of EHR use, and the baseline burnout score. Five ML models were constructed to predict burnout as a continuous score: penalized linear regression, support vector machine, neural network, random forest, and gradient boosting machine. Results: 88 trainee physicians participated and completed 416 surveys; >10 million audit log actions were collected (Mean [Standard Deviation]=25,691 [14,331] actions per month, per physician). The workload feature set predicted burnout score with a mean absolute error (MAE) of 0.602 (95% Confidence Interval (CI), 0.412–0.826), and was able to predict burnout status with an average AUROC of 0.595 (95% CI 0.355–0.808) and average accuracy 0.567 (95% CI 0.393–0.742). The temporal feature set had a similar performance, with MAE 0.596 (95% CI 0.391–0.826), and AUROC 0.581 (95% CI 0.343–0.790). The addition of the baseline burnout score to the workload features improved the model performance to a mean AUROC of 0.829 (95% CI 0.607–0.996) and mean accuracy of 0.781 (95% CI 0.587–0.936); however, this performance was not meaningfully different than using the baseline burnout score alone. Conclusions: Current findings illustrate the complexities of predicting burnout exclusively based on clinical work activities as captured in the EHR, highlighting its multi-factorial and individualized nature. Future prediction studies of burnout should account for individual factors (e.g., resilience, physiological measurements such as sleep) and associated system-level factors (e.g., leadership).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Shanafelt, T. D., Boone, S., Tan, L., Dyrbye, L. N., Sotile, W., Satele, D., West, C. P., Sloan, J., & Oreskovich, M. R. (2012) · Archives of Internal Medicine · doi:10.1001/archinternmed.2012.3199 · Cited by 3,439
Monsell, S. (2003) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(03)00028-7 · Cited by 2,984
Shanafelt, T. D., Hasan, O., Dyrbye, L. N., Sinsky, C., Satele, D., Sloan, J., & West, C. P. (2015) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2015.08.023 · Cited by 2,885
Physician burnout: contributors, consequences and solutions
West, C. P., Dyrbye, L. N., & Shanafelt, T. D. (2018) · Journal of Internal Medicine · doi:10.1111/joim.12752 · Cited by 2,768
Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis
West, C. P., Dyrbye, L. N., Erwin, P. J., & Shanafelt, T. D. (2016) · The Lancet · doi:10.1016/s0140-6736(16)31279-x · Cited by 2,362
Burnout and Medical Errors Among American Surgeons
Shanafelt, T. D., Balch, C. M., Bechamps, G., Russell, T., Dyrbye, L., Satele, D., Collicott, P., Novotny, P. J., Sloan, J., & Freischlag, J. (2010) · Annals of Surgery · doi:10.1097/sla.0b013e3181bfdab3 · Cited by 2,099
Prevalence of Burnout Among Physicians: A Systematic Review
Rotenstein, L. S., Torre, M., Ramos, M. A., Rosales, R. C., Guille, C., Sen, S., & Mata, D. A. (2018) · JAMA · doi:10.1001/jama.2018.12777 · Cited by 1,904
Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494
Shanafelt, T. D., Dyrbye, L. N., Sinsky, C., Hasan, O., Satele, D., Sloan, J., & West, C. P. (2016) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2016.05.007 · Cited by 1,086
Association of resident fatigue and distress with perceived medical errors
West, C. P. (2009) · JAMA · doi:10.1001/jama.2009.1389 · Cited by 999
Research works printed but not linked: 37
- Dyrbye, 2014, Burnout among US medical students, residents, and early career physicians relative to the general US population, Acad. Med., 89, 443, 10.1097/ACM.0000000000000134 · DOI · Paper page
- West, 2011, Quality of life, burnout, educational debt, and medical knowledge among internal medicine residents, JAMA, 306, 952, 10.1001/jama.2011.1247 · DOI · Paper page
- Allegra, 2005, Prevalence of burnout in the US oncology community: results of a 2003 survey, J. Oncol. Pract., 1, 140, 10.1200/jop.2005.1.4.140 · doi:10.1200/jop.2005.1.4.140 · Paper page
- Embriaco, 2007, High level of burnout in intensivists: prevalence and associated factors, Am. J. Respiratory Crit. Care Med., 175, 686, 10.1164/rccm.200608-1184OC · DOI · Paper page
- Shanafelt, 2011, Special report: suicidal ideation among American surgeons, Arch. Surg., 146, 54, 10.1001/archsurg.2010.292 · DOI · Paper page
- van der Heijden, 2008, Suicidal thoughts among medical residents with burnout, Arch. Suicide Res., 12, 344, 10.1080/13811110802325349 · DOI · Paper page
- Oreskovich, 2012, Prevalence of alcohol use disorders among American surgeons, Arch. Surg., 147, 168, 10.1001/archsurg.2011.1481 · DOI · Paper page
- J. Klein, K. Grosse Frie, K. Blum, O. von dem Knesebeck, Burnout and perceived quality of care among German clinicians in surgery, Int. J. Quality Health Care 22(6) (2010) 525-530. · DOI · Paper page
- C.S. Dewa, D. Loong, S. Bonato, L. Trojanowski, The relationship between physician burnout and quality of healthcare in terms of safety and acceptability: a systematic review. BMJ Open 7(6) (2017) e015141. · DOI · Paper page
- Panagioti, 2018, Association between physician burnout and patient safety, professionalism, and patient satisfaction: a systematic review and meta-analysis, JAMA Int. Med., 178, 1317, 10.1001/jamainternmed.2018.3713 · DOI · Paper page
- M. Del Carmen, J. Herman, S. Rao, M.K. Hidrue, D. Ting, S.R. Lehrhoff, S. Lenz, J. Heffernan, and T.G. Ferris, Trends and factors associated with physician burnout at a multispecialty academic faculty practice organization, JAMA Network Open 2(3) (2019) e190554. · DOI · Paper page
- Shanafelt, 2014, Satisfaction with work-life balance and the career and retirement plans of US oncologists, J. Clin. Oncol., 32, 1127, 10.1200/JCO.2013.53.4560 · DOI · Paper page
- Han, 2019, Estimating the attributable cost of physician burnout in the United States, Ann. Intern. Med., 170, 784, 10.7326/M18-1422 · DOI · Paper page
- Kannampallil, 2021, Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects, J. Am. Med. Inform. Assoc., 28, 1032, 10.1093/jamia/ocaa305 · DOI
- Ballermann, 2011, Validation of the Work Observation Method By Activity Timing (WOMBAT) method of conducting time-motion observations in critical care settings: an observational study, BMC Med. Inf. Decis. Making, 11, 32, 10.1186/1472-6947-11-32 · DOI · Paper page
- DesRoches, 2008, Registered nurses' use of electronic health records: findings from a national survey, Medscape J. Med., 10, 164
- Ouyang, 2016, Internal medicine resident computer usage: an electronic audit of an inpatient service, JAMA Int. Med., 176, 252, 10.1001/jamainternmed.2015.6831 · DOI · Paper page
- Adler-Milstein, 2020, EHR audit logs: a new goldmine for health services research?, J. Biomed. Inform., 101, 10.1016/j.jbi.2019.103343 · DOI · Paper page
- N. Ratanawongsa, G.Y. Matta, F.B. Bohsali, M.S. Chisolm, Reducing misses and near misses related to multitasking on the electronic health record: observational study and qualitative analysis, JMIR Human Factors 5(1) (2018) e4. · DOI
- T.A. Lieu, E.M. Warton, J.A. East, M.F. Moeller, S. Prausnitz, M. Ballesca, G. Mark, F. Akbar, S. Awsare, Y.-F.I. Chen, Evaluation of Attention Switching and Duration of Electronic Inbox Work Among Primary Care Physicians. JAMA Network Open 4(1) (2021) e2031856. · DOI
- Kroth, 2018, The electronic elephant in the room: Physicians and the electronic health record, JAMIA Open, 1, 49, 10.1093/jamiaopen/ooy016 · DOI · Paper page
- Tai-Seale, 2019, Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records, Health Aff., 38, 1073, 10.1377/hlthaff.2018.05509 · DOI · Paper page
- Adler-Milstein, 2020, Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians, J. Am. Med. Inform. Assoc., 27, 531, 10.1093/jamia/ocz220 · DOI · Paper page
- Collins, 2015, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement, J. Brit. Surg., 102, 148, 10.1002/bjs.9736 · DOI · Paper page
- Maslach, 1986, Vol. 21
- Sinha, 2021, Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings, Applied Clinical Informatics, 12, 737, 10.1055/s-0041-1733851 · DOI
- Overhage, 2020, Physician time spent using the electronic health record during outpatient encounters: a descriptive study, Ann. Intern. Med., 172, 169, 10.7326/M18-3684 · DOI · Paper page
- Baxter, 2021, Measures of electronic health record use in outpatient settings across vendors, J. Am. Med. Inform. Assoc., 28, 955, 10.1093/jamia/ocaa266 · DOI · Paper page
- Barandas, 2020, TSFEL: Time series feature extraction library, SoftwareX, 11, 10.1016/j.softx.2020.100456 · DOI · Paper page
- Maslach, 2001, Job burnout, Annu. Rev. Psychol., 52, 397, 10.1146/annurev.psych.52.1.397 · DOI · Paper page
- Tran, 2019, Burnout and EHR use among academic primary care physicians with varied clinical workloads, AMIA Summits Trans. Sci. Proc., 2019, 136
- Balch, 2010, Surgeon distress as calibrated by hours worked and nights on call, J. Am. Coll. Surg., 211, 609, 10.1016/j.jamcollsurg.2010.06.393 · DOI · Paper page
- S.S. Lou, D. Lew, D. Harford, C. Lu, B.A. Evanoff, J.G. Duncan, and T.G. Kannampallil, A Longitudinal Study of Burnout and Clinical Workload Measured With Electronic Health Record Audit Logs, in: American Medical Informatics Association Annual Symposium. San Diego, CA, 2021.
- Shanafelt, 2015, Impact of organizational leadership on physician burnout and satisfaction, Mayo Clin. Proc., 90, 432, 10.1016/j.mayocp.2015.01.012 · DOI · Paper page
- T.D. Shanafelt, M.S. Makowski, H. Wang, B. Bohman, M. Leonard, R.A. Harrington, L. Minor, M. Trockel, Association of burnout, professional fulfillment, and self-care practices of physician leaders with their independently Rated leadership effectiveness. JAMA Network Open 3(6) (2020) e207961. · DOI · Paper page
- Shanafelt, 2017, Leadership and physician burnout: using the annual review to reduce burnout and promote engagement, Am. J. Med. Qual., 32, 563, 10.1177/1062860617691605 · DOI
- R. Fischer, P. Mattos, C. Teixeira, D.S. Ganzerla, R.G. Rosa, F.A. Bozza, Association of Burnout With Depression and Anxiety in Critical Care Clinicians in Brazil, JAMA Network Open 3(12) (2020) e2030898. · DOI
Cited by
Owens, L. M., Wilda, J. J., Hahn, P. Y., Koehler, T., & Fletcher, J. J. (2024) · Family Practice · doi:10.1093/fampra/cmad092 · Cited by 86
Levin, C., Naimi, E., & Saban, M. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105566 · Cited by 17
Lou, S. S., Liu, H., Harford, D., Lu, C., & Kannampallil, T. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac239 · Cited by 13
Self-citationPerceived value of the electronic health record and its association with physician burnout
Livaudais, M., Deng, D., Frederick, T., Grey-Theriot, F., & Kroth, P. J. (2022) · Applied Clinical Informatics · doi:10.1055/s-0042-1755372 · Cited by 6
Chen, J., Cutrona, S. L., Dharod, A., Bunch, S. C., Foley, K. L., Ostasiewski, B., Hale, E. R., Bridges, A., Moses, A., Donny, E. C., et al. (2023) · JMIR Medical Informatics · doi:10.2196/43097 · Cited by 4
“The metrics described in these studies measure EHR use and associated burden (eg, total time on EHR, time spent using the EHR after hours, time spent on chart review per patient per day) [48-52] nonspecific to CDS tools.”
Background
Metadata
- Authors
- Lou, S. S., Liu, H., Warner, B. C., Harford, D., Lu, C., & Kannampallil, T.
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2022-03
- Volume
- 127
- Pages
- 104015
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W4210433515
- Cited by
- 49
- References
- 32
- FWCI
- 10.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 365 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Predicting physician burnout using clinical activity logs: Model performance and lessons learned. https://app.cochat.ai/papers/10.1016/j.jbi.2022.104015
- The paper
- Lou, S. S., Liu, H., Warner, B. C., Harford, D., Lu, C., & Kannampallil, T. (2022). Predicting physician burnout using clinical activity logs: Model performance and lessons learned. Journal of Biomedical Informatics. https://doi.org/10.1016/j.jbi.2022.104015
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.