CoChat

Predicting physician burnout using clinical activity logs: Model performance and lessons learned

Open access
Lou, Sunny S., Liu, Hanyang, Warner, Benjamin C., Harford, Derek, Lu, Chenyang, Kannampallil, Thomas · 2022 · Journal of Biomedical Informatics
Cited by 49

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.

Abstract

Background: Burnout is a significant public health concern affecting more than half of the healthcare workforce; however, passive screening tools to detect burnout are lacking. We investigated the ability of machine learning (ML) techniques to identify burnout using passively collected electronic health record (EHR)-based audit log data. Method: Physician trainees participated in a longitudinal study where they completed monthly burnout surveys and provided access to their EHR-based audit logs. Using the monthly burnout scores as the target outcome, we trained ML models using combinations of features derived from audit log data—aggregate measures of clinical workload, time series-based temporal measures of EHR use, and the baseline burnout score. Five ML models were constructed to predict burnout as a continuous score: penalized linear regression, support vector machine, neural network, random forest, and gradient boosting machine. Results: 88 trainee physicians participated and completed 416 surveys; >10 million audit log actions were collected (Mean [Standard Deviation]=25,691 [14,331] actions per month, per physician). The workload feature set predicted burnout score with a mean absolute error (MAE) of 0.602 (95% Confidence Interval (CI), 0.412–0.826), and was able to predict burnout status with an average AUROC of 0.595 (95% CI 0.355–0.808) and average accuracy 0.567 (95% CI 0.393–0.742). The temporal feature set had a similar performance, with MAE 0.596 (95% CI 0.391–0.826), and AUROC 0.581 (95% CI 0.343–0.790). The addition of the baseline burnout score to the workload features improved the model performance to a mean AUROC of 0.829 (95% CI 0.607–0.996) and mean accuracy of 0.781 (95% CI 0.587–0.936); however, this performance was not meaningfully different than using the baseline burnout score alone. Conclusions: Current findings illustrate the complexities of predicting burnout exclusively based on clinical work activities as captured in the EHR, highlighting its multi-factorial and individualized nature. Future prediction studies of burnout should account for individual factors (e.g., resilience, physiological measurements such as sleep) and associated system-level factors (e.g., leadership).

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2022.104015

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 32 references

Research works printed but not linked: 37

  1. Dyrbye, 2014, Burnout among US medical students, residents, and early career physicians relative to the general US population, Acad. Med., 89, 443, 10.1097/ACM.0000000000000134 · DOI · Paper page
  2. West, 2011, Quality of life, burnout, educational debt, and medical knowledge among internal medicine residents, JAMA, 306, 952, 10.1001/jama.2011.1247 · DOI · Paper page
  3. Allegra, 2005, Prevalence of burnout in the US oncology community: results of a 2003 survey, J. Oncol. Pract., 1, 140, 10.1200/jop.2005.1.4.140 · doi:10.1200/jop.2005.1.4.140 · Paper page
  4. Embriaco, 2007, High level of burnout in intensivists: prevalence and associated factors, Am. J. Respiratory Crit. Care Med., 175, 686, 10.1164/rccm.200608-1184OC · DOI · Paper page
  5. Shanafelt, 2011, Special report: suicidal ideation among American surgeons, Arch. Surg., 146, 54, 10.1001/archsurg.2010.292 · DOI · Paper page
  6. van der Heijden, 2008, Suicidal thoughts among medical residents with burnout, Arch. Suicide Res., 12, 344, 10.1080/13811110802325349 · DOI · Paper page
  7. Oreskovich, 2012, Prevalence of alcohol use disorders among American surgeons, Arch. Surg., 147, 168, 10.1001/archsurg.2011.1481 · DOI · Paper page
  8. J. Klein, K. Grosse Frie, K. Blum, O. von dem Knesebeck, Burnout and perceived quality of care among German clinicians in surgery, Int. J. Quality Health Care 22(6) (2010) 525-530. · DOI · Paper page
  9. C.S. Dewa, D. Loong, S. Bonato, L. Trojanowski, The relationship between physician burnout and quality of healthcare in terms of safety and acceptability: a systematic review. BMJ Open 7(6) (2017) e015141. · DOI · Paper page
  10. Panagioti, 2018, Association between physician burnout and patient safety, professionalism, and patient satisfaction: a systematic review and meta-analysis, JAMA Int. Med., 178, 1317, 10.1001/jamainternmed.2018.3713 · DOI · Paper page
  11. M. Del Carmen, J. Herman, S. Rao, M.K. Hidrue, D. Ting, S.R. Lehrhoff, S. Lenz, J. Heffernan, and T.G. Ferris, Trends and factors associated with physician burnout at a multispecialty academic faculty practice organization, JAMA Network Open 2(3) (2019) e190554. · DOI · Paper page
  12. Shanafelt, 2014, Satisfaction with work-life balance and the career and retirement plans of US oncologists, J. Clin. Oncol., 32, 1127, 10.1200/JCO.2013.53.4560 · DOI · Paper page
  13. Han, 2019, Estimating the attributable cost of physician burnout in the United States, Ann. Intern. Med., 170, 784, 10.7326/M18-1422 · DOI · Paper page
  14. Kannampallil, 2021, Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects, J. Am. Med. Inform. Assoc., 28, 1032, 10.1093/jamia/ocaa305 · DOI
  15. Ballermann, 2011, Validation of the Work Observation Method By Activity Timing (WOMBAT) method of conducting time-motion observations in critical care settings: an observational study, BMC Med. Inf. Decis. Making, 11, 32, 10.1186/1472-6947-11-32 · DOI · Paper page
  16. DesRoches, 2008, Registered nurses' use of electronic health records: findings from a national survey, Medscape J. Med., 10, 164
  17. Ouyang, 2016, Internal medicine resident computer usage: an electronic audit of an inpatient service, JAMA Int. Med., 176, 252, 10.1001/jamainternmed.2015.6831 · DOI · Paper page
  18. Adler-Milstein, 2020, EHR audit logs: a new goldmine for health services research?, J. Biomed. Inform., 101, 10.1016/j.jbi.2019.103343 · DOI · Paper page
  19. N. Ratanawongsa, G.Y. Matta, F.B. Bohsali, M.S. Chisolm, Reducing misses and near misses related to multitasking on the electronic health record: observational study and qualitative analysis, JMIR Human Factors 5(1) (2018) e4. · DOI
  20. T.A. Lieu, E.M. Warton, J.A. East, M.F. Moeller, S. Prausnitz, M. Ballesca, G. Mark, F. Akbar, S. Awsare, Y.-F.I. Chen, Evaluation of Attention Switching and Duration of Electronic Inbox Work Among Primary Care Physicians. JAMA Network Open 4(1) (2021) e2031856. · DOI
  21. Kroth, 2018, The electronic elephant in the room: Physicians and the electronic health record, JAMIA Open, 1, 49, 10.1093/jamiaopen/ooy016 · DOI · Paper page
  22. Tai-Seale, 2019, Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records, Health Aff., 38, 1073, 10.1377/hlthaff.2018.05509 · DOI · Paper page
  23. Adler-Milstein, 2020, Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians, J. Am. Med. Inform. Assoc., 27, 531, 10.1093/jamia/ocz220 · DOI · Paper page
  24. Collins, 2015, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement, J. Brit. Surg., 102, 148, 10.1002/bjs.9736 · DOI · Paper page
  25. Maslach, 1986, Vol. 21
  26. Sinha, 2021, Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings, Applied Clinical Informatics, 12, 737, 10.1055/s-0041-1733851 · DOI
  27. Overhage, 2020, Physician time spent using the electronic health record during outpatient encounters: a descriptive study, Ann. Intern. Med., 172, 169, 10.7326/M18-3684 · DOI · Paper page
  28. Baxter, 2021, Measures of electronic health record use in outpatient settings across vendors, J. Am. Med. Inform. Assoc., 28, 955, 10.1093/jamia/ocaa266 · DOI · Paper page
  29. Barandas, 2020, TSFEL: Time series feature extraction library, SoftwareX, 11, 10.1016/j.softx.2020.100456 · DOI · Paper page
  30. Maslach, 2001, Job burnout, Annu. Rev. Psychol., 52, 397, 10.1146/annurev.psych.52.1.397 · DOI · Paper page
  31. Tran, 2019, Burnout and EHR use among academic primary care physicians with varied clinical workloads, AMIA Summits Trans. Sci. Proc., 2019, 136
  32. Balch, 2010, Surgeon distress as calibrated by hours worked and nights on call, J. Am. Coll. Surg., 211, 609, 10.1016/j.jamcollsurg.2010.06.393 · DOI · Paper page
  33. S.S. Lou, D. Lew, D. Harford, C. Lu, B.A. Evanoff, J.G. Duncan, and T.G. Kannampallil, A Longitudinal Study of Burnout and Clinical Workload Measured With Electronic Health Record Audit Logs, in: American Medical Informatics Association Annual Symposium. San Diego, CA, 2021.
  34. Shanafelt, 2015, Impact of organizational leadership on physician burnout and satisfaction, Mayo Clin. Proc., 90, 432, 10.1016/j.mayocp.2015.01.012 · DOI · Paper page
  35. T.D. Shanafelt, M.S. Makowski, H. Wang, B. Bohman, M. Leonard, R.A. Harrington, L. Minor, M. Trockel, Association of burnout, professional fulfillment, and self-care practices of physician leaders with their independently Rated leadership effectiveness. JAMA Network Open 3(6) (2020) e207961. · DOI · Paper page
  36. Shanafelt, 2017, Leadership and physician burnout: using the annual review to reduce burnout and promote engagement, Am. J. Med. Qual., 32, 563, 10.1177/1062860617691605 · DOI
  37. R. Fischer, P. Mattos, C. Teixeira, D.S. Ganzerla, R.G. Rosa, F.A. Bozza, Association of Burnout With Depression and Anxiety in Critical Care Clinicians in Brazil, JAMA Network Open 3(12) (2020) e2030898. · DOI

Cited by

See all 49

Metadata

Authors
Lou, S. S., Liu, H., Warner, B. C., Harford, D., Lu, C., & Kannampallil, T.
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2022-03
Volume
127
Pages
104015
ISSN
1532-0464 (ISSN-L)
OpenAlex
W4210433515
Cited by
49
References
32
FWCI
10.07
Type
journal article
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Predicting physician burnout using clinical activity logs: Model performance and lessons learned. https://app.cochat.ai/papers/10.1016/j.jbi.2022.104015
The paper
Lou, S. S., Liu, H., Warner, B. C., Harford, D., Lu, C., & Kannampallil, T. (2022). Predicting physician burnout using clinical activity logs: Model performance and lessons learned. Journal of Biomedical Informatics. https://doi.org/10.1016/j.jbi.2022.104015

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.