CoChat
Sign Up
Cited by 58Electronic Health Records SystemsMedical Coding and Health InformationNursing Diagnosis and Documentation

Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures

Open access
Rule, Adam, Melnick, Edward R, Apathy, Nate C·2022·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocac177PMC PMC9748581
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: The aim of this article is to compare the aims, measures, methods, limitations, and scope of studies that employ vendor-derived and investigator-derived measures of electronic health record (EHR) use, and to assess measure consistency across studies. Materials and Methods: We searched PubMed for articles published between July 2019 and December 2021 that employed measures of EHR use derived from EHR event logs. We coded the aims, measures, methods, limitations, and scope of each article and compared articles employing vendor-derived and investigator-derived measures. Results: One hundred and two articles met inclusion criteria; 40 employed vendor-derived measures, 61 employed investigator-derived measures, and 1 employed both. Studies employing vendor-derived measures were more likely than those employing investigator-derived measures to observe EHR use only in ambulatory settings (83% vs 48%, P = .002) and only by physicians or advanced practice providers (100% vs 54% of studies, P < .001). Studies employing vendor-derived measures were also more likely to measure durations of EHR use (P < .001 for 6 different activities), but definitions of measures such as time outside scheduled hours varied widely. Eight articles reported measure validation. The reported limitations of vendor-derived measures included measure transparency and availability for certain clinical settings and roles. Discussion: Vendor-derived measures are increasingly used to study EHR use, but only by certain clinical roles. Although poorly validated and variously defined, both vendor- and investigator-derived measures of EHR time are widely reported. Conclusion: The number of studies using event logs to observe EHR use continues to grow, but with inconsistent measure definitions and significant differences between studies that employ vendor-derived and investigator-derived measures.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocac177

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most prior studies in this field used aggregated metrics based on predefined categories of micro-level actions or vendor-derived metrics.

    “Most prior studies in this field used aggregated metrics based on predefined categories of micro-level actions or vendor-derived metrics.”

    From Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling (2025) · cites this paper as [33]

    The Results (Differences in study measures; page unavailable) describe counts of actions or activities and durations derived from EHR logs. The Results (Differences in study methods; page unavailable) also describe mapping individual actions to activities, while the Abstract Results report that 40 of 102 articles used vendor-derived measures. The paper does not establish that most studies used aggregated metrics based on predefined action categories; only 16 of 37 articles reporting durations for specific activities described how actions were mapped. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • There are known limitations to the use of EHR metadata, especially with approximations of time-based metrics.

    “There are also known limitations to the use of EHR metadata, especially with approximations of time-based metrics, [30]”

    From Effect of an Electronic Health Record-Based Intervention on Documentation Practices (2024) · cites this paper as [30]

  • There is currently no consensus on how to define the idle time threshold, and accurately identifying active EHR use remains an open research question.

    “There is currently no consensus on how to define this threshold, and accurately identifying active EHR use remains an open research question.33”

    From Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling (2025) · cites this paper as [33]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 127 · Cited by 58

Research works printed but not linked: 116

  1. [1]Apathy NC, Hare AJ, Fendrich S, et al Early changes in billing and notes after evaluation and management guideline change. Ann Intern Med 2022; 175 (4): 499– 504. 35188791
  2. [2]Nath B, Williams B, Jeffery MM, et al Trends in electronic health record inbox messaging during the COVID-19 pandemic in an ambulatory practice network in New England. JAMA Netw Open 2021; 4 (10): e2131490. 34636917 PMC8511977
  3. [3]Holmgren AJ, Downing NL, Tang M, et al Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use. J Am Med Inform Assoc 2022; 29 (3): 453– 60. 34888680 PMC8689796
  4. [5]Rittenberg E, Liebman JB, Rexrode KM. Primary care physician gender and electronic health record workload. J Gen Intern Med. doi: 10.1007/s11606-021-07298-z PMC9550938 34993875 · DOI
  5. [6]Adler-Milstein J, Zhao W, Willard-Grace R, et al Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. J Am Med Inform Assoc 2020; 27 (4): 531– 8. 32016375 PMC7647261
  6. [7]Tai-Seale M, Dillon EC, Yang Y, et al Physicians’ well-being linked to in-basket messages generated by algorithms in electronic health records. Health Aff (Millwood) 2019; 38 (7): 1073– 8. 31260371
  7. [8]Yan Q, Jiang Z, Harbin Z, et al Exploring the relationship between electronic health records and provider burnout: a systematic review. J Am Med Inform Assoc 2021; 28 (5): 1009– 21. 33659988 PMC8068439
  8. [9]Kannampallil T, Abraham J, Lou SS, et al Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects. J Am Med Inform Assoc 2021; 28 (5): 1032– 7. 33355360 PMC8068434
  9. [10]Zheng K, Guo MH, Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (5): 704– 10. 21527407 PMC3168304
  10. [11]Rule A, Chiang MF, Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods. J Am Med Inform Assoc 2020; 27 (3): 480– 90. 31750912 PMC7025338
  11. [12]Adler-Milstein J, Adelman JS, Tai-Seale M, et al EHR audit logs: a new goldmine for health services research? J Biomed Inform 2020; 101: 103343. 31821887
  12. [13]Baxter SL, Apathy NC, Cross DA, et al Measures of electronic health record use in outpatient settings across vendors. J Am Med Inform Assoc 2021; 28 (5): 955– 9. 33211862 PMC8068413
  13. [14]Cohen GR, Boi J, Johnson C, et al Measuring time clinicians spend using EHRs in the inpatient setting: a national, mixed-methods study. J Am Med Inform Assoc 2021; 28 (8): 1676– 82. 33899105 PMC8324233
  14. [15]Lourie EM, Stevens LA, Webber EC. Measuring success: perspectives from three optimization programs on assessing impact in the age of burnout. JAMIA Open 2020; 3 (4): 492– 5. 33655200 PMC7903326
  15. [16]Hron JD, Lourie E. Have you got the time? Challenges using vendor electronic health record metrics of provider efficiency. J Am Med Inform Assoc 2020; 27 (4): 644– 6. 32016394 PMC7647287
  16. [17]Melnick ER, Sinsky CA, Krumholz HM. Implementing measurement science for electronic health record use. JAMA 2021; 325 (21): 2149– 50. 33818587
  17. [18]Rotenstein LS, Sinsky C, Cassel CK. How to measure progress in addressing physician well-being: beyond burnout. JAMA 2021; 326 (21): 2129– 30. 34757427
  18. [19]Melnick ER, Ong SY, Fong A, et al Characterizing physician EHR use with vendor derived data: a feasibility study and cross-sectional analysis. J Am Med Inform Assoc 2021; 28 (7): 1383– 92. 33822970 PMC8279798
  19. [20]Bright TJ, Wong A, Dhurjati R, et al Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012; 157 (1): 29– 43. 22751758
  20. [22]Middleton B, Sittig DF, Wright A. Clinical decision support: a 25 year retrospective and a 25 year vision. Yearb Med Inform 2016; 5 (Suppl 1): S103– 116. 27488402 PMC5171504
  21. [23]Sinsky CA, Rule A, Cohen G, et al Metrics for assessing physician activity using electronic health record log data. J Am Med Inform Assoc 2020; 27 (4): 639– 43. 32027360 PMC7075531
  22. [24]Giliberto JP, Ator G, Carroll TL, et al National trends in daily ambulatory electronic health record use by otolaryngologists. Laryngoscope 2021; 131 (5): 975– 81. 32876342
  23. [25]Overhage JM, Johnson KB. Pediatrician electronic health record time use for outpatient encounters. Pediatrics 2020; 146 (6): e20194017. 33139456
  24. [26]Watson MD, Elhage SA, Scully C, et al Electronic health record usage among nurse practitioners, physician assistants, and junior residents. J Am Assoc Nurse Pract 2021; 33 (3): 200– 4. 32740334
  25. [27]Holmgren AJ, Downing NL, Bates DW, et al Assessment of electronic health record use between US and Non-US health systems. JAMA Intern Med 2021; 181 (2): 251– 9. 33315048 PMC7737152
  26. [28]Melnick ER, Fong A, Nath B, et al Analysis of electronic health record use and clinical productivity and their association with physician turnover. JAMA Netw Open 2021; 4 (10): e2128790. 34636911 PMC8511970
  27. [29]Overhage JM, McCallie D. Physician time spent using the electronic health record during outpatient encounters: a descriptive study. Ann Intern Med 2020; 172 (3): 169– 74. 31931523
  28. [30]Lam C, Shumaker K, Butt M, et al Impact of medical scribes on physician and patient satisfaction in dermatology. Arch Dermatol Res 2022; 314 (1): 71– 6. 33683446
  29. [31]Maloney SR, Peterson S, Kao AM, et al Surgery resident time consumed by the electronic health record. J Surg Educ 2020; 77 (5): 1056– 62. 32305335
  30. [32]Holmgren AJ, Lindeman B, Ford EW. Resident physician experience and duration of electronic health record use. Appl Clin Inform 2021; 12 (4): 721– 8. 34348409 PMC8354368
  31. [33]Hilliard RW, Haskell J, Gardner RL. Are specific elements of electronic health record use associated with clinician burnout more than others? J Am Med Inform Assoc 2020; 27 (9): 1401– 10. 32719859 PMC7647296
  32. [34]Beiser M, Lu V, Paul S, et al Electronic health record usage patterns: assessing telemedicine’s impact on the provider experience during the COVID-19 pandemic. Telemed J E Health 2021; 27 (8): 934– 8. 33600728
  33. [35]Wandell GM, Giliberto JP. Otolaryngology resident clinic participation and attending electronic health record efficiency—a user activity logs study. Laryngoscope Investig Otolaryngol 2021; 6 (5): 968– 74. PMC8513420 34667838
  34. [37]Moore C, Valenti A, Robinson E, et al Using log data to measure provider EHR activity at a cancer center during rapid telemedicine deployment. Appl Clin Inform 2021; 12 (3): 629– 36. 34261172 PMC8279818
  35. [38]Hollister-Meadows L, Richesson RL, De Gagne J, et al Association between evidence-based training and clinician proficiency in electronic health record use. J Am Med Inform Assoc 2021; 28 (4): 824– 31. 33575787 PMC7973447
  36. [39]Anderson J, Leubner J, Brown SR. EHR overtime: an analysis of time spent after hours by family physicians. Fam Med 2020; 52 (2): 135– 7. 32050270
  37. [40]Baxter SL, Gali HE, Chiang MF, et al Promoting quality face-to-face communication during ophthalmology encounters in the electronic health record era. Appl Clin Inform 2020; 11 (1): 130– 41. 32074650 PMC7030957
  38. [41]Jhaveri P, Abdulahad D, Fogel B, et al Impact of scribe intervention on documentation in an outpatient pediatric primary care practice. Acad Pediatr 2022; 22 (2): 289– 95. 34020102
  39. [42]Khairat S, Zalla L, Gartland A, et al Association between proficiency and efficiency in electronic health records among pediatricians at a major academic health system. Front Digit Health 2021; 3: 689646. 34713161 PMC8521844
  40. [43]Nguyen OT, Turner K, Apathy NC, et al Primary care physicians’ electronic health record proficiency and efficiency behaviors and time interacting with electronic health records: a quantile regression analysis. J Am Med Inform Assoc 2022; 29 (3): 461– 71. 34897493 PMC8800512
  41. [44]Baxter SL, Gali HE, Mehta MC, et al Multicenter analysis of electronic health record use among ophthalmologists. Ophthalmology 2021; 128 (1): 165– 6. 32525047 PMC7719588
  42. [45]Sieja A, Whittington MD, Patterson VP, et al The influence of a Sprint optimization and training intervention on time spent in the electronic health record (EHR). JAMIA Open 2021; 4 (3): ooab073. 34435176 PMC8382277
  43. [46]Mosquera MJ, Ward HB, Holland C, et al Using objective clinical metrics to understand the relationship between the electronic health record and physician well-being: observational pilot study. BJPsych Open 2021; 7 (5): e174. 34545781 PMC8485348
  44. [49]Tait SD, Oshima SM, Ren Y, et al Electronic health record use by sex among physicians in an academic health care system. JAMA Intern Med 2021; 181 (2): 288– 90. 33284311 PMC7851731
  45. [50]Saag HS, Shah K, Jones SA, et al Pajama time: working after work in the electronic health record. J Gen Intern Med 2019; 34 (9): 1695– 6. 31073856 PMC6712097
  46. [51]Wilkinson K, Seo K, Pierce R, et al Electronic Medical Record Specialty Group Comparison by Multinomial Logistic Regression. In 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI). Victoria, BC, Canada: IEEE; 2021: 415– 21. doi: 10.1109/ICHI52183.2021.00067. · DOI
  47. [52]Verma G, Ivanov A, Benn F, et al Analyses of electronic health records utilization in a large community hospital. PLoS One 2020; 15 (7): e0233004. 32609757 PMC7329072
  48. [53]Tajirian T, Stergiopoulos V, Strudwick G, et al The influence of electronic health record use on physician burnout: cross-sectional survey. J Med Internet Res 2020; 22 (7): e19274. 32673234 PMC7392132
  49. [54]Simpson JR, Lin C-T, Sieja A, et al Optimizing the electronic health record: an inpatient sprint addresses provider burnout and improves electronic health record satisfaction. J Am Med Inform Assoc 2021; 28 (3): 628– 31. 33029643 PMC7936398
  50. [55]Krawiec C, Stetter C, Kong L, et al Impact of patient census and admission mortality on pediatric intensive care unit attending electronic health record activity: a preliminary study. Appl Clin Inform 2020; 11 (2): 226– 34. 32215894 PMC7096317
  51. [56]Olson E, Rushnell C, Khan A, et al Emergency medicine residents spend over 7.5 months of their 3-year residency on the electronic health record. AEM Educ Train 2021; 5 (4): e10697. 34693185 PMC8517589
  52. [57]Watson MD, Elhage SA, Green JM, et al Surgery residents spend nearly 8 months of their 5-year training on the electronic health record (EHR). J Surg Educ 2020; 77 (6): e237–44. 32654998
  53. [58]Loszko A, Watson M, Khan A, et al Acute care surgeons spend more time than general surgeons on the electronic health record (EHR). Am Surg 2021; 31348211061102. doi: 10.1177/00031348211061102. 34933572 · DOI
  54. [60]Neprash HT, Everhart A, McAlpine D, et al Measuring primary care exam length using electronic health record data. Med Care 2021; 59 (1): 62– 6. 33301282
  55. [61]Long CP, Tai-Seale M, El-Kareh R, et al Electronic health record use among ophthalmology residents while on call. J Acad Ophthalmol (2017) 2020; 12 (2): e143–50. 33274310 PMC7710324
  56. [62]Lieu TA, Warton EM, East JA, et al Evaluation of attention switching and duration of electronic inbox work among primary care physicians. JAMA Netw Open 2021; 4 (1): e2031856. doi: 10.1001/jamanetworkopen.2020.31856. 33475754 PMC7821028 · DOI
  57. [63]Perros I, Yan X, Jones JB, et al Using the PARAFAC2 tensor factorization on EHR audit data to understand PCP desktop work. J Biomed Inform 2020; 101: 103312. 31627022
  58. [65]Chen JS, Hribar MR, Goldstein IH, et al Electronic health record note review in an outpatient specialty clinic: who is looking? JAMIA Open 2021; 4 (3): ooab044. 34345803 PMC8325486
  59. [66]Attipoe S, Huang Y, Schweikhart S, et al Factors associated with electronic health record usage among primary care physicians after hours: retrospective cohort study. JMIR Hum Factors 2019; 6 (3): e13779. 31573912 PMC6819131
  60. [67]Sinha A, Shanafelt TD, Trockel M, et al Novel nonproprietary measures of ambulatory electronic health record use associated with physician work exhaustion. Appl Clin Inform 2021; 12 (3): 637– 46. 34261173 PMC8279819
  61. [68]Ganguli I, Sheridan B, Gray J, et al Physician work hours and the gender pay gap—evidence from primary care. N Engl J Med 2020; 383 (14): 1349– 57. 32997909 PMC10854207
  62. [69]Micek MA, Arndt B, Tuan W-J, et al Physician burnout and timing of electronic health record use. ACI Open 2020; 4 (1): e1– e8. PMC10553367 37800093
  63. [70]Akbar F, Mark G, Warton EM, et al Physicians’ electronic inbox work patterns and factors associated with high inbox work duration. J Am Med Inform Assoc 2021; 28 (5): 923– 30. 33063087 PMC8068414
  64. [71]Akbar F, Mark G, Prausnitz S, et al Physician stress during electronic health record inbox work: in situ measurement with wearable sensors. JMIR Med Inform 2021; 9 (4): e24014. 33908888 PMC8116996
  65. [72]Hribar MR, Dusek HL, Goldstein IH, et al Methods for large-scale quantitative analysis of scribe impacts on clinical documentation. AMIA Annu Symp Proc 2020; 2020: 573– 82. 33936431 PMC8075531
  66. [73]Mohler SA, Mears SC, Kathiresan AR, et al Electronic medical record audit time logs as a measure of preoperative work before total joint arthroplasty. J Arthroplasty 2021; 36 (7): 2250– 3. 33618957 PMC9454105
  67. [74]Zhu X, Tu S-P, Sewell D, et al Measuring electronic communication networks in virtual care teams using electronic health records access-log data. Int J Med Inform 2019; 128: 46– 52. 31160011
  68. [75]Hatch B, Tillotson C, Huguet N, et al Implementation and adoption of a health insurance support tool in the electronic health record: a mixed methods analysis within a randomized trial. BMC Health Serv Res 2020; 20 (1): 428. 32414376 PMC7227079
  69. [76]Steitz BD, Unertl KM, Levy MA. An analysis of electronic health record work to manage asynchronous clinical messages among breast cancer care teams. Appl Clin Inform 2021; 12 (4): 877– 87. 34528233 PMC8443401
  70. [77]Dusek HL, Goldstein IH, Rule A, et al Clinical documentation during scribed and non-scribed ophthalmology office visits. Ophthalmol Sci 2021; 1 (4): 100088. 35059685 PMC8765735
  71. [79]Sinha A, Stevens LA, Su F, et al Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings. Appl Clin Inform 2021; 12 (4): 737– 44. 34380167 PMC8357459
  72. [80]Soleimani H, Adler-Milstein J, Cucina RJ, et al Automating measurement of trainee work hours. J Hosp Med 2021; 16 (7): 404– 8. 33929943
  73. [81]Gong JJ, Soleimani H, Murray SG, et al Characterizing styles of clinical note production and relationship to clinical work hours among first-year residents. J Am Med Inform Assoc 2021; 29 (1): 120– 7. 34963142 PMC8714268
  74. [82]Schumacher DJ, Wu DTY, Meganathan K, et al A feasibility study to attribute patients to primary interns on inpatient ward teams using electronic health record data. Acad Med 2019; 94 (9): 1376– 83. 31460936
  75. [83]Dziorny AC, Orenstein EW, Lindell RB, et al Pediatric trainees systematically under-report duty hour violations compared to electronic health record defined shifts. PLoS One 2019; 14 (12): e0226493. 31830096 PMC6907762
  76. [84]Kim C, Lehmann CU, Hatch D, et al Provider networks in the neonatal intensive care unit associate with length of stay. IEEE Conf Collab Internet Comput 2019; 2019: 127– 34. 32637942 PMC7339831
  77. [85]Li P, Chen B, Rhodes E, et al Measuring collaboration through concurrent electronic health record usage: network analysis study. JMIR Med Inform 2021; 9 (9): e28998. 34477566 PMC8449299
  78. [86]Chen Y, Lehmann CU, Hatch LD, et al Modeling care team structures in the neonatal intensive care unit through network analysis of EHR audit logs. Methods Inf Med 2019; 58 (4-05): 109– 23. 32170716 PMC7356932
  79. [87]Yan C, Zhang X, Gao C, et al Collaboration structures in COVID-19 critical care: retrospective network analysis study. JMIR Hum Factors 2021; 8 (1): e25724. 33621187 PMC7942392
  80. [88]Wang M, Pantell MS, Gottlieb LM, et al Documentation and review of social determinants of health data in the EHR: measures and associated insights. J Am Med Inform Assoc 2021; 28 (12): 2608– 16. 34549294 PMC8633631
  81. [89]Mannering H, Yan C, Gong Y, et al Assessing neonatal intensive care unit structures and outcomes before and during the COVID-19 pandemic: network analysis study. J Med Internet Res 2021; 23 (10): e27261. 34637393 PMC8530253
  82. [90]Payne TH, Keller C, Arora P, et al Writing practices associated with electronic progress notes and the preferences of those who read them: descriptive study. J Med Internet Res 2021; 23 (10): e30165. 34612825 PMC8529482
  83. [91]Chen B, Alrifai W, Gao C, et al Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning. J Am Med Inform Assoc 2021; 28 (6): 1168– 77. 33576432 PMC8200270
  84. [92]Despins LA, Kim JH, Deroche C, et al Factors influencing how intensive care unit nurses allocate their time. West J Nurs Res 2019; 41 (11): 1551– 75. 30632467
  85. [93]Horn JJ, Doucette JN, Sweeney NL. An essential clinical dataset intervention for nursing documentation of a pediatric admission history database. J Pediatr Nurs 2021; 59: 110– 4. 33845323
  86. [94]Sutton DE, Fogel JR, Giard AS, et al Defining an essential clinical dataset for admission patient history to reduce nursing documentation burden. Appl Clin Inform 2020; 11 (3): 464– 73. 32643778 PMC7344356
  87. [95]Chi J, Bentley J, Kugler J, et al How are medical students using the electronic health record (EHR)?: an analysis of EHR use on an inpatient medicine rotation. PLoS One 2019; 14 (8): e0221300. 31419265 PMC6697335
  88. [96]Kim S, Ku S, Kim T, et al Effective use of mobile electronic medical records by medical interns in real clinical settings: mixed methods study. JMIR Mhealth Uhealth 2020; 8 (12): e23622. 33320098 PMC7772071
  89. [97]Jung KY, Kim S, Kim K, et al Frequent mobile electronic medical records users respond more quickly to emergency department consultation requests: retrospective quantitative study. JMIR Mhealth Uhealth 2020; 8 (2): e14487. 32130157 PMC7055754
  90. [98]Mai MV, Orenstein EW, Manning JD, et al Attributing patients to pediatric residents using electronic health record features augmented with audit logs. Appl Clin Inform 2020; 11 (3): 442– 51. 32583389 PMC7314655
  91. [99]Jones B, Zhang X, Malin BA, et al Learning tasks of pediatric providers from electronic health record audit logs. AMIA Annu Symp Proc 2020; 2020: 612– 8. 33936435 PMC8075537
  92. [100]Gellert G, Crouch JF, Gibson LA, et al An evaluation of the clinical and financial value of work station single sign-on in 19 hospitals. Perspect Health Inf Manag 2019; 16: 1a. PMC6669364 31423117
  93. [101]Steitz BD, Sulieman L, Wright A, et al Association of immediate release of test results to patients with implications for clinical workflow. JAMA Netw Open 2021; 4 (10): e2129553. 34661667 PMC8524306
  94. [102]Lin JA, Pierce L, Murray SG, et al Estimation of surgical resident duty hours and workload in real time using electronic health record data. J Surg Educ 2021; 78 (6): e232– 8. 34507910 PMC9335013
  95. [103]Hong P, Herigon JC, Uptegraft C, et al Use of clinical data to augment healthcare worker contact tracing during the COVID-19 pandemic. J Am Med Inform Assoc 2021; 29 (1): 142– 8. 34623426 PMC8714290
  96. [104]Kawamoto K, Kukhareva P, Shakib JH, et al Association of an electronic health record add-on app for neonatal bilirubin management with physician efficiency and care quality. JAMA Netw Open 2019; 2 (11): e1915343. 31730181 PMC6902796
  97. [105]Ruppel H, Bhardwaj A, Manickam RN, et al Assessment of electronic health record search patterns and practices by practitioners in a large integrated health care system. JAMA Netw Open 2020; 3 (3): e200512. 32142128 PMC7060491
  98. [106]Sulieman L, Steitz B, Rosenbloom ST. Analysis of employee patient portal use and electronic health record access at an academic medical center. Appl Clin Inform 2020; 11 (3): 433– 41. 32557441 PMC7299683
  99. [107]Adler-Milstein J, Wang MD. The impact of transitioning from availability of outside records within electronic health records to integration of local and outside records within electronic health records. J Am Med Inform Assoc 2020; 27 (4): 606– 12. 32134449 PMC7647298
  100. [108]Nestor JG, Fedotov A, Fasel D, et al An electronic health record (EHR) log analysis shows limited clinician engagement with unsolicited genetic test results. JAMIA Open 2021; 4 (1): ooab014. 33709066 PMC7935499
  101. [109]Stevens LA, Pageler NM, Hahn JS. Improved medical student engagement with EHR documentation following the 2018 centers for medicare and medicaid billing changes. Appl Clin Inform 2021; 12 (3): 582– 8. 34233368 PMC8263129
  102. [110]Henriksen BS, Goldstein IH, Rule A, et al Electronic health records in ophthalmology: source and method of documentation. Am J Ophthalmol 2020; 211: 191– 9. 31811860 PMC7073273
  103. [111]Zallman L, Altman W, Chu L, et al Do medical scribes help primary care providers respond more quickly to out-of-visit tasks? J Am Board Fam Med 2021; 34 (1): 70– 7. 33452084
  104. [112]Steitz BD, Levy MA. Evaluating the scope of clinical electronic messaging to coordinate care in a breast cancer cohort. Stud Health Technol Inform 2019; 264: 808– 12. 31438036 PMC12038858
  105. [113]Chu L, Toomay SM, Ginnings CS, et al “D/C the CC (Carbon Copy)”—improving the EHR signal-to-noise ratio for clinicians by selective feature de-implementation. Stud Health Technol Inform 2019; 264: 1915– 6. 31438405
  106. [114]Lutz MK, Matthews LA, Tuan W-J, et al Did you get my message? Trends and predictors of inflammatory bowel disease patient-to-provider electronic messaging. Inflamm Bowel Dis 2022; 28 (6): 963– 6. 34757419
  107. [115]Steitz BD, Unertl KM, Levy MA. Characterizing communication patterns among members of the clinical care team to deliver breast cancer treatment. J Am Med Inform Assoc 2020; 27 (2): 236– 43. 31682267 PMC7647266
  108. [116]Rule A, Bedrick S, Chiang MF, et al Length and redundancy of outpatient progress notes across a decade at an academic medical center. JAMA Netw Open 2021; 4 (7): e2115334. 34279650 PMC8290305
  109. [118]Rule A, Goldstein IH, Chiang MF, et al Clinical documentation as end-user programming. Proc SIGCHI Conf Hum Factor Comput Syst 2020; 2020: 1– 13. PMC7901830 33629079
  110. [119]DiAngi YT, Stevens LA, Halpern-Felsher B, et al Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers’ perceived control over their workload in the EHR. JAMIA Open 2019; 2 (2): 222– 30. 31984357 PMC6952029
  111. [120]Marmor RA, Clay B, Millen M, et al The impact of physician EHR usage on patient satisfaction. Appl Clin Inform 2018; 9 (1): 11– 4. 29298451 PMC5801886
  112. [121]Krawiec C, Marker C, Stetter C, et al Tracking resident pre-rounding electronic health record usage. Int J Health Care Qual Assur 2019; 32 (3): 611– 20. 31018798
  113. [122]Aziz F, Talhelm L, Keefer J, et al Vascular surgery residents spend one fifth of their time on electronic health records after duty hours. J Vasc Surg 2019; 69 (5): 1574– 9. 31010521
  114. [123]Tran B, Lenhart A, Ross R, et al Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc 2019; 2019: 136– 44. 31258965 PMC6568076
  115. [125]Collins S, Couture B, Kang MJ, et al Quantifying and visualizing nursing flowsheet documentation burden in acute and critical care. AMIA Annu Symp Proc 2018; 2018: 348– 57. 30815074 PMC6371331
  116. [126]Kannampallil T, Adler-Milstein J. Using electronic health record audit log data for research:insights from early efforts. J Am Med Inform Assoc 2023; 30 (1): 167–71. PMC9748594 36173351

Authors
Rule, A., Melnick, E. R., & Apathy, N. C.
Affiliations
  • University of Wisconsin–Madison (US)
  • Yale University (US)
  • Regenstrief Institute (US)
Corresponding author
Rule, Adam
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2022-09-29
Volume
30
Issue
1
Pages
144-154
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocac177
PMC
PMC9748581
OpenAlex
W4297964841
Cited by
58
References
127
Citations by year
2026: 12 · 2025: 22 · 2024: 14 · 2023: 8 · 2022: 2
FWCI
1.68
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • IUScholarWorks (Indiana University) (submittedVersion, cc-by)
Funders
  • Agency for Healthcare Research and Quality (T32-HS026116-04)
  • National Institute on Drug Abuse, American Medical Association, and Agency for Healthcare Research and Quality
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
  • Health Sciences › Nursing › Issues, ethics and legal aspects › Nursing Diagnosis and Documentation
Keywords
electronic health records, event logs, measure validation, outpatient care
MeSH terms
Commerce, Humans, Physicians, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review