Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Raw audit logs provide a comprehensive record of clinicians' activities on an electronic health record (EHR) and have considerable potential for studying clinician behaviors. However, research using raw audit logs is limited because they lack context for clinical tasks, leading to difficulties in interpretation. We describe a novel unsupervised approach using the comparison and visualization of EHR action embeddings to learn context and structure from raw audit log activities. Using a dataset of 15 767 634 raw audit log actions performed by 88 intern physicians over 6 months of EHR use across inpatient and outpatient settings, we demonstrated that embeddings can be used to learn the situated context for EHR-based work activities, identify discrete clinical workflows, and discern activities typically performed across diverse contexts. Our approach represents an important methodological advance in raw audit log research, facilitating the future development of metrics and predictive models to measure clinician behaviors at the macroscale.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Most of these studies have relied on embedding short-run sequences or individual actions within a sequence, without accounting for the temporal progression or dependencies within action sequences.”
From A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions (2026) · cites this paper as [37]
In the citing paper
“However, much of the past work in HCI, or in studies using audit logs 35-37 have not made meaningful attempts to determine whether human action sequences have inherent underlying sequential structure, similar to a language.”
From A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions (2026) · cites this paper as [37]
In the citing paper
“Notably, this threshold has been used in several studies analyzing raw log data.24,25,31,46,47”
From Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling (2025) · cites this paper as [46]
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972
Tai-Seale, M., Olson, C. W., Li, J., Chan, A. S., Morikawa, C., Durbin, M., Wang, W., & Luft, H. S. (2017) · Health Affairs · doi:10.1377/hlthaff.2016.0811 · Cited by 379
Moy, A. J., Schwartz, J. M., Chen, R., Sadri, S., Lucas, E., Cato, K. D., & Rossetti, S. C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa325 · Cited by 284
Westbrook, J. I., Ampt, A., Kearney, L., & Rob, M. I. (2008) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2008.tb01762.x · Cited by 180
Metrics for assessing physician activity using electronic health record log data
Sinsky, C. A., Rule, A., Cohen, G., Arndt, B. G., Shanafelt, T. D., Sharp, C. D., Baxter, S. L., Tai-Seale, M., Yan, S., Chen, Y., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz223 · Cited by 162
Rule, A., Chiang, M. F., & Hribar, M. R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz196 · Cited by 136
Cited by
Harnessing EHR data for health research
Tang, A. S., Woldemariam, S. R., Miramontes, S., Norgeot, B., Oskotsky, T. T., & Sirota, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-03074-8 · Cited by 87
“Understanding and characterizing patterns in the data can give insight into affected populations, practices for treatment prescription and current clinical behaviors 26–28 , as well as identifying potential differences in subgroups and disparities in current care 29–31 .”
Embedding Methods for Electronic Health Record Research
Kauffman, J., Miotto, R., Klang, E., Costa, A. B., Norgeot, B., Žitnik, M., Shameer, K., Fei, W., Nadkarni, G. N., & Glicksberg, B. S. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094729 · Cited by 16
Emerging Domains for Measuring Health Care Delivery With Electronic Health Record Metadata
Tawfik, D. S., Rule, A., Alexanian, A., Cross, D. A., Holmgren, A. J., Lou, S. S., Hinz, E. M., Rose, C. C., Viswanadham, R. V. N., Mishuris, R. G., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/64721 · Cited by 12
Self-citationKim, S., Warner, B. C., Lew, D., Lou, S. S., & Kannampallil, T. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae171 · Cited by 8
Self-citationReimagining clinical AI: from clickstreams to clinical insights with EHR use metadata
Yan, C., Zhang, X., Kannampallil, T., Adler‐Milstein, J., & Chen, Y. (2025) · npj Health Systems · doi:10.1038/s44401-025-00040-5 · Cited by 7
Self-citationMoy, A. J., Cato, K., Kim, E. Y., Withall, J. B., & Rossetti, S. C. (2023) · PubMed Central · Cited by 5
Chen, J., Cutrona, S. L., Dharod, A., Moses, A., Bridges, A., Ostasiewski, B., Foley, K. L., Houston, T. K., iDAPT (Implementation & Informatics Developing Adaptable Processes and Technologies for Cancer Control) Implementation Science Center for Cancer Control, Chen, J., et al. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf050 · Cited by 5
Kim, S., Lou, S. S., Cobb, A. D., Jha, S., & Kannampallil, T. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag163 · Cited by 3
Self-citationUsing Electronic Health Record Metadata to Understand Clinician Work and Behavior
Holmgren, A. J., Steitz, B. D., Lou, S. S., & Apathy, N. C. (2025) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-82971-0_15 · Cited by 2
Self-citationChrimes, D., Thomo, A., Kuo, M.-H., Borycki, E. M., & Kushniruk, A. W. (2026) · JMIR Human Factors · doi:10.2196/86076 · Cited by 1
Hierarchical spatio-temporal graph network for risk prediction
Chen, F., Jia, H. B., Zhou, W., Zhu, L., & Xiao, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-33873-z · Cited by 1
Kim, S., Lou, S., Cobb, A., Jha, S., & Kannampallil, T. (2026) · medRxiv · doi:10.64898/2026.06.24.26356449
Self-citationExploring Task Patterns in EHR Workflows Using Action Sequence Embedding and Graph-Based Analysis
Kim, S., Kannampallil, T., Wick, B. D., Holmgren, A. J., Thombley, R. L., & Lou, S. S. (2025) · Studies in health technology and informatics · doi:10.3233/shti251045
Self-citation
Metadata
- Authors
- Lou, S. S., Liu, H., Harford, D., Lu, C., & Kannampallil, T.
- Affiliations
- Washington University in St. Louis (US)
- Corresponding author
- Kannampallil, Thomas
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-12-08
- Volume
- 30
- Issue
- 3
- Pages
- 539-544
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC9933072
- OpenAlex
- W4310943494
- Cited by
- 13
- References
- 31
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 2 · 2023: 2
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Washington University
- BJC Healthcare Big Ideas Healthcare Innovation Competition
- NIH (5T32GM108539-07)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- electronic health records, audit logs, unsupervised learning, clinician behavior, clinical workflow, medical interns, inpatient and outpatient care, situated context
- MeSH terms
- Humans, Physicians, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.