A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions
Open accessThe DOI matches its Crossref record; 29 of the 64 references it cites were found in a registry or an index, and 2 are not checked yet; 5 of the 26 sources its claims cite were checked for support.
Abstract
Objective Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (e.g., total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions. Materials and Methods Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under two encodings: (1) symbolic field-based tokens and (2) semantic natural language audit log action descriptions. A first-order Markov model, which used only the immediate prior action, served as baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and two temporally distinct out-of-sample (OOS) periods. Results In the early period test set, the semantic LLM achieved the highest accuracy (0.7418, 95%CI [0.7415–0.7420]) compared to the symbolic LLM (0.3838, 95% CI: 0.3836–0.3840) and Markov baseline (0.4553, 95%CI [0.4551–0.4555]). The semantic approach was also robust to temporal drift in EHR interaction patterns seen in the two OOS periods (semantic LLM vs Markov accuracy, OOS-1: 0.6509 vs 0.3169; OOS-2: 0.6232 vs 0.2648). Discussion and Conclusions Semantic LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy, and demonstrated robustness to temporal drift, suggesting that LLMs can effectively model complex clinical workflows.
Verified 2026-11-19 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
This study fine-tuned Llama-3-8B on >295 million inpatient EHR audit log actions to predict the next clinician action, comparing a semantic (word-based) encoding against a symbolic (field-based) encoding and a first-order Markov baseline. The semantic LLM achieved the highest next-action prediction accuracy and degraded less than the Markov baseline across two temporally distinct out-of-sample periods.
Key findings
- On the early-period test set (20,461,603 actions), the word-based LLM reached accuracy 0.7418 (95% CI 0.7415–0.7420), versus the field-based LLM at 0.3838 (0.3836–0.3840) and the Markov baseline at 0.4553 (0.4551–0.4555).
- The semantic LLM was more robust to temporal drift: OOS-1 accuracy 0.6509 (0.6508–0.6511) vs Markov 0.3169 (0.3168–0.3170); OOS-2 accuracy 0.6232 (0.6231–0.6233) vs Markov 0.2648 (0.2647–0.2649).
- Restricting to likely user-initiated actions (≥1 second apart) preserved the advantage: early-period 0.6596 (0.6594–0.6599) vs Markov 0.3416 (0.3413–0.3419); OOS-1 0.6005 (0.6003–0.6006) vs 0.0430 (0.0429–0.0431); OOS-2 0.5723 (0.5722–0.5725) vs 0.0843 (0.0843–0.0844).
- The symbolic LLM underperformed even the Markov baseline, which the authors attribute to its inability to leverage pretrained linguistic knowledge and to hyperparameters/tokenizer design better suited to natural language.
Methods
- Retrospective study at the inpatient settings of a single large academic health system (St. Louis, Missouri), using Epic Clarity audit logs from January 1, 2019 to June 1, 2024; sequences comprised the 30 minutes of audit log actions preceding each inpatient medication or procedure order (2,478,503 orders; 295,912,142 audit log actions).
- Temporal data splitting: early period (2019.01–2020.07) randomly split 70:10:20 for training, early stopping, and evaluation; two out-of-sample cohorts (OOS-1: 2021.06–2022.05; OOS-2: 2023.06–2024.05) used only for evaluation.
- Llama-3-8B fine-tuned with QLoRA under two encodings — field-based symbolic METRIC_ID tokens with a custom tokenizer, and word-based natural language METRIC_DESCRIPTION tokens with the pretrained tokenizer (generated text mapped to the nearest valid action description via TF-IDF character n-gram embeddings and cosine similarity) — compared against a first-order Markov bigram baseline; metrics were next-action accuracy and accuracy@top-5 with 95% CIs from 1,000 bootstrap resamples.
Limitations
- Stated: single academic health system with inpatient audit logs, limiting generalizability to other institutions, EHR configurations, specialties, or ambulatory settings; models trained only on action sequences preceding orders; the word-based LLM was evaluated on substantially fewer actions than the symbolic and Markov approaches (e.g., 13,397,377 vs 20,461,603 actions, ~65%, in the early-period test set) because subword tokenization and truncation to the most recent tokens dropped the earliest actions of long sequences; dependence on vendor-provided action descriptions that may be inconsistent, ambiguous, or institution-specific; substantially greater computational resources than the simpler baselines; and changes in clinician composition in the OOS periods that may have affected performance changes.
- Evident from the text: the semantic model's raw free-text output rarely produced an exact valid audit log action label, requiring retrieval-based mapping to a constrained vocabulary; the authors also note that some temporal degradation may reflect shifts in user mix rather than EHR interface or workflow changes alone, and that performance was not stratified by clinician role, specialty, care setting, or order type.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
2 Valid · 4 Not valid · 4 Not checked · 43 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, much of the past work in HCI, or in studies using audit logs 35-37 have not made meaningful attempts to determine whether human action sequences have inherent underlying sequential structure, similar to a language.”
What the sources say
- Not matchedcites [35] 35 Jones, B., Zhang, X., Malin, B. A. & Chen, Y. Learning Tasks of Pediatric Providers from Electronic Health Record Audit Logs. AMIA Annu Symp Proc 2020 , 612-618 (2020).
We searched by its title and found no matching paper.
- Could not verifycites [36] 36 Chen, B. et al. Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning. Journal of the American Medical Informatics Association 28 , 1168-1177 (2021). sourceExperimental — accuracy not yet measured
The Introduction describes this paper’s method for finding patterns in EHR audit-log event sequences, and ‘Learning tasks from sessions’ describes how it groups sessions into tasks. Neither passage assesses whether much prior HCI or audit-log work investigated language-like sequential structure; searches for that comparison and for ‘language’ found no such assessment in our copy. (LLM, conf=0.72)
- Not checked — no full text of the sourcecites [37] 37 Lou, S. S., Liu, H., Harford, D., Lu, C. & Kannampallil, T. Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach. J Am Med Inform Assoc 30 , 539-544 (2023). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Most of these studies have relied on embedding short-run sequences or individual actions within a sequence, without accounting for the temporal progression or dependencies within action sequences.”
What the sources say
- Not checked — no full text of the sourcecites [34] 34 Perros, I., Yan, X., Jones, J. B., Sun, J. & Stewart, W. F. Using the PARAFAC2 tensor factorization on EHR audit data to understand PCP desktop work. Journal of Biomedical Informatics 101 , 103312 (2020). source
- Not matchedcites [35] 35 Jones, B., Zhang, X., Malin, B. A. & Chen, Y. Learning Tasks of Pediatric Providers from Electronic Health Record Audit Logs. AMIA Annu Symp Proc 2020 , 612-618 (2020).
We searched by its title and found no matching paper.
- Could not verifycites [36] 36 Chen, B. et al. Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning. Journal of the American Medical Informatics Association 28 , 1168-1177 (2021). sourceExperimental — accuracy not yet measured
The “Learning tasks from sessions” section describes this paper’s own session-level embedding and clustering method, not how most prior NLP studies represented audit-log sequences. Searches for prior NLP approaches, short-run or individual-action embeddings, and claims about their treatment of temporal dependencies found no passage in our copy addressing that comparison. The indexed passages provide no page numbers. (LLM, conf=0.72)
- Not checked — no full text of the sourcecites [37] 37 Lou, S. S., Liu, H., Harford, D., Lu, C. & Kannampallil, T. Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach. J Am Med Inform Assoc 30 , 539-544 (2023). source
- Not checked — no full text of the sourcecites [38] 38 Kim, S., Lou, S., Baratta, L. & Kannampallil, T. Classifying Clinical Work Settings Using EHR Audit Logs: A Machine Learning Approach. The American Journal of Managed Care 29 (2023). source
- Not matchedcites [39] 39 Kim, S. et al. in MEDINFO 2025-HealthcareSmart x MedicineDeep: Proceedings of the 20th World Congress on Medical and Health Informatics.
We searched by its title and found no matching paper.
- Could not verifycites [40] 40 Zhang, X., Yan, C., Malin, B. A., Patel, M. B. & Chen, Y. Predicting next-day discharge via electronic health record access logs. J Am Med Inform Assoc 28 , 2670-2680 (2021). sourceExperimental — accuracy not yet measured
Our copy of Zhang et al. (2021) surveys prior EHR audit-log work only by application area — health process modeling, clinical task/workflow analysis, time-motion studies, user-user interaction structure, and privacy audits (INTRODUCTION; DISCUSSION AND CONCLUSION) — and nowhere characterizes those studies' modeling of action sequences, embeddings, or handling of temporal progression/dependencies. The only statements about temporal modelling concern the authors' own models: the best model used counts of each of 819 interaction types in the recent 24 h, and the bidirectional RETAIN-based RNN built because 'LGBM ... is challenging ... to embed the temporal relationship in user-EHR interactions' did not improve AUROC (Model development; Overall performance). The limitations paragraph likewise refers to this study's own focus on raw interaction sequences, not prior NLP work (DISCUSSION AND CONCLUSION). Searched: 'embedding short-run sequences / temporal progression', 'prior studies sequence modeling NLP audit log', 'bag-of-words frequencies ignoring order' — no passage addresses the claim. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“free-text generation using the semantic approach rarely produced the exact valid audit log action label.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the early-period test set, the LLM achieved an accuracy of 0.6596 (95% CI: 0.6594-0.6599) compared to the Markov baseline (0.3416, 95% CI: 0.3413-0.3419).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This performance advantage persisted across two temporally distinct out-of-sample periods”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
26 more claims
Create a free account to see all 53 claims and to verify the 2 references still open.
- It may state two separate claims as one. 13
- Probably not a substantial claim. 12
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Evaluation on user-initiated actions
“defined as actions occurring greater than one second after a previous action event.”
§ Tables and Figures
“Action sequences was restricted to consecutive actions that occurred at least 1 second apart”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Tables and Figures
“| OOS-1 | Markov Baseline | 0.0430 (0.0429-0.0431) | 0.1916 (0.1915-0.1017) |”
§ Tables and Figures
“95% confidence intervals (CI) from 1000 bootstrap resampling iterations are reported across all metrics.”
Citations
Cites
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
On understanding computers and cognition: A new foundation for design
Winograd, T., & Flores, F. (1987) · Artificial Intelligence · doi:10.1016/0004-3702(87)90026-9 · Cited by 3,677
Use of Electronic Health Records in U.S. Hospitals
Jha, A. K., DesRoches, C. M., Campbell, E. G., Donelan, K., Rao, S. R., Ferris, T. G., Shields, A., Rosenbaum, S., & Blumenthal, D. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0900592 · Cited by 1,473
The keystroke-level model for user performance time with interactive systems
Card, S. K., Moran, T. P., & Newell, A. (1980) · Communications of the ACM · doi:10.1145/358886.358895 · Cited by 1,174
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationAdler-Milstein, J., Zhao, W., Willard-Grace, R., Knox, M., & Grumbach, K. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz220 · Cited by 356
The effects of domain knowledge on search tactic formulation
Wildemuth, B. M. (2004) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.10367 · Cited by 341
The evolution of cognitive structures and processes.: (426532004-001)
Hayes-Roth, B. (1976) · PsycEXTRA Dataset · doi:10.1037/e426532004-001 · Cited by 232
Cited by
Clinical Audit Logs as Multi-Axial Traces of Care DeliveryPreprint
Eberhard, B. W., Apathy, N. C., & Johnson, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.15397
“[6, 18–22] A multi-axial reading adds a more specific prediction about where that value should concentrate.”
Metadata
- Authors
- Kim, S., Lou, S., Cobb, A., Jha, S., & Kannampallil, T.
- Affiliations
- Institute for Informatics, Data Science, and Biostatistics
- Washington University in St. Louis (US)
- SRI International (US)
- Corresponding author
- Kannampallil, Thomas
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2026-06-26
- OpenAlex
- W7166135565
- Cited by
- 0
- References
- 55
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- medRxiv (acceptedVersion)
- Funders
- Agency for Healthcare Research and Quality (R01HS029020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Keywords
- large language models, sequence modeling, next action prediction, temporal drift, inpatient settings, Llama-3-8B
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions. https://app.cochat.ai/papers/10.64898/2026.06.24.26356449
- The paper
- Kim, S., Lou, S., Cobb, A., Jha, S., & Kannampallil, T. (2026). A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions. medRxiv. https://doi.org/10.64898/2026.06.24.26356449
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.