CoChat

A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions

Open access

The DOI matches its Crossref record; 29 of the 64 references it cites were found in a registry or an index, and 2 are not checked yet; 5 of the 26 sources its claims cite were checked for support.

Abstract

Objective Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (e.g., total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions. Materials and Methods Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under two encodings: (1) symbolic field-based tokens and (2) semantic natural language audit log action descriptions. A first-order Markov model, which used only the immediate prior action, served as baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and two temporally distinct out-of-sample (OOS) periods. Results In the early period test set, the semantic LLM achieved the highest accuracy (0.7418, 95%CI [0.7415–0.7420]) compared to the symbolic LLM (0.3838, 95% CI: 0.3836–0.3840) and Markov baseline (0.4553, 95%CI [0.4551–0.4555]). The semantic approach was also robust to temporal drift in EHR interaction patterns seen in the two OOS periods (semantic LLM vs Markov accuracy, OOS-1: 0.6509 vs 0.3169; OOS-2: 0.6232 vs 0.2648). Discussion and Conclusions Semantic LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy, and demonstrated robustness to temporal drift, suggesting that LLMs can effectively model complex clinical workflows.

Verified 2026-11-19 · 2 references not checked yet

Overview

CoChat Summary

TL;DR

This study fine-tuned Llama-3-8B on >295 million inpatient EHR audit log actions to predict the next clinician action, comparing a semantic (word-based) encoding against a symbolic (field-based) encoding and a first-order Markov baseline. The semantic LLM achieved the highest next-action prediction accuracy and degraded less than the Markov baseline across two temporally distinct out-of-sample periods.

Key findings

  • On the early-period test set (20,461,603 actions), the word-based LLM reached accuracy 0.7418 (95% CI 0.7415–0.7420), versus the field-based LLM at 0.3838 (0.3836–0.3840) and the Markov baseline at 0.4553 (0.4551–0.4555).
  • The semantic LLM was more robust to temporal drift: OOS-1 accuracy 0.6509 (0.6508–0.6511) vs Markov 0.3169 (0.3168–0.3170); OOS-2 accuracy 0.6232 (0.6231–0.6233) vs Markov 0.2648 (0.2647–0.2649).
  • Restricting to likely user-initiated actions (≥1 second apart) preserved the advantage: early-period 0.6596 (0.6594–0.6599) vs Markov 0.3416 (0.3413–0.3419); OOS-1 0.6005 (0.6003–0.6006) vs 0.0430 (0.0429–0.0431); OOS-2 0.5723 (0.5722–0.5725) vs 0.0843 (0.0843–0.0844).
  • The symbolic LLM underperformed even the Markov baseline, which the authors attribute to its inability to leverage pretrained linguistic knowledge and to hyperparameters/tokenizer design better suited to natural language.

Methods

  • Retrospective study at the inpatient settings of a single large academic health system (St. Louis, Missouri), using Epic Clarity audit logs from January 1, 2019 to June 1, 2024; sequences comprised the 30 minutes of audit log actions preceding each inpatient medication or procedure order (2,478,503 orders; 295,912,142 audit log actions).
  • Temporal data splitting: early period (2019.01–2020.07) randomly split 70:10:20 for training, early stopping, and evaluation; two out-of-sample cohorts (OOS-1: 2021.06–2022.05; OOS-2: 2023.06–2024.05) used only for evaluation.
  • Llama-3-8B fine-tuned with QLoRA under two encodings — field-based symbolic METRIC_ID tokens with a custom tokenizer, and word-based natural language METRIC_DESCRIPTION tokens with the pretrained tokenizer (generated text mapped to the nearest valid action description via TF-IDF character n-gram embeddings and cosine similarity) — compared against a first-order Markov bigram baseline; metrics were next-action accuracy and accuracy@top-5 with 95% CIs from 1,000 bootstrap resamples.

Limitations

  • Stated: single academic health system with inpatient audit logs, limiting generalizability to other institutions, EHR configurations, specialties, or ambulatory settings; models trained only on action sequences preceding orders; the word-based LLM was evaluated on substantially fewer actions than the symbolic and Markov approaches (e.g., 13,397,377 vs 20,461,603 actions, ~65%, in the early-period test set) because subword tokenization and truncation to the most recent tokens dropped the earliest actions of long sequences; dependence on vendor-provided action descriptions that may be inconsistent, ambiguous, or institution-specific; substantially greater computational resources than the simpler baselines; and changes in clinician composition in the OOS periods that may have affected performance changes.
  • Evident from the text: the semantic model's raw free-text output rarely produced an exact valid audit log action label, requiring retrieval-based mapping to a constrained vocabulary; the authors also note that some temporal degradation may reflect shifts in user mix rather than EHR interface or workflow changes alone, and that performance was not stratified by clinician role, specialty, care setting, or order type.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.64898/2026.06.24.26356449

Cites

29 of 64 references verified0 citation errors · 33 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Overview of the experimental pipeline for modeling clinician interaction sequences. The EHR action history of 1,006,261 orders in the previous 30 minute were collected from 2019.01-2020.07 and structured into action sequences. The samples were then split for model fine-tuning and within-period evaluation. An LLM was fine-tuned to predict the next EHR given previous actions, at each time step within sequences. The model's inference capabilities were initially tested on a held-out set coming from the same period of the fine-tuning set ('earlyperiod test set'). Additional external validation is conducted using EHR action sequences from two separate sets from later time periods (OOS-1: 2021.06-2022.05, OOS-2: 2023.06-2024.05) to test model generalizability. s. en 5)
Figure 1 Overview of the experimental pipeline for modeling clinician interaction sequences. The EHR action history of 1,006,261 orders in the previous 30 minute were collected from 2019.01-2020.07 and structured into action sequences. The samples were then split for model fine-tuning and within-period evaluation. An LLM was fine-tuned to predict the next EHR given previous actions, at each time step within sequences. The model's inference capabilities were initially tested on a held-out set coming from the same period of the fine-tuning set ('earlyperiod test set'). Additional external validation is conducted using EHR action sequences from two separate sets from later time periods (OOS-1: 2021.06-2022.05, OOS-2: 2023.06-2024.05) to test model generalizability. s. en 5)From the PDF
Figure 2 Overview of the LLM pipeline that learns and infers next EHR action in a sequence at each step. The core components are: (i) sequence tokenization of structured audit logs, (ii) transfer learning from pretrained LLMs (e.g., Llama-3), and (iii) parameter-efficient fine-tuning using model quantization and low-rank adaptation (QLoRA). ce g
Figure 2 Overview of the LLM pipeline that learns and infers next EHR action in a sequence at each step. The core components are: (i) sequence tokenization of structured audit logs, (ii) transfer learning from pretrained LLMs (e.g., Llama-3), and (iii) parameter-efficient fine-tuning using model quantization and low-rank adaptation (QLoRA). ce gFrom the PDF
Figure 3 Example of field-based (symbolic) versus word-based (semantic) tokenization of clinician action sequences. The top row shows original action descriptions. The middle row shows symbolic tokenization (field-based), preserving action sequence structure. The bottom row shows word-based tokenization, enabling semantic modeling while fragmenting individual actions into multiple natural language tokens. w
Figure 3 Example of field-based (symbolic) versus word-based (semantic) tokenization of clinician action sequences. The top row shows original action descriptions. The middle row shows symbolic tokenization (field-based), preserving action sequence structure. The bottom row shows word-based tokenization, enabling semantic modeling while fragmenting individual actions into multiple natural language tokens. wFrom the PDF

Verification

Claims the paper makes

2 Valid · 4 Not valid · 4 Not checked · 43 Own results and interpretations

26 more claims

Create a free account to see all 53 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 13
  • Probably not a substantial claim. 12
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 64 references

Cited by

See all 1

Metadata

Authors
Kim, S., Lou, S., Cobb, A., Jha, S., & Kannampallil, T.
Affiliations
  • Institute for Informatics, Data Science, and Biostatistics
  • Washington University in St. Louis (US)
  • SRI International (US)
Corresponding author
Kannampallil, Thomas
Venue
medRxiv
Publisher
openRxiv
Published
2026-06-26
OpenAlex
W7166135565
Cited by
0
References
55
Type
posted content
Language
en
Open access
Locations
medRxiv (acceptedVersion)
Funders
Agency for Healthcare Research and Quality (R01HS029020)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
Keywords
large language models, sequence modeling, next action prediction, temporal drift, inpatient settings, Llama-3-8B

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions. https://app.cochat.ai/papers/10.64898/2026.06.24.26356449
The paper
Kim, S., Lou, S., Cobb, A., Jha, S., & Kannampallil, T. (2026). A language model framework for sequence modeling of EHR audit logs to characterize clinician-EHR interactions. medRxiv. https://doi.org/10.64898/2026.06.24.26356449

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.