CoChat

The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 8 of the 19 sources its claims cite were checked for support.

Scalable and accurate deep learning with electronic health records

Open access
Rajkomar, Alvin, Oren, Eyal, Chen, Kai, Dai, Andrew M., Hajaj, Nissan, Hardt, Michaela, Liu, Peter J., Liu, Xiaobing, Marcus, Jake, Sun, Mimi, Sundberg, Patrik, Yee, Hector, Zhang, Kun, Zhang, Yi, Flores, Gerardo, Duggan, Gavin E., Irvine, Jamie, Le, Quoc, Litsch, Kurt, Mossin, Alexander, Tansuwan, Justin, Wang, De, Wexler, James, Wilson, Jimbo, Ludwig, Dana, Volchenboum, Samuel L., Chou, Katherine, Pearson, Michael, Madabushi, Srinivasan, Shah, Nigam H., Butte, Atul J., Howell, Michael D., Cui, Claire, Corrado, Greg S., Dean, Jeffrey · 2018 · npj Digital Medicine

Abstract

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two US academic medical centers with 216,221 adult patients hospitalized for at least 24 h. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting: in-hospital mortality (area under the receiver operator curve [AUROC] across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed traditional, clinically-used predictive models in all cases. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios. In a case study of a particular prediction, we demonstrate that neural networks can be used to identify relevant information from the patient's chart.

Overview

CoChat Summary

## TL;DR

This retrospective study shows that a single deep learning pipeline using raw EHR data in FHIR format can predict multiple outcomes across two academic hospitals without site-specific feature harmonization. The models outperformed traditional baselines for inpatient mortality, 30-day readmission, long length of stay, and discharge diagnosis prediction.

## Key findings

  • The study included 216,221 hospitalizations from 114,003 unique patients; in-hospital death occurred in 2.3% (4,930/216,221), unplanned 30-day readmission in 12.9% (27,918/216,221), and long length of stay in 23.9%.
  • For inpatient mortality at 24 h after admission, AUROC was 0.95 at Hospital A and 0.93 at Hospital B, versus 0.85 and 0.86 for the augmented Early Warning Score baseline.
  • For 30-day readmission at discharge, AUROC was 0.77 at Hospital A and 0.76 at Hospital B, versus 0.70 and 0.68 for the modified HOSPITAL baseline; for long length of stay at 24 h, AUROC was 0.86 and 0.85, versus 0.76 and 0.74 for the modified Liu baseline.
  • For predicting all discharge diagnoses (14,025 ICD-9 codes), weighted AUROC was 0.87/0.86 at admission and 0.90 at discharge for both hospitals; micro-F1 at discharge was 0.41 (Hospital A) and 0.40 (Hospital B).

## Methods

  • Design: Retrospective study comparing deep learning models to traditional predictive models across two hospitals, with separate results reported for each site.
  • Data/sample: De-identified EHR data from UCSF (2012–2016) and UCM (2009–2016), including adults 18+ with hospitalizations of 24 h or longer; UCM also included free-text notes.
  • Analysis: Raw EHR data were converted into time-ordered FHIR-based token sequences and used to train three model types (LSTM, attention-based TANN, and boosted time-based decision stumps), with outputs ensembled; data were split 80/10/10 into development/validation/test sets and evaluated using AUROC, calibration curves, work-up-to-detection ratio, and micro-F1.

## Limitations

  • The authors state it is a retrospective study, with “all of the usual limitations.”
  • They state that prospective trials are needed to show that accurate predictions improve care.
  • They note their approach does not harmonize data between sites, limiting transfer to new sites/cohorts; they also say the methods are computationally intensive and require specialized expertise.
  • Evident from the text: the study’s diagnosis task is restricted to ICD-9 codes, and the authors caution that AUROC differences between hospitals, especially regarding notes, cannot be attributed solely to notes because the cohorts differed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-018-0029-1

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 This boxplot displays the amount of data (on a log scale) in the EHR, along with its temporal variation across the course of an admission. We define a token as a single data element in the electronic health record, like a medication name, at a specific point in time. Each token is considered as a potential predictor by the deep learning model. The line within the boxplot represents the median, the box represents the interquartile range (IQR), and the whiskers are 1.5 times the IQR. The number of tokens increased steadily from admission to discharge. At discharge, the median number of tokens for Hospital A was 86,477 and for Hospital B was 122,961
Fig. 1 This boxplot displays the amount of data (on a log scale) in the EHR, along with its temporal variation across the course of an admission. We define a token as a single data element in the electronic health record, like a medication name, at a specific point in time. Each token is considered as a potential predictor by the deep learning model. The line within the boxplot represents the median, the box represents the interquartile range (IQR), and the whiskers are 1.5 times the IQR. The number of tokens increased steadily from admission to discharge. At discharge, the median number of tokens for Hospital A was 86,477 and for Hospital B was 122,961CC BY · PubMed Central
Fig. 2 The area under the receiver operating characteristic curves are shown for predictions of inpatient mortality made by deep learning and baseline models at 12 h increments before and after hospital admission. For inpatient mortality, the deep learning model achieves higher discrimination at every prediction time compared to the baseline for both the University of California, San Francisco (UCSF) and University of Chicago Medicine (UCM) cohorts. Both models improve in the first 24 h, but the deep learning model achieves a similar level of accuracy approximately 24 h earlier for UCM and even 48 h earlier for UCSF. The error bars represent the bootstrapped 95% confidence interval
Fig. 2 The area under the receiver operating characteristic curves are shown for predictions of inpatient mortality made by deep learning and baseline models at 12 h increments before and after hospital admission. For inpatient mortality, the deep learning model achieves higher discrimination at every prediction time compared to the baseline for both the University of California, San Francisco (UCSF) and University of Chicago Medicine (UCM) cohorts. Both models improve in the first 24 h, but the deep learning model achieves a similar level of accuracy approximately 24 h earlier for UCM and even 48 h earlier for UCSF. The error bars represent the bootstrapped 95% confidence intervalCC BY · PubMed Central
Fig. 3 The patient record shows a woman with metastatic breast cancer with malignant pleural effusions and empyema. The patient timeline at the top of the figure contains circles for every time-step for which at least a single token exists for the patient, and the horizontal lines show the data type. There is a close-up view of the most recent data points immediately preceding a prediction made 24 h after admission. We trained models for each data type and highlighted in red the tokens which the models attended to—the non-highlighted text was not attended to but is shown for context. The models pick up features in the medications, nursing flowsheets, and clinical notes relevant to the prediction
Fig. 3 The patient record shows a woman with metastatic breast cancer with malignant pleural effusions and empyema. The patient timeline at the top of the figure contains circles for every time-step for which at least a single token exists for the patient, and the horizontal lines show the data type. There is a close-up view of the most recent data points immediately preceding a prediction made 24 h after admission. We trained models for each data type and highlighted in red the tokens which the models attended to—the non-highlighted text was not attended to but is shown for context. The models pick up features in the medications, nursing flowsheets, and clinical notes relevant to the predictionCC BY · PubMed Central
Fig. 4 Data from each health system were mapped to an appropriate FHIR (Fast Healthcare Interoperability Resources) resource and placed in temporal order. This conversion did not harmonize or standardize the data from each health system other than map them to the appropriate resource. The deep learning model could use all data available prior to the point when the prediction was made. Therefore, each prediction, regardless of the task, used the same data
Fig. 4 Data from each health system were mapped to an appropriate FHIR (Fast Healthcare Interoperability Resources) resource and placed in temporal order. This conversion did not harmonize or standardize the data from each health system other than map them to the appropriate resource. The deep learning model could use all data available prior to the point when the prediction was made. Therefore, each prediction, regardless of the task, used the same dataCC BY · PubMed Central

Verification

Claims the paper makes

7 Valid · 0 Not valid · 6 Not checked · 33 Own results and interpretations

18 more claims

Create a free account to see all 46 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 16
  • Probably not a substantial claim. 10

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

8 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 72 references

Research works printed but not linked: 6

  1. Dai, A. M. & Le, Q. V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems 28 (eds Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R.), pp 3079–3087 (Curran Associates, Inc. Red Hook, NY, 2015).
  2. Aczon, M. et al. Dynamic mortality risk predictions in pediatric critical care using recurrent neural networks. arXiv [stat.ML] (2017).
  3. Suresh, H. et al. Clinical intervention prediction and understanding using deep networks. arXiv [cs.LG] (PMLR, Los Angeles, CA, USA, 2017).
  4. Razavian, N., Marcus, J. & Sontag, D. Multi-task prediction of disease onsets from longitudinal laboratory tests. In Proceedings of the 1st Machine Learning for Healthcare Conference, (eds F. Doshi-Velez, J. Fackler, D. Kale and B. Wallace, J. Wiens) Vol. 56, pp 73–100 (PMLR, Los Angeles, CA, 2016).
  5. Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. arXiv [cs.LG] (2016).
  6. Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv [cs.CL] (2014).

Cited by

See all 2,648

Metadata

Authors
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., Sundberg, P., Yee, H., Zhang, K., Zhang, Y., Flores, G., Duggan, G. E., Irvine, J., Le, Q., Litsch, K., Mossin, A., Tansuwan, J., Wang, D., Wexler, J., Wilson, J., Ludwig, D., Volchenboum, S. L., Chou, K., Pearson, M., Madabushi, S., Shah, N. H., Butte, A. J., Howell, M. D., Cui, C., Corrado, G. S., & Dean, J.
Affiliations
  • Google (United States) (US)
  • University of California, San Francisco (US)
  • University of Chicago (US)
  • Stanford University (US)
Corresponding author
Rajkomar, Alvin
Venue
npj Digital Medicine
Published
2018-05-08
Volume
1
Issue
1
Pages
18-18
ISSN
2398-6352 (ISSN-L)
OpenAlex
W2784499877
Cited by
2,648
References
72
Citations by year
2026: 494 · 2025: 457 · 2024: 370 · 2023: 280 · 2022: 228 · 2021: 304 · 2020: 275 · 2019: 186 · 2018: 54
FWCI
115.37
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • arXiv (Cornell University) (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
electronic health records, FHIR, deep learning, in-hospital mortality prediction, unplanned hospital readmission, length of hospital stay, hospital discharge diagnoses, clinical notes, data harmonization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Scalable and accurate deep learning with electronic health records. https://app.cochat.ai/papers/10.1038/s41746-018-0029-1
The paper
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., Sundberg, P., Yee, H., Zhang, K., Zhang, Y., Flores, G., Duggan, G. E., Irvine, J., Le, Q., Litsch, K., . . . Dean, J. (2018). Scalable and accurate deep learning with electronic health records. npj Digital Medicine. https://doi.org/10.1038/s41746-018-0029-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.