The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 8 of the 19 sources its claims cite were checked for support.
Scalable and accurate deep learning with electronic health records
Open accessAbstract
Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient's record. We propose a representation of patients' entire raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two US academic medical centers with 216,221 adult patients hospitalized for at least 24 h. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting: in-hospital mortality (area under the receiver operator curve [AUROC] across sites 0.93-0.94), 30-day unplanned readmission (AUROC 0.75-0.76), prolonged length of stay (AUROC 0.85-0.86), and all of a patient's final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed traditional, clinically-used predictive models in all cases. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios. In a case study of a particular prediction, we demonstrate that neural networks can be used to identify relevant information from the patient's chart.
Overview
CoChat Summary
## TL;DR
This retrospective study shows that a single deep learning pipeline using raw EHR data in FHIR format can predict multiple outcomes across two academic hospitals without site-specific feature harmonization. The models outperformed traditional baselines for inpatient mortality, 30-day readmission, long length of stay, and discharge diagnosis prediction.
## Key findings
- The study included 216,221 hospitalizations from 114,003 unique patients; in-hospital death occurred in 2.3% (4,930/216,221), unplanned 30-day readmission in 12.9% (27,918/216,221), and long length of stay in 23.9%.
- For inpatient mortality at 24 h after admission, AUROC was 0.95 at Hospital A and 0.93 at Hospital B, versus 0.85 and 0.86 for the augmented Early Warning Score baseline.
- For 30-day readmission at discharge, AUROC was 0.77 at Hospital A and 0.76 at Hospital B, versus 0.70 and 0.68 for the modified HOSPITAL baseline; for long length of stay at 24 h, AUROC was 0.86 and 0.85, versus 0.76 and 0.74 for the modified Liu baseline.
- For predicting all discharge diagnoses (14,025 ICD-9 codes), weighted AUROC was 0.87/0.86 at admission and 0.90 at discharge for both hospitals; micro-F1 at discharge was 0.41 (Hospital A) and 0.40 (Hospital B).
## Methods
- Design: Retrospective study comparing deep learning models to traditional predictive models across two hospitals, with separate results reported for each site.
- Data/sample: De-identified EHR data from UCSF (2012–2016) and UCM (2009–2016), including adults 18+ with hospitalizations of 24 h or longer; UCM also included free-text notes.
- Analysis: Raw EHR data were converted into time-ordered FHIR-based token sequences and used to train three model types (LSTM, attention-based TANN, and boosted time-based decision stumps), with outputs ensembled; data were split 80/10/10 into development/validation/test sets and evaluated using AUROC, calibration curves, work-up-to-detection ratio, and micro-F1.
## Limitations
- The authors state it is a retrospective study, with “all of the usual limitations.”
- They state that prospective trials are needed to show that accurate predictions improve care.
- They note their approach does not harmonize data between sites, limiting transfer to new sites/cohorts; they also say the methods are computationally intensive and require specialized expertise.
- Evident from the text: the study’s diagnosis task is restricted to ICD-9 codes, and the authors caution that AUROC differences between hospitals, especially regarding notes, cannot be attributed solely to notes because the cohorts differed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
7 Valid · 0 Not valid · 6 Not checked · 33 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We are not aware of a study that predicts as many ICD codes as this study, but our micro-F1 score exceeds that shown on the smaller MIMIC-III dataset when predicting fewer diagnoses (0.40 vs 0.28).68”
§ Scalable and accurate deep learning with electronic health records > Discussion
What the sources say
- Partly supportedcites [68] Perotte, A. et al. Diagnosis code assignment: models and evaluation metrics. J. Am. Med. Inform. Assoc. 21, 231–237 (2014). sourceExperimental — accuracy not yet measured
The Results > Prediction models section reports a 27.6% F-measure for the flat SVM, consistent with the cited 0.28 after rounding. But Materials and methods > Datasets identifies the dataset as MIMIC II, not MIMIC-III; the Results also report a higher 39.5% F-measure for this paper’s hierarchy-based model. No page numbers are available in our copy. (LLM, conf=0.89)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A major challenge in using more of the data available for each patient has been the lack of standards and semantic interoperability of health data from multiple sites.25”
§ Scalable and accurate deep learning with electronic health records > Introduction > Related work
What the sources say
- Partly supportedcites [25] Sun, H. et al. Semantic processing of EHR data for clinical research. J. Biomed. Inform. 58, 247–259 (2015). sourceExperimental — accuracy not yet measured
The Introduction (p.1) and Conclusion (Section 5) of our copy clearly support the semantic-interoperability half of the claim: 'the semantic interoperability between different standards remains a challenge', and 'there is no single standard that could serve as the interoperability hub for all the existing clinical data sources and clinical applications', with the Abstract framing the whole paper as responding to 'a growing need to semantically process and integrate clinical data from different sources for clinical research' (multi-site sources are illustrated in Section 4.1.1: a German hospital Oracle database and an Italian regional IHE QED registry). However, the claim's 'lack of standards' is not what this paper says: the Introduction states that standards such as HL7, openEHR and ISO 13606 HAVE been developed to tackle storage and exchange of clinical data, and the residual problem is interoperability AMONG them plus the absence of one unifying hub standard, not an absence of standards. The claim also generalises the source's specific concern (secondary use/reuse of EHR data for clinical research applications) to 'using more of the data available for each patient'. (LLM, conf=0.70)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the current study, we caution that the differences in AUROC across the two hospitals (one with and one without notes) cannot be ascribed to the presence or absence of notes given the difference in cohorts.45”
§ Scalable and accurate deep learning with electronic health records > Discussion
What the sources say
- Supportedcites [45] Walsh, C. & Hripcsak, G. The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions. J. Biomed. Inform. 52, 418–426 (2014). source
The Discussion states that cohort selection affects discriminatory performance and that comparing performance across studies requires the cohorts to be the same. This supports the citing authors’ caution against attributing their two hospitals’ AUROC difference to the presence or absence of notes when the cohorts also differ; the two-hospital comparison is their own study, not a result attributed to this source. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For classifying all diagnosis codes, the weighted AUROCs at admission were 0.87 for Hospital A and 0.86 for Hospital B.”
§ Scalable and accurate deep learning with electronic health records > Results > Inferring discharge diagnoses
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on 2 assumptions not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For predicting long length of stay, the AUROCs at 24 h after admission were 0.86 (95% CI 0.86–0.87) for Hospital A and 0.85 (95% CI 0.84–0.86) for Hospital B.”
§ Scalable and accurate deep learning with electronic health records > Results > Long length of stay
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
18 more claims
Create a free account to see all 46 claims and to verify the 2 references still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 10
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The most important difference is that the study included co-morbidities, which have been proven to enable better predictions of disease progression trends and treatment outcomes.25,26”
From Development and validation of a screening tool for sepsis without laboratory results in the emergency department: a machine learning study (2025) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For both structured codes and unstructured notes, standardized vocabularies including FHIR [6] and UMLS CUIs [62,63] are introduced to transfer raw tokens into unique phrases to reduce the sparsity of the input data.”
From Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review (2021) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some FEMRs have tackled this problem by combining unstructured EHR data (i.e., text) with structured EMR data to boost performance on specific phenotyping and prediction tasks69,70.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [69]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“To improve performance and provide a degree of interpretability to the model, an attention mechanism is adopted in 13 studies [6,14,27,31,43,57,64,72,76–78,80,88].”
From Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review (2021) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Advances in artificial intelligence (AI) are driving a revolution across science, enabling exciting new results across chemistry,80–82 biology,83,84 and medicine.85–88”
From Artificial Intelligence and Early Detection of Pancreatic Cancer: 2020 Summative Review (2021) · cites this paper as [85]
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 9×
“Patient records vary significantly in length and density of data-points (e.g. vital sign measurements in an intensive care unit vs outpatient clinic), so we formulated three deep learning neural-network model architectures that take advantage of such data in different ways: one based on recurrent neural networks (LSTM)…”
MethodInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
APACHE II: A severity of disease classification system
KNAUS, W. A., DRAPER, E. A., WAGNER, D. P., & ZIMMERMAN, J. E. (1985) · Critical Care Medicine · doi:10.1097/00003246-198510000-00009 · Cited by 13,664
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · cited 1×
“Our central insight is that rather than explicitly harmonizing EHR data, mapping it into a highly curated set of structured predictors variables and then feeding those variables into a statistical model, we can instead learn to simultaneously harmonize inputs and predict medical events through direct feature learning…”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“ecord, which includes clinical free-text notes as well as large amounts of structured and semi-structured data. Because of the availability of Medical Information Mart for Intensive Care (MIMIC) data [48], many prior studies also have focused on ICU patients from a single center [92, 41]; other single-center studies h…”
MethodGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Deep learning emerged as the preferred machine learning approach in machine perception problems ranging from computer vision to speech recognition, but has more recently proven useful in natural language processing, sequence prediction, and mixed modality data settings([36, 16, 21, 56]).”
BackgroundMandell, L. A., Wunderink, R. G., Anzueto, A., Bartlett, J. G., Campbell, G. D., Dean, N. C., Dowell, S. F., File, T. M., Musher, D. M., Niederman, M. S., et al. (2007) · Clinical Infectious Diseases · doi:10.1086/511159 · Cited by 6,202 · cited 1×
“of only 27 variables, rely on traditional generalized linear models, and are built using data at a single center. In clinical practice, simpler models are most commonly deployed, such as the CURB-65 [68, 60], which is a 5-factor model, or single-parameter warning scores [20, 45]. A major challenge in using more of the…”
MethodAPACHE II: a severity of disease classification system.
Knaus, W. A., Draper, E. S., Wagner, D. P., & Zimmerman, J. E. (1985) · PubMed · Cited by 6,034
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652 · cited 1×
“In our case study, the model identified elements of the patient’s history and radiology findings to render its prediction, which are critical data-points that a clinician would also use [41].”
Method
Research works printed but not linked: 6
- Dai, A. M. & Le, Q. V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems 28 (eds Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R.), pp 3079–3087 (Curran Associates, Inc. Red Hook, NY, 2015).
- Aczon, M. et al. Dynamic mortality risk predictions in pediatric critical care using recurrent neural networks. arXiv [stat.ML] (2017).
- Suresh, H. et al. Clinical intervention prediction and understanding using deep networks. arXiv [cs.LG] (PMLR, Los Angeles, CA, USA, 2017).
- Razavian, N., Marcus, J. & Sontag, D. Multi-task prediction of disease onsets from longitudinal laboratory tests. In Proceedings of the 1st Machine Learning for Healthcare Conference, (eds F. Doshi-Velez, J. Fackler, D. Kale and B. Wallace, J. Wiens) Vol. 56, pp 73–100 (PMLR, Los Angeles, CA, 2016).
- Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. arXiv [cs.LG] (2016).
- Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv [cs.CL] (2014).
Cited by
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450
A guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312
“Here, large-scale RNNs are beginning to demonstrate impressive predictive results by combining structured and unstructured data in a semisupervised way 33 .”
BackgroundMethodSelf-citationRajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
Self-citationThe potential for artificial intelligence in healthcare
Davenport, T., & Kalakota, R. (2019) · Future Healthcare Journal · doi:10.7861/futurehosp.6-2-94 · Cited by 3,800
“To our knowledge thus far there have been no jobs eliminated and EHR systems, have been somewhat responsible for the lack of job impact.”
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342
Metadata
- Authors
- Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., Sundberg, P., Yee, H., Zhang, K., Zhang, Y., Flores, G., Duggan, G. E., Irvine, J., Le, Q., Litsch, K., Mossin, A., Tansuwan, J., Wang, D., Wexler, J., Wilson, J., Ludwig, D., Volchenboum, S. L., Chou, K., Pearson, M., Madabushi, S., Shah, N. H., Butte, A. J., Howell, M. D., Cui, C., Corrado, G. S., & Dean, J.
- Affiliations
- Google (United States) (US)
- University of California, San Francisco (US)
- University of Chicago (US)
- Stanford University (US)
- Corresponding author
- Rajkomar, Alvin
- Venue
- npj Digital Medicine
- Published
- 2018-05-08
- Volume
- 1
- Issue
- 1
- Pages
- 18-18
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC6550175
- OpenAlex
- W2784499877
- Cited by
- 2,648
- References
- 72
- Citations by year
- 2026: 494 · 2025: 457 · 2024: 370 · 2023: 280 · 2022: 228 · 2021: 304 · 2020: 275 · 2019: 186 · 2018: 54
- FWCI
- 115.37
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- arXiv (Cornell University) (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- electronic health records, FHIR, deep learning, in-hospital mortality prediction, unplanned hospital readmission, length of hospital stay, hospital discharge diagnoses, clinical notes, data harmonization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Scalable and accurate deep learning with electronic health records. https://app.cochat.ai/papers/10.1038/s41746-018-0029-1
- The paper
- Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., Sundberg, P., Yee, H., Zhang, K., Zhang, Y., Flores, G., Duggan, G. E., Irvine, J., Le, Q., Litsch, K., . . . Dean, J. (2018). Scalable and accurate deep learning with electronic health records. npj Digital Medicine. https://doi.org/10.1038/s41746-018-0029-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.