Cited by 928Machine Learning in HealthcareArtificial Intelligence in HealthcareECG Monitoring and Analysis

Using recurrent neural network models for early detection of heart failure onset

Open access
Choi, Edward, Schuetz, Andy, Stewart, Walter F, Sun, Jimeng·2017·Journal of the American Medical Informatics Association

Abstract

Objective: We explored whether use of deep learning to model temporal relations among events in electronic health records (EHRs) would improve model performance in predicting initial diagnosis of heart failure (HF) compared to conventional methods that ignore temporality. Materials and Methods: Data were from a health system's EHR on 3884 incident HF cases and 28 903 controls, identified as primary care patients, between May 16, 2000, and May 23, 2013. Recurrent neural network (RNN) models using gated recurrent units (GRUs) were adapted to detect relations among time-stamped events (eg, disease diagnosis, medication orders, procedure orders, etc.) with a 12- to 18-month observation window of cases and controls. Model performance metrics were compared to regularized logistic regression, neural network, support vector machine, and K-nearest neighbor classifier approaches. Results: Using a 12-month observation window, the area under the curve (AUC) for the RNN model was 0.777, compared to AUCs for logistic regression (0.747), multilayer perceptron (MLP) with 1 hidden layer (0.765), support vector machine (SVM) (0.743), and K-nearest neighbor (KNN) (0.730). When using an 18-month observation window, the AUC for the RNN model increased to 0.883 and was significantly higher than the 0.834 AUC for the best of the baseline methods (MLP). Conclusion: Deep learning models adapted to leverage temporal relations appear to improve performance of models for detection of incident heart failure with a short observation window of 12-18 months.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocw112

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. [11]Cho K Van Merrienboer B Gulcehre C Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Empirical Methods in Natural Language Processing (EMNLP). 2014:1724–1734. Doha, Qatar.
  2. [14]Krizhevsky A Sutskever I Hinton G Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS) 2012:1106–1114. Lake Tahoe, Nevada, United States.
  3. [17]Lee H Pham P Largman Y Ng A Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems (NIPS) 2009;1096–1104. Vancouver, British Columbia, Canada.
  4. [27]Jozefowicz R Zaremba W Sutskever I An empirical exploration of recurrent network architectures. In International Conference on Machine Learning (ICML). 2015:2342–2350. Lille, France.
  5. [40]Schulam P Saria S A framework for individualizing predictions of disease trajectories by exploiting multi-resolution structure. In Advances in Neural Information Processing Systems (NIPS) 2015:748–756. Montreal, Quebec, Canada.
  6. [43]Goldberg Y A primer on neural network models for natural language processing. In arXiv preprint arXiv:1510.00726 2015. · arXiv
  7. [45]Bergstra J Breuleux O Bastien F Theano: a CPU and GPU Math Expression Compiler. In Python for Scientific Computing Conference 2010.
  8. [50]Medi-Span Electronic Drug File (MED-File) v2. Wolters Kluwer Clinical Drug Information. http://www.wolterskluwercdi.com/drug-data/medi-span-electronic-drug-file/. Accessed April 2016.

Cited by

Sorted by citations

Authors
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J.
Affiliations
  • Georgia Institute of Technology (US)
  • Sutter Health (US)
Corresponding author
Sun, Jimeng
Venue
Journal of the American Medical Informatics Association
Published
2016-08-13
Volume
24
Issue
2
Pages
361-370
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
OpenAlex
W2518582440
Cited by
928
References
52
Citations by year
2026: 52 · 2025: 83 · 2024: 99 · 2023: 103 · 2022: 107 · 2021: 131 · 2020: 125 · 2019: 130 · 2018: 70 · 2017: 27
FWCI
62.19
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
Funders
  • National Science Foundation (IIS-1418511 and CCF-1533768)
  • National Heart, Lung, and Blood Institute (1R01H116832-01)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
Keywords
heart failure, early detection, recurrent neural networks, gated recurrent unit, electronic health records, temporal relations, primary care patients, observation window
MeSH terms
Machine Learning, Heart Failure, Humans, Logistic Models, Neural Networks, Computer, Area Under Curve, Early Diagnosis, Electronic Health Records, Support Vector Machine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review