CoChat
Sign Up
PreprintCited by 879Machine Learning in HealthcareArtificial Intelligence in HealthcareTime Series Analysis and Forecasting

Learning to Diagnose with LSTM Recurrent Neural Networks

Open access
Lipton, Zachary C., Kale, David C., Elkan, Charles, Wetzel, Randall·2015·arXiv
DOI 10.48550/arxiv.1511.03677arXiv 1511.03677
Open this paper in CoChat — free accountOpen PDF

Abstract

Clinical medical data, especially in the intensive care unit (ICU), consist of multivariate time series of observations. For each patient visit (or episode), sensor data and lab test results are recorded in the patient's Electronic Health Record (EHR). While potentially containing a wealth of insights, the data is difficult to mine effectively, owing to varying length, irregular sampling and missing data. Recurrent Neural Networks (RNNs), particularly those using Long Short-Term Memory (LSTM) hidden units, are powerful and increasingly popular models for learning from sequence data. They effectively model varying length sequences and capture long range dependencies. We present the first study to empirically evaluate the ability of LSTMs to recognize patterns in multivariate time series of clinical measurements. Specifically, we consider multilabel classification of diagnoses, training a model to classify 128 diagnoses given 13 frequently but irregularly sampled clinical measurements. First, we establish the effectiveness of a simple LSTM network for modeling clinical data. Then we demonstrate a straightforward and effective training strategy in which we replicate targets at each sequence step. Trained only on raw time series, our models outperform several strong baselines, including a multilayer perceptron trained on hand-engineered features.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1511.03677

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multiple-instance configurations incur high computational cost from running multiple instances.

    “These include the high computational cost incurred by running multiple instances [18,19]”

    From TVGeAN: Tensor Visibility Graph-Enhanced Attention Network for Versatile Multivariant Time Series Learning Tasks (2024) · cites this paper as [18]

    The classification-performance discussion (p. 9) says auxiliary-output models trained more slowly and required more epochs. It does not describe running multiple instances or attribute computational cost to doing so. Searches for multiple-instance configurations and training multiple models found no passage addressing that claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 44

  1. Aleks, Norm, Russell, Stuart J, Madden, Michael G, Morabito, Diane, Staudenmayer, Kristan, Cohen, Mitchell, and Manley, Geoffrey T. Probabilistic detection of short events, with application to critical care monitoring. In Advances in Neural Information Processing Systems (NIPS) 21 , pp. 49-56, 2009.
  2. Amari, Shun-ichi and Cichocki, Andrzej. Adaptive blind signal processing-neural network approaches. Proceedings of the IEEE , 86(10):2026-2048, 1998.
  3. Baxt, W.G. Application of artificial neural networks to clinical medicine. The Lancet , 346(8983): 1135-1138, 1995.
  4. Caruana, Rich, Baluja, Shumeet, Mitchell, Tom, et al. Using the future to 'sort out' the present: Rankprop and multitask learning for medical risk evaluation. In Advances in Neural Information Processing Systems (NIPS) 8 , pp. 959-965, 1996.
  5. Che, Zhengping, Kale, David C., Li, Wenzhe, Bahadori, Mohammad Taha, and Liu, Yan. Deep computational phenotyping. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , pp. 507-516. ACM, 2015.
  6. Dabek, Filip and Caban, Jesus J. A neural network based model for predicting psychological conditions. In Brain Informatics and Health , pp. 252-261. Springer, 2015.
  7. Dai, Andrew M and Le, Quoc V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) 28 , pp. 3061-3069, 2015.
  8. De Mulder, Wim, Bethard, Steven, and Moens, Marie-Francine. A survey on the application of recurrent neural networks to statistical language modeling. Computer Speech & Language , 30(1): 61-98, 2015.
  9. Elman, Jeffrey L. Finding structure in time. Cognitive Science , 14(2):179-211, 1990.
  10. Fleming, Susannah, Thompson, Matthew, Stevens, Richard, Heneghan, Carl, Plddemann, Annette, Maconochie, Ian, Tarassenko, Lionel, and Mant, David. Normal ranges of heart rate and respiratory rate in children from birth to 18 years: A systematic review of observational studies. The Lancet , pp. 1011-1018, 2011.
  11. Gers, Felix and Schmidhuber, J¨ urgen. Recurrent nets that time and count. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks (IJCNN) , volume 3, pp. 189-194. IEEE, 2000.
  12. Gers, Felix A., Schmidhuber, J¨ urgen, and Cummins, Fred. Learning to forget: Continual prediction with LSTM. Neural Computation , 12(10):2451-2471, 2000.
  13. Gers, Felix A., Schraudolph, Nicol N., and Schmidhuber, J¨ urgen. Learning precise timing with LSTM recurrent networks. The Journal of Machine Learning Research , 3:115-143, 2003.
  14. Ghassemi, Marzyeh, Pimentel, Marco AF, Naumann, Tristan, Brennan, Thomas, Clifton, David A, Szolovits, Peter, and Feng, Mengling. A multivariate timeseries modeling approach to severity of illness assessment and forecasting in ICU with sparse, heterogeneous clinical data. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.
  15. Graves, Alex. Supervised Sequence Labelling with Recurrent Neural Networks , volume 385 of Studies in Computational Intelligence . Springer-Verlag Berlin Heidelberg, 2012.
  16. Graves, Alex. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 , 2013. · arXiv
  17. Graves, Alex, Liwicki, Marcus, Fern´ andez, Santiago, Bertolami, Roman, Bunke, Horst, and Schmidhuber, J¨ urgen. A novel connectionist system for unconstrained handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 31(5):855-868, 2009.
  18. Lasko, Thomas A., Denny, Joshua C., and Levy, Mia A. Computational phenotype discovery using unsupervised feature learning over noisy, sparse, and irregular clinical data. PLoS ONE , 8(6): e66341, 06 2013.
  19. Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeplysupervised nets. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics (AISTATS) , 2015.
  20. Lipton, Zachary C, Elkan, Charles, and Naryanaswamy, Balakrishnan. Optimal thresholding of classifiers to maximize F1 measure. In Machine Learning and Knowledge Discovery in Databases , pp. 225-239. Springer, 2014.
  21. Lipton, Zachary C., Berkowitz, John, and Elkan, Charles. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv:1506.00019 , 2015. · arXiv
  22. Liu, I, Ramakrishnan, Bhiksha, et al. Bach in 2014: Music composition with recurrent neural network. arXiv preprint arXiv:1412.3191 , 2014. · arXiv
  23. Liwicki, Marcus, Graves, Alex, Bunke, Horst, and Schmidhuber, J¨ urgen. A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks. In Proceedings of the Ninth International Conference on Document Analysis and Recognition , volume 1, pp. 367-371, 2007.
  24. Marlin, Ben M., Kale, David C., Khemani, Robinder G., and Wetzel, Randall C. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium (IHI) , 2012.
  25. Mason, Robert J., Broaddus, V. Courtney, Martin, Thomas, King Jr., Talmadge E., Schraufnagel, Dean, Murray, John F., and Nadel, Jay A. Murray and Nadel's textbook of respiratory medicine: 2-volume set . Elsevier Health Sciences, 2010.
  26. Ng, Joe Yue-Hei, Hausknecht, Matthew, Vijayanarasimhan, Sudheendra, Vinyals, Oriol, Monga, Rajat, and Toderici, George. Beyond short snippets: Deep networks for video classification. arXiv preprint arXiv:1503.08909 , 2015. · arXiv
  27. Pascanu, Razvan, Gulcehre, Caglar, Cho, Kyunghyun, and Bengio, Yoshua. How to construct deep recurrent neural networks. In International Conference on Learning Representations (ICLR) , 2014.
  28. Pham, Vu, Bluche, Th´ eodore, Kermorvant, Christopher, and Louradour, J´ erˆ ome. Dropout improves recurrent neural networks for handwriting recognition. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on , pp. 285-290. IEEE, 2014.
  29. Pollack, M. M., Patel, K. M., and Ruttimann, U. E. PRISM III: an updated Pediatric Risk of Mortality score. Critical Care Medicine , 24(5):743-752, 1996.
  30. Pollastri, Gianluca, Przybylski, Darisz, Rost, Burkhard, and Baldi, Pierre. Improving the prediction of protein secondary structure in three and eight classes using recurrent neural networks and profiles. Proteins: Structure, Function, and Bioinformatics , 47(2):228-235, 2002.
  31. Quinn, John, Williams, Christopher KI, McIntosh, Neil, et al. Factorial switching linear dynamical systems applied to physiological condition monitoring. IEEE Transactions on Pattern Analysis and Machine Intelligence , 31(9):1537-1551, 2009.
  32. Rughani, Anand I., Dumont, Travis M., Lu, Zhenyu, Bongard, Josh, Horgan, Michael A., Penar, Paul L., and Tranmer, Bruce I. Use of an artificial neural network to predict head injury outcome: clinical article. Journal of Neurosurgery , 113(3):585-590, 2010.
  33. Saria, Suchi, Koller, Daphne, and Penn, Anna. Learning individual and population level traits from clinical temporal data. In Proc. Neural Information Processing Systems (NIPS), Predictive Models in Personalized Medicine Workshop . Citeseer, 2010.
  34. Schulam, Peter, Wigley, Fredrick, and Saria, Suchi. Clustering longitudinal clinical marker trajectories from electronic health data: Applications to phenotyping and endotype discovery. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.
  35. Silipo, Rosaria and Marchesi, Carlo. Artificial neural networks for automatic ecg analysis. IEEE Transactions on Signal Processing , 46(5):1417-1425, 1998.
  36. Stanculescu, Ioan, Williams, Christopher K, Freer, Yvonne, et al. Autoregressive hidden markov models for the early detection of neonatal sepsis. Biomedical and Health Informatics, IEEE Journal of , 18(5):1560-1570, 2014a.
  37. Stanculescu, Ioan, Williams, Christopher KI, and Freer, Yvonne. A hierarchical switching linear dynamical system applied to the detection of sepsis in neonatal condition monitoring. In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence (UAI) , 2014b.
  38. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS) 27 , pp. 3104-3112, 2014.
  39. Tresp, Volker and Briegel, Thomas. A solution for missing data in recurrent neural networks with an application to blood glucose prediction. In Advances in Neural Information Processing Systems (NIPS) 10 , pp. 971-977. 1998.
  40. ¨ Ubeyli, Elif Derya. Combining recurrent neural networks with eigenvector methods for classification of ecg beats. Digital Signal Processing , 19(2):320-329, 2009.
  41. Vohradsk´ y, Jiˇ r´ ı. Neural network model of gene expression. The FASEB Journal , 15(3):846-854, 2001.
  42. Xu, Rui, Wunsch II, Donald, and Frank, Ronald. Inference of genetic regulatory networks with recurrent neural network models using particle swarm optimization. IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB) , 4(4):681-692, 2007.
  43. Yeung, Serena, Russakovsky, Olga, Jin, Ning, Andriluka, Mykhaylo, Mori, Greg, and Fei-Fei, Li. Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint arXiv:1507.05738 , 2015. · arXiv
  44. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014. · arXiv

Cited by

Sorted by citations

Authors
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1511.03677
arXiv
1511.03677
OpenAlex
W2141173017
Cited by
879
References
4
Citations by year
2026: 3 · 2025: 47 · 2024: 59 · 2023: 74 · 2022: 51 · 2021: 115 · 2020: 138 · 2019: 161 · 2018: 134 · 2017: 76 · 2016: 19 · 2015: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
Keywords
diagnostic prediction, long short-term memory, recurrent neural networks, multivariate time series, electronic health records, intensive care unit, multi-label classification, irregularly sampled data, missing data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review