CoChat
Sign Up
Cited by 406Machine Learning in HealthcareArtificial Intelligence in HealthcareSepsis Diagnosis and Treatment

Benchmarking deep learning models on large healthcare datasets

Open access
Purushotham, Sanjay, Meng, Chuizheng, Che, Zhengping, Liu, Yan·2018·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2018.04.007
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep learning models (aka Deep Neural Networks) have revolutionized many fields including computer vision, natural language processing, speech recognition, and is being increasingly used in clinical healthcare applications. However, few works exist which have benchmarked the performance of the deep learning models with respect to the state-of-the-art machine learning models and prognostic scoring systems on publicly available healthcare datasets. In this paper, we present the benchmarking results for several clinical prediction tasks such as mortality prediction, length of stay prediction, and ICD-9 code group prediction using Deep Learning models, ensemble of machine learning models (Super Learner algorithm), SAPS II and SOFA scores. We used the Medical Information Mart for Intensive Care III (MIMIC-III) (v1.4) publicly available dataset, which includes all patients admitted to an ICU at the Beth Israel Deaconess Medical Center from 2001 to 2012, for the benchmarking tasks. Our results show that deep learning models consistently outperform all the other approaches especially when the 'raw' clinical time series data is used as input features to the models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2018.04.007

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Late Fusion with simple concatenation was the best-performing model in evaluations conducted by the previous paper that used the exact same partition as ours.

    “Late Fusion (LF) with simple concatenation was the best-performing model in the evaluations conducted by the previous paper that used the exact same partition as ours [129].”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [129]

    Section 4.4.5 identifies MMDL, not Late Fusion with simple concatenation, as the model that mostly outperformed the comparators across the evaluated tasks. The MMDL model section describes a shared representation layer, and §4.3 reports five-fold stratified cross-validation; our copy does not establish that the citing paper used the exact same partition. (LLM, conf=0.85)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

9 of the 57 references couldn’t be matched to a known paper.

Research works printed but not linked: 24

  1. Z. Che, S. Purushotham, K. Cho, D. Sontag, Y. Liu, Recurrent Neural Networks for Multivariate Time Series with Missing Values, 2016, arXiv preprint arXiv:1606. 01865.
  2. H. Harutyunyan, H. Khachatrian, D.C. Kale, A. Galstyan, Multitask Learning and Benchmarking with Clinical Time Series Data, 2017, arXiv preprint arXiv:1703. 07771.
  3. A.E. Johnson, T.J. Pollard, R.G. Mark, Reproducibility in critical care: a mortality prediction case study, in: Machine Learning for Healthcare Conference, 2017, 361-376.
  4. Z. Che, S. Purushotham, R. Khemani, Y. Liu, Interpretable deep models for icu outcome prediction, in: AMIA Annual Symposium Proceedings, vol. 2016, American Medical Informatics Association, 2016, p. 371.
  5. R. Joshi, P. Szolovits, Prognostic physiology: modeling patient severity in intensive care units using radial domain folding, in: AMIA Annual Symposium Proceedings, vol. 2012, American Medical Informatics Association, 2012, p. 1276.
  6. L.-W. Lehman, M. Saeed, W. Long, J. Lee, R. Mark, Risk stratification of icu patients using topic models inferred from unstructured progress notes, in: AMIA Annual Symposium Proceedings, vol. 2012, American Medical Informatics Association, 2012, p. 505.
  7. S. Joshi, S. Gunasekar, D. Sontag, G. Joydeep, Identifiable phenotyping using constrained non-negative matrix factorization, in: Machine Learning for Healthcare Conference, 2016, pp. 17-41.
  8. Y.-F. Luo, A. Rumshisky, Interpretable topic features for post-ICU mortality pre- diction, in: AMIA Annual Symposium Proceedings, vol. 2016, American Medical Informatics Association, 2016, p. 827.
  9. G. Doig, K. Inman, W. Sibbald, C. Martin, J. Robertson, Modeling mortality in the intensive care unit: comparing the performance of a back-propagation, associative- learning neural network with multivariate logistic regression, in: Proceedings of the Annual Symposium on Computer Application in Medical Care, American Medical Informatics Association, 1993, p. 361.
  10. R. Caruana, S. Baluja, T. Mitchell, Using the future to sort out the present: Rankprop and multitask learning for medical risk evaluation, in: Advances in Neural Information Processing Systems, 1996, pp. 959-965.
  11. Z. Che, S. Purushotham, R. Khemani, Y. Liu, Distilling Knowledge From Deep Networks with Applications to Healthcare Domain, 2015 arXiv preprint arXiv:1512. 03542.
  12. Z.C. Lipton, D.C. Kale, C. Elkan, R. Wetzell, Learning to Diagnose with LSTM Recurrent Neural Networks, 2015 arXiv preprint arXiv:1511.03677. · arXiv
  13. S. Purushotham, W. Carvalho, T. Nilanon, Y. Liu, Variational recurrent adversarial deep domain adaptation, in: International Conference on Learning Representations (ICLR), 2017.
  14. I. Silva, G. Moody, D.J. Scott, L.A. Celi, R.G. Mark, Predicting in-hospital mortality of ICU patients: the physionet/computing in cardiology challenge 2012, in: Computing in Cardiology (CinC), 2012, IEEE, 2012, pp. 245-248.
  15. G. Dahl, A.-r. Mohamed, G.E. Hinton, et al, Phone recognition with the mean- covariance restricted boltzmann machine, in: Advances in Neural Information Processing Systems, 2010, pp. 469-477.
  16. A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep con- volutional neural networks, in: Advances in Neural Information Processing Systems, 2012, pp. 1097-1105.
  17. A. Bordes, X. Glorot, J. Weston, Y. Bengio, Joint learning of words and meaning representations for open-text semantic parsing, in: Artificial Intelligence and Statistics, 2012, pp. 127-135.
  18. T. Mikolov, W.-T. Yih, G. Zweig, Linguistic regularities in continuous space word representations, in: hlt-Naacl, vol. 13, 2013, pp. 746-751.
  19. T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems, 2013, pp. 3111-3119.
  20. V. Nair, G.E. Hinton, Rectified linear units improve restricted boltzmann machines, in: Proceedings of the 27th International Conference on Machine Learning (ICML), 2010, pp. 807-814.
  21. J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, 2014 arXiv preprint arXiv:1412.3555. · arXiv
  22. N. Srivastava, R.R. Salakhutdinov, Multimodal learning with deep boltzmann ma- chines, in: Advances in Neural Information Processing Systems, 2012, pp. 2222-2230.
  23. F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, Y. Bengio, Theano: New Features and Speed Improvements, 2012, arXiv preprint arXiv:1211.5590. · arXiv
  24. B. Xu, N. Wang, T. Chen, M. Li, Empirical Evaluation of Rectified Activations in Convolutional Network, 2015 arXiv preprint arXiv:1505.00853. · arXiv

Cited by

Sorted by citations

Related

Authors
Purushotham, S., Meng, C., Che, Z., & Liu, Y.
Affiliations
  • University of Southern California (US)
  • Tsinghua University (CN)
Corresponding author
Liu, Yan
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2018-07
Volume
83
Pages
112-134
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2018.04.007
OpenAlex
W2806031239
Cited by
406
References
57
Citations by year
2026: 37 · 2025: 57 · 2024: 39 · 2023: 51 · 2022: 66 · 2021: 66 · 2020: 50 · 2019: 36 · 2018: 4
FWCI
21.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion, other-oa)
  • PubMed (publishedVersion)
Funders
  • NSF (IIS-1134990, IIS-1254206)
  • Samsung GRO Grant
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
Keywords
deep learning benchmarking, clinical prediction, mortality prediction, length of stay prediction, MIMIC-III, intensive care unit patients, super learner, SAPS II, SOFA score, clinical time series, prognostic score
MeSH terms
Deep Learning, Aged, Aged, 80 and over, Algorithms, Female, Forecasting, Humans, Intensive Care Units, Length of Stay, Male, Middle Aged, Databases, Factual, Neural Networks, Computer, Hospital Mortality, Benchmarking, International Classification of Diseases

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review