CoChat
Sign Up
Cited by 24Machine Learning in HealthcareDomain Adaptation and Few-Shot LearningTime Series Analysis and Forecasting

A comprehensive EHR timeseries pre-training benchmark

Open access
McDermott, Matthew, Nestor, Bret, Kim, Evan, Zhang, Wancong, Goldenberg, Anna, Szolovits, Peter, Ghassemi, Marzyeh·2021·Proceedings of the Conference on Health, Inference, and Learning
DOI 10.1145/3450439.3451877
Open this paper in CoChat — free accountOpen PDF

Abstract

Pre-training (PT) has been used successfully in many areas of machine learning. One area where PT would be extremely impactful is over electronic health record (EHR) data. Successful PT strategies on this modality could improve model performance in data-scarce contexts such as modeling for rare diseases or allowing smaller hospitals to benefit from data from larger health systems. While many PT strategies have been explored in other domains, much less exploration has occurred for EHR data. One reason for this may be the lack of standardized benchmarks suitable for developing and testing PT algorithms. In this work, we establish a PT benchmark dataset for EHR timeseries data, establishing cohorts, a diverse set of fine-tuning tasks, and PT-focused evaluation regimes across two public EHR datasets: MIMIC-III and eICU. This benchmark fills an essential hole in the field by enabling a robust manner of iterating on PT strategies for this modality. To show the value of this benchmark and provide baselines for further research, we also profile two simple PT algorithms: a self-supervised, masked imputation system and a weakly-supervised, multi-task system. We find that PT strategies (in particular weakly-supervised PT methods) can offer significant gains over traditional learning in few-shot settings, especially on tasks with strong class imbalance. Our full benchmark and code are publicly available at https://github.com/mmcdermott/comprehensive_MTL_EHR

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3450439.3451877

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The most popular prediction tasks for FEMRs are binary classification tasks such as mortality, heart failure, and long length-of-stay, but even the same nominal task can have widely divergent definitions across papers.

    “The most popular prediction tasks are binary classification tasks such as mortality, heart failure, and long length-of-stay, but even the same nominal task can have widely divergent definitions across papers78.”

    From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [78]

    Section 5.2 describes varying definitions of the same task, including mortality predictions at 24 and 48 hours; Section 5.2.6 and its Prior Art passage describe binary long-length-of-stay predictions using different thresholds. The paper also treats mortality as a well-studied binary task, but the passages do not establish that these are the most popular prediction tasks or identify heart failure as one of them. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The majority of evaluation tasks span one data modality.

    “Currently, the majority of evaluation tasks span one data modality78”

    From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [78]

    The High-Level Overview (Section 4; page unavailable) describes this paper’s ten downstream tasks using EHR time-series data. Neither it nor the task descriptions in Section 5.2 establish that the majority of evaluation tasks generally span one data modality; searches for single-modality and multimodal evaluation tasks found no such statement. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 38

  1. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly Available Clinical BERT Embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Work- shop. Association for Computational Linguistics, Minneapolis, Minnesota, USA, 72-78. https://doi.org/10.18653/v1/W19-1909 · DOI · Paper page
  2. Aparna Balagopalan, Jekaterina Novikova, Matthew B A Mcdermott, Bret Nestor, Tristan Naumann, and Marzyeh Ghassemi. 2020. Cross-Language Aphasia Detec- tion using Optimal Transport Domain Adaptation. In Proceedings of the Machine Learning for Health NeurIPS Workshop (Proceedings of Machine Learning Research),
  3. Adrian V. Dalca, Matthew B.A. McDermott, Emily Alsentzer, Samuel G. Finlayson, Michael Oberst, Fabian Falck, and Brett Beaulieu-Jones (Eds.), Vol. 116. PMLR, 202-219. http://proceedings.mlr.press/v116/balagopalan20a.html
  4. James Bergstra, Daniel Yamins, and David Cox. 2013. Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. In Proceedings of the 30th International Conference on Ma- chine Learning (Proceedings of Machine Learning Research), Sanjoy Dasgupta and David McAllester (Eds.), Vol. 28. PMLR, Atlanta, Georgia, USA, 115-123. http://proceedings.mlr.press/v28/bergstra13.html
  5. Dimitris Bertsimas, Jean Pauphilet, Jennifer Stevens, and Manu Tandon. 2020. Pre- dicting inpatient flow at a major hospital using interpretable analytics. medRxiv (2020).
  6. Chun-Hao Chang, Mingjie Mai, and Anna Goldenberg. 2019. Dynamic Mea- surement Scheduling for Event Forecasting using Deep RL. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, Long Beach, California, USA, 951-960. http://proceedings.mlr.press/v97/ chang19a.html
  7. Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports 8, 1 (17 Apr 2018), 6085. https://doi.org/10.1038/s41598- 018-24271-9 · DOI · Paper page
  8. Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent Neural Networks for Multivariate Time Series with Missing Values. Scientific Reports 8, 1 (April 2018), 6085. https://doi.org/10.1038/s41598- 018-24271-9 · DOI · Paper page
  9. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1724-1734. https://doi.org/10.3115/v1/D14-1179 · DOI · Paper page
  10. Edward Choi, Cao Xiao, Walter Stewart, and Jimeng Sun. 2018. MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/ 934b535800b1cba8f96a5d72f72f1611-Paper.pdf
  11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248-255. https://doi.org/10.1109/CVPR.2009. 5206848 · DOI · Paper page
  12. Daisy Yi Ding, Chloé Simpson, Stephen Pfohl, Dave C Kale, Kenneth Jung, and Nigam H Shah. 2019. The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data.. In Biocomputing 2019. World Scientific, 18-29. https://doi.org/10.1142/9789813279827_0003 · DOI
  13. Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. 2018. SOM-VAE: Interpretable Discrete Representation Learning on Time Series. arXiv:1806.02199 [cs, stat] (June 2018). http://arxiv.org/abs/1806. 02199 arXiv: 1806.02199. · arXiv
  14. Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L Beam, Irene Y Chen, and Rajesh Ranganath. 2019. Practical guidance on artificial intelligence for health-care data. The Lancet Digital Health 1, 4 (2019), e157-e159.
  15. Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. 2019. Multitask learning and benchmarking with clinical time series data. Scientific data 6, 1 (2019), 1-18.
  16. George Hripcsak, David J Albers, and Adler Perotte. 2015. Parameter- izing time in electronic health record studies. Journal of the Amer- ican Medical Informatics Association 22, 4 (02 2015), 794-804. https: //doi.org/10.1093/jamia/ocu051 arXiv:https://academic.oup.com/jamia/article- pdf/22/4/794/25894265/ocu051.pdf · DOI
  17. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data 3, 1 (May 2016), 1-9. https://doi.org/10.1038/sdata.2016.35 · DOI · Paper page
  18. Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, Vol. 29. 4601-4609. https://proceedings.neurips.cc/paper/2016/file/ 16026d60ff9b54410b3435b403afd226-Paper.pdf
  19. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A New Benchmark Datasetfor Cross-lingual Pre- training, Understanding and Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 6008-6018. https://doi.org/10.18653/v1/2020.emnlp- main.484 · DOI
  20. Sharon L Lojun, Christina J Sauper, Mitchell Medow, William J Long, Roger G Mark, and Regina Barzilay. 2010. Investigating resuscitation code assignment in the intensive care unit using structured and unstructured data. In AMIA Annual Symposium Proceedings, Vol. 2010. American Medical Informatics Association, 467.
  21. Yuan Luo, Peter Szolovits, Anand S Dighe, and Jason M Baron. 2016. Using machine learning to predict laboratory test results. American journal of clinical pathology 145, 6 (2016), 778-788.
  22. Aya A Mitani and Sebastien Haneuse. 2020. Small Data Challenges of Studying Rare Diseases. JAMA Network Open 3, 3 (2020), e201965-e201965.
  23. Bret Nestor, Matthew B. A. McDermott, Willie Boag, Gabriela Berner, Tristan Naumann, Michael C. Hughes, Anna Goldenberg, and Marzyeh Ghassemi. 2019. Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks. In Proceedings of the 4th Machine Learning for Healthcare Conference (Proceedings of Machine Learning Research), Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (Eds.), Vol. 106. PMLR, Ann Arbor, Michigan, 381-405. http://proceedings.mlr.press/v106/nestor19a.html
  24. Xuefeng Peng, Yi Ding, David Wihl, Omer Gottesman, Matthieu Komorowski, Li- wei H Lehman, Andrew Ross, Aldo Faisal, and Finale Doshi-Velez. 2018. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. In AMIA Annual Symposium Proceedings, Vol. 2018. American Medical Informatics Association, 887.
  25. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2227-2237.
  26. Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. 2018. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific data 5 (2018), 180178.
  27. Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. 2019. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems. 9689-9701.
  28. Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. 2019. Pre-training of Graph Augmented Transformers for Medication Recommendation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI- 19. International Joint Conferences on Artificial Intelligence Organization, 5953- 5959. https://doi.org/10.24963/ijcai.2019/825 · DOI · Paper page
  29. Yuqi Si and Kirk Roberts. 2019. Deep Patient Representation of Clinical Notes via Multi-Task Learning for Mortality Prediction. AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Sci- ence 2019 (06 May 2019), 779-788. https://pubmed.ncbi.nlm.nih.gov/31259035 31259035[pmid].
  30. Yuqi Si, Jingqi Wang, Hua Xu, and Kirk Roberts. 2019. Enhancing clinical con- cept extraction with contextual embeddings. Journal of the American Medical Informatics Association 26, 11 (2019), 1297-1304.
  31. Vergil N Slee. 1978. The International classification of diseases: ninth revision (ICD-9). Annals of internal medicine 88, 3 (1978), 424-426.
  32. Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. 2017. Clinical Intervention Prediction and Understanding using Deep Networks. In Proceedings of the 2nd Machine Learning for Healthcare Conference (Proceedings of Machine Learning Research), Finale Doshi-Velez, Jim Fackler, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (Eds.), Vol. 68. PMLR, Boston, Massachusetts, 322-337. http://proceedings.mlr.press/ v68/suresh17a.html
  33. Shirly Wang, Matthew B. A. McDermott, Geeticka Chauhan, Marzyeh Ghas- semi, Michael C. Hughes, and Tristan Naumann. 2020. MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III. In Proceedings of the ACM Conference on Health, Inference, and Learning (CHIL 2020). Association for Computing Machinery, New York, NY, USA, 222-235. https://doi.org/10.1145/3368555.3384469 · DOI
  34. Xiang Wang, Fei Wang, and Jianying Hu. 2014. A multi-task learning frame- work for joint disease risk prediction and comorbidity discovery. In 2014 22nd International Conference on Pattern Recognition. IEEE, 220-225.
  35. Mike Wu, Marzyeh Ghassemi, Mengling Feng, Leo A Celi, Peter Szolovits, and Finale Doshi-Velez. 2016. Understanding vasopressor intervention and weaning: risk prediction in a public heterogeneous clinical time series database. Journal of the American Medical Informatics Association 24, 3 (10 2016), 488-495. https: //doi.org/10.1093/jamia/ocw138 arXiv:https://academic.oup.com/jamia/article- pdf/24/3/488/25421894/ocw138.pdf · DOI
  36. Yuan Xue, Nan Du, Anne Mottram, Martin Seneviratne, and Andrew M Dai. 2020. Learning to Select Best Forecast Tasks for Clinical Outcome Prediction. Advances in Neural Information Processing Systems 33 (2020).
  37. Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. 2020. VIME: Extending the Success of Self-and Semi-supervised Learning to Tabular Domain. Advances in Neural Information Processing Systems 33 (2020).
  38. Henghui Zhu, Ioannis Ch Paschalidis, and Amir Tahmasebi. 2018. Clinical concept extraction with contextual word embedding. arXiv preprint arXiv:1810.10566 (2018). · arXiv

Cited by

Sorted by citations

Authors
McDermott, M., Nestor, B., Kim, E., Zhang, W., Goldenberg, A., Szolovits, P., & Ghassemi, M.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • University of Toronto (CA)
  • New York University (US)
  • CSAIL, MIT
ORCID
Venue
Proceedings of the Conference on Health, Inference, and Learning
Publisher
ACM
Published
2021-04-08
Pages
257-278
DOI
10.1145/3450439.3451877
OpenAlex
W3136047647
Cited by
24
References
4
Citations by year
2026: 1 · 2025: 7 · 2024: 4 · 2023: 6 · 2022: 4 · 2021: 2
FWCI
1.85
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by)
Funders
  • National Institutes of Health (LM013337-01)
  • Natural Sciences and Engineering Research Council of Canada
  • IBM
  • New Frontiers in Research Fund (2019-00844)
  • Canadian Institutes of Health Research
  • Canadian Institute for Advanced Research
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
Keywords
electronic health records, pre-training, benchmark dataset, MIMIC-III, eICU, self-supervised learning, weakly supervised learning, multi-task learning, few-shot learning, class imbalance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review