CoChat

Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series

Open access
Tipirneni, Sindhu, Reddy, Chandan K. · 2022 · ACM Transactions on Knowledge Discovery from Data

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

Multivariate time-series data are frequently observed in critical care settings and are typically characterized by sparsity (missing information) and irregular time intervals. Existing approaches for learning representations in this domain handle these challenges by either aggregation or imputation of values, which in-turn suppresses the fine-grained information and adds undesirable noise/overhead into the machine learning model. To tackle this problem, we propose a S elf-supervised Tra nsformer for T ime- S eries (STraTS) model, which overcomes these pitfalls by treating time-series as a set of observation triplets instead of using the standard dense matrix representation. It employs a novel Continuous Value Embedding technique to encode continuous time and variable values without the need for discretization. It is composed of a Transformer component with multi-head attention layers, which enable it to learn contextual triplet embeddings while avoiding the problems of recurrence and vanishing gradients that occur in recurrent architectures. In addition, to tackle the problem of limited availability of labeled data (which is typically observed in many healthcare applications), STraTS utilizes self-supervision by leveraging unlabeled data to learn better representations by using time-series forecasting as an auxiliary proxy task. Experiments on real-world multivariate clinical time-series benchmark datasets demonstrate that STraTS has better prediction performance than state-of-the-art methods for mortality prediction, especially when labeled data is limited. Finally, we also present an interpretable version of STraTS, which can identify important measurements in the time-series data. Our data preprocessing and model implementation codes are available at https://github.com/sindhura97/STraTS .

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3516367

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 An illustrative example of a multivariate clinical time-series with irregular time points and missing values.
Fig. 1 An illustrative example of a multivariate clinical time-series with irregular time points and missing values.CC-BY-NC · From the PDF
Fig. 2 An illustration of input and output construction for target and self-supervision (forecasting) tasks. The target task uses a fixed length observation window to predict in-hospital mortality. The forecasting task has an observation window that is followed by a fixed length prediction window in which only a subset of variables may be observed. Note that several observation windows are considered for each time-series for the forecasting task.
Fig. 2 An illustration of input and output construction for target and self-supervision (forecasting) tasks. The target task uses a fixed length observation window to predict in-hospital mortality. The forecasting task has an observation window that is followed by a fixed length prediction window in which only a subset of variables may be observed. Note that several observation windows are considered for each time-series for the forecasting task.CC-BY-NC · From the PDF
Fig. 3 The overall architecture of the proposed STraTS model. The Input Triplet Embedding module embeds each observation triplet, the Contextual Triplet Embedding module encodes contextual information for the triplets, the Fusion Self-Attention module computes time-series embedding, which is concatenated with demographics embedding and passed through a dense layer to generate predictions for target and self-supervision (forecasting) tasks.
Fig. 3 The overall architecture of the proposed STraTS model. The Input Triplet Embedding module embeds each observation triplet, the Contextual Triplet Embedding module encodes contextual information for the triplets, the Fusion Self-Attention module computes time-series embedding, which is concatenated with demographics embedding and passed through a dense layer to generate predictions for target and self-supervision (forecasting) tasks.CC-BY-NC · From the PDF
Fig. 4 Mortality prediction performance on MIMIC-III dataset for different percentages of labeled data averaged over 10 runs.
Fig. 4 Mortality prediction performance on MIMIC-III dataset for different percentages of labeled data averaged over 10 runs.CC-BY-NC · From the PDF
Fig. 5 Mortality prediction performance on PhysioNet-2012 dataset for different percentages of labeled data averaged over 10 runs.
Fig. 5 Mortality prediction performance on PhysioNet-2012 dataset for different percentages of labeled data averaged over 10 runs.CC-BY-NC · From the PDF
Fig. 6 Case study: An illustration of a few time-series with contribution scores for a patient from MIMIC-III dataset.
Fig. 6 Case study: An illustration of a few time-series with contribution scores for a patient from MIMIC-III dataset.CC-BY-NC · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 13

  1. Edwin V. Bonilla, Kian Ming Adam Chai, and Christopher K. I. Williams. 2007. Multi-task gaussian process prediction. In Proceedings of the 21st Annual Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 153-160.
  2. Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter F. Stewart. 2016. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems. 3504-3512.
  3. A. Farhana and S. L. Lappin. 2021. Biochemistry, Lactate Dehydrogenase.[Updated 2020 May 17]. StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing (2021).
  4. Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten M. Borgwardt. 2020. Set functions for time series. In Proceedings of the 37th International Conference on Machine Learning, (ICML'20), Vol. 119. PMLR, 4353-4363.
  5. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd Inter- national Conference on Learning Representations, (ICLR'15).
  6. Steven Cheng-Xian Li and Benjamin M. Marlin. 2015. Classification of sparse and irregularly sampled time series with mixtures of expected gaussian kernels and random features. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. AUAI Press, 484-493.
  7. Steven Cheng-Xian Li and Benjamin M. Marlin. 2016. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (2016). 1804-1812.
  8. Zachary C. Lipton, David C. Kale, and Randall C. Wetzel. 2016. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning in Health Care, MLHC 2016 (JMLR Workshop and Conference Proceedings, Vol. 56). JMLR.org, 253-270.
  9. Yulia Rubanova, Tian Qi Chen, and David Duvenaud. 2019. Latent ordinary differential equations for irregularly- sampled time series. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, (2019) NeurIPS'19. 5321-5331.
  10. Satya Narayan Shukla and Benjamin M. Marlin. 2019. Interpolation-prediction networks for irregularly sampled time series. In Proceedings of the 7th International Conference on Learning Representations, (ICLR'19). Retrieved from Open- Review.net. https://openreview.net/forum?id=r1efr3C9Ym.
  11. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceed- ings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (2014). 3104-3112.
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
  13. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Gen- eralized autoregressive pretraining for language understanding. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS'19). 5754-5764.

Cited by

See all 151

Metadata

Authors
Tipirneni, S., & Reddy, C. K.
Affiliations
Virginia Tech (US)
Venue
ACM Transactions on Knowledge Discovery from Data
Published
2022-07-30
Volume
16
Issue
6
Pages
1-17
ISSN
  • 1556-4681 (ISSN-L)
  • 1556-472X
OpenAlex
W4283370926
Cited by
130
References
18
Citations by year
2026: 34 · 2025: 40 · 2024: 33 · 2023: 20 · 2022: 3
FWCI
7.90
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Knowledge Discovery from Data (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • VTechWorks (Virginia Tech) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University) (cc-by)
Funders
US National Science Foundation (IIS-1838730)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
Keywords
self-supervised learning, transformer models, irregularly sampled time series, missing data, time series forecasting, mortality prediction, limited labeled data, interpretable models, intensive care, representation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.