Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Multivariate time-series data are frequently observed in critical care settings and are typically characterized by sparsity (missing information) and irregular time intervals. Existing approaches for learning representations in this domain handle these challenges by either aggregation or imputation of values, which in-turn suppresses the fine-grained information and adds undesirable noise/overhead into the machine learning model. To tackle this problem, we propose a S elf-supervised Tra nsformer for T ime- S eries (STraTS) model, which overcomes these pitfalls by treating time-series as a set of observation triplets instead of using the standard dense matrix representation. It employs a novel Continuous Value Embedding technique to encode continuous time and variable values without the need for discretization. It is composed of a Transformer component with multi-head attention layers, which enable it to learn contextual triplet embeddings while avoiding the problems of recurrence and vanishing gradients that occur in recurrent architectures. In addition, to tackle the problem of limited availability of labeled data (which is typically observed in many healthcare applications), STraTS utilizes self-supervision by leveraging unlabeled data to learn better representations by using time-series forecasting as an auxiliary proxy task. Experiments on real-world multivariate clinical time-series benchmark datasets demonstrate that STraTS has better prediction performance than state-of-the-art methods for mortality prediction, especially when labeled data is limited. Finally, we also present an interpretable version of STraTS, which can identify important measurements in the time-series data. Our data preprocessing and model implementation codes are available at https://github.com/sindhura97/STraTS .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, interpolation-based models add undesirable noise and extra overhead to the model which usually worsens as the time series become increasingly sparse [54].”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Though this technique has shown great performance boosts with image [15] and text [9, 31] data, its application to time-series data has been limited.”
BackgroundGoldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 2×
“The success of deep learning in image and text domains realized by convolutional and recurrent networks [7, 28], and Transformer models [30] have inspired the application of these architectures to develop better prediction models for time-series data as well.”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 10×
“For MIMIC-III dataset, we set the maximum number of time-steps for GRU-D and InterpNet, and the maximum no. of observations for STraTS using the 99th percentile for the same.”
MethodInfluentialProceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Gaussian Processes in Machine Learning
Rasmussen, C. E. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-28650-9_4 · Cited by 5,397
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1803.01271 · Cited by 4,655 · cited 2×
“We expected the interpolation-based models GRU-D and InterpNet to outperform the simpler models GRU, TCN, and SaND.”
BackgroundRecurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · cited 10×
“Existing sequence models for clinical time-series [4] and other interpolationbased models [26] address this issue by including a learnable imputation or interpolation component.”
BackgroundMethodInfluentialSelf-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036 · cited 1×
“Though this technique has shown great performance boosts with image [15] and text [9, 31] data, its application to time-series data has been limited.”
BackgroundA Transformer-based Framework for Multivariate Time Series Representation Learning
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., & Eickhoff, C. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467401 · Cited by 1,278 · cited 1×
“[33] pretrained a Transformer model using a denoising objective and showed improved performance on regression and classification tasks with dense multivariate time-series.”
Method
Research works printed but not linked: 13
- Edwin V. Bonilla, Kian Ming Adam Chai, and Christopher K. I. Williams. 2007. Multi-task gaussian process prediction. In Proceedings of the 21st Annual Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 153-160.
- Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter F. Stewart. 2016. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems. 3504-3512.
- A. Farhana and S. L. Lappin. 2021. Biochemistry, Lactate Dehydrogenase.[Updated 2020 May 17]. StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing (2021).
- Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten M. Borgwardt. 2020. Set functions for time series. In Proceedings of the 37th International Conference on Machine Learning, (ICML'20), Vol. 119. PMLR, 4353-4363.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd Inter- national Conference on Learning Representations, (ICLR'15).
- Steven Cheng-Xian Li and Benjamin M. Marlin. 2015. Classification of sparse and irregularly sampled time series with mixtures of expected gaussian kernels and random features. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. AUAI Press, 484-493.
- Steven Cheng-Xian Li and Benjamin M. Marlin. 2016. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (2016). 1804-1812.
- Zachary C. Lipton, David C. Kale, and Randall C. Wetzel. 2016. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning in Health Care, MLHC 2016 (JMLR Workshop and Conference Proceedings, Vol. 56). JMLR.org, 253-270.
- Yulia Rubanova, Tian Qi Chen, and David Duvenaud. 2019. Latent ordinary differential equations for irregularly- sampled time series. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, (2019) NeurIPS'19. 5321-5331.
- Satya Narayan Shukla and Benjamin M. Marlin. 2019. Interpolation-prediction networks for irregularly sampled time series. In Proceedings of the 7th International Conference on Learning Representations, (ICLR'19). Retrieved from Open- Review.net. https://openreview.net/forum?id=r1efr3C9Ym.
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceed- ings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (2014). 3104-3112.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Gen- eralized autoregressive pretraining for language understanding. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS'19). 5754-5764.
Cited by
Transformers in Time-Series Analysis: A Tutorial
Ahmed, S., Nielsen, I. E., Tripathi, A., Siddiqui, S., Ramachandran, R. P., & Rasool, G. (2023) · Circuits, Systems, and Signal Processing · doi:10.1007/s00034-023-02454-8 · Cited by 305
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“Moreover, STraTS [54] first encodes the time series data into triple representations to avoid the limitations of using basic RNN and CNN in modeling irregular and sparse time series data and then builds the transformer-based forecasting model for modeling multivariate medical clinical time series.”
BackgroundMethodA systematic review for transformer-based long-term series forecasting
Su, L., Zuo, X., Li, R., Wang, X., Zhao, H., & Huang, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11044-2 · Cited by 192
Patharkar, A., Cai, F., Al-Hindawi, F., & Wu, T. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1386760 · Cited by 108
“For instance, Tipirneni and Reddy (2022) introduced the Self-supervised Transformer for Time-Series (STraTS) model, which treats each time-series as observation triplets (time, variable, value) instead of matrices as done by conventional methods.”
BackgroundA Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?
Gonçalves, T., Rio-Torto, I., Teixeira, L. F., & Cardoso, J. S. (2022) · IEEE Access · doi:10.1109/access.2022.3206449 · Cited by 100
“[181] pro1010 posed a Transformer-based architecture to address multi1011 variate clinical time series with missing values.”
Background
Metadata
- Authors
- Tipirneni, S., & Reddy, C. K.
- Affiliations
- Virginia Tech (US)
- Venue
- ACM Transactions on Knowledge Discovery from Data
- Published
- 2022-07-30
- Volume
- 16
- Issue
- 6
- Pages
- 1-17
- ISSN
- 1556-4681 (ISSN-L)
- 1556-472X
- OpenAlex
- W4283370926
- Cited by
- 130
- References
- 18
- Citations by year
- 2026: 34 · 2025: 40 · 2024: 33 · 2023: 20 · 2022: 3
- FWCI
- 7.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Knowledge Discovery from Data (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- VTechWorks (Virginia Tech) (submittedVersion, cc-by-nc)
- arXiv (Cornell University) (cc-by)
- Funders
- US National Science Foundation (IIS-1838730)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Keywords
- self-supervised learning, transformer models, irregularly sampled time series, missing data, time series forecasting, mortality prediction, limited labeled data, interpretable models, intensive care, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series. https://app.cochat.ai/papers/10.1145/3516367
- The paper
- Tipirneni, S., & Reddy, C. K. (2022). Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series. ACM Transactions on Knowledge Discovery from Data. https://doi.org/10.1145/3516367
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.