Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Multivariate time-series data are frequently observed in critical care settings and are typically characterized by sparsity (missing information) and irregular time intervals. Existing approaches for learning representations in this domain handle these challenges by either aggregation or imputation of values, which in-turn suppresses the fine-grained information and adds undesirable noise/overhead into the machine learning model. To tackle this problem, we propose a S elf-supervised Tra nsformer for T ime- S eries (STraTS) model, which overcomes these pitfalls by treating time-series as a set of observation triplets instead of using the standard dense matrix representation. It employs a novel Continuous Value Embedding technique to encode continuous time and variable values without the need for discretization. It is composed of a Transformer component with multi-head attention layers, which enable it to learn contextual triplet embeddings while avoiding the problems of recurrence and vanishing gradients that occur in recurrent architectures. In addition, to tackle the problem of limited availability of labeled data (which is typically observed in many healthcare applications), STraTS utilizes self-supervision by leveraging unlabeled data to learn better representations by using time-series forecasting as an auxiliary proxy task. Experiments on real-world multivariate clinical time-series benchmark datasets demonstrate that STraTS has better prediction performance than state-of-the-art methods for mortality prediction, especially when labeled data is limited. Finally, we also present an interpretable version of STraTS, which can identify important measurements in the time-series data. Our data preprocessing and model implementation codes are available at https://github.com/sindhura97/STraTS .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, interpolation-based models add undesirable noise and extra overhead to the model which usually worsens as the time series become increasingly sparse [54].”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Though this technique has shown great performance boosts with image [15] and text [9, 31] data, its application to time-series data has been limited.”
BackgroundGoldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 2×
“The success of deep learning in image and text domains realized by convolutional and recurrent networks [7, 28], and Transformer models [30] have inspired the application of these architectures to develop better prediction models for time-series data as well.”
BackgroundMIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 10×
“For MIMIC-III dataset, we set the maximum number of time-steps for GRU-D and InterpNet, and the maximum no. of observations for STraTS using the 99th percentile for the same.”
MethodInfluentialProceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
Gaussian Processes in Machine Learning
Rasmussen, C. E. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-28650-9_4 · Cited by 5,397
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
Bai, S., Kolter, J. Z., & Koltun, V. (2018) · arXiv · doi:10.48550/arxiv.1803.01271 · Cited by 4,655 · cited 2×
“We expected the interpolation-based models GRU-D and InterpNet to outperform the simpler models GRU, TCN, and SaND.”
BackgroundRecurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · cited 10×
“Existing sequence models for clinical time-series [4] and other interpolationbased models [26] address this issue by including a learnable imputation or interpolation component.”
BackgroundMethodInfluentialSelf-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036 · cited 1×
“Though this technique has shown great performance boosts with image [15] and text [9, 31] data, its application to time-series data has been limited.”
BackgroundA Transformer-based Framework for Multivariate Time Series Representation Learning
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., & Eickhoff, C. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467401 · Cited by 1,278 · cited 1×
“[33] pretrained a Transformer model using a denoising objective and showed improved performance on regression and classification tasks with dense multivariate time-series.”
Method
Research works printed but not linked: 13
- Edwin V. Bonilla, Kian Ming Adam Chai, and Christopher K. I. Williams. 2007. Multi-task gaussian process prediction. In Proceedings of the 21st Annual Conference on Advances in Neural Information Processing Systems. Curran Associates, Inc., 153-160.
- Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter F. Stewart. 2016. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems. 3504-3512.
- A. Farhana and S. L. Lappin. 2021. Biochemistry, Lactate Dehydrogenase.[Updated 2020 May 17]. StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing (2021).
- Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten M. Borgwardt. 2020. Set functions for time series. In Proceedings of the 37th International Conference on Machine Learning, (ICML'20), Vol. 119. PMLR, 4353-4363.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd Inter- national Conference on Learning Representations, (ICLR'15).
- Steven Cheng-Xian Li and Benjamin M. Marlin. 2015. Classification of sparse and irregularly sampled time series with mixtures of expected gaussian kernels and random features. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. AUAI Press, 484-493.
- Steven Cheng-Xian Li and Benjamin M. Marlin. 2016. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems (2016). 1804-1812.
- Zachary C. Lipton, David C. Kale, and Randall C. Wetzel. 2016. Directly modeling missing data in sequences with RNNs: Improved classification of clinical time series. In Proceedings of the 1st Machine Learning in Health Care, MLHC 2016 (JMLR Workshop and Conference Proceedings, Vol. 56). JMLR.org, 253-270.
- Yulia Rubanova, Tian Qi Chen, and David Duvenaud. 2019. Latent ordinary differential equations for irregularly- sampled time series. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, (2019) NeurIPS'19. 5321-5331.
- Satya Narayan Shukla and Benjamin M. Marlin. 2019. Interpolation-prediction networks for irregularly sampled time series. In Proceedings of the 7th International Conference on Learning Representations, (ICLR'19). Retrieved from Open- Review.net. https://openreview.net/forum?id=r1efr3C9Ym.
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceed- ings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (2014). 3104-3112.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Gen- eralized autoregressive pretraining for language understanding. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS'19). 5754-5764.
Cited by
Transformers in Time-Series Analysis: A Tutorial
Ahmed, S., Nielsen, I. E., Tripathi, A., Siddiqui, S., Ramachandran, R. P., & Rasool, G. (2023) · Circuits, Systems, and Signal Processing · doi:10.1007/s00034-023-02454-8 · Cited by 305
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“Moreover, STraTS [54] first encodes the time series data into triple representations to avoid the limitations of using basic RNN and CNN in modeling irregular and sparse time series data and then builds the transformer-based forecasting model for modeling multivariate medical clinical time series.”
BackgroundMethodA systematic review for transformer-based long-term series forecasting
Su, L., Zuo, X., Li, R., Wang, X., Zhao, H., & Huang, B. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11044-2 · Cited by 192
Patharkar, A., Cai, F., Al-Hindawi, F., & Wu, T. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1386760 · Cited by 108
“For instance, Tipirneni and Reddy (2022) introduced the Self-supervised Transformer for Time-Series (STraTS) model, which treats each time-series as observation triplets (time, variable, value) instead of matrices as done by conventional methods.”
BackgroundA Survey on Attention Mechanisms for Medical Applications: are we Moving Toward Better Algorithms?
Gonçalves, T., Rio-Torto, I., Teixeira, L. F., & Cardoso, J. S. (2022) · IEEE Access · doi:10.1109/access.2022.3206449 · Cited by 100
“[181] pro1010 posed a Transformer-based architecture to address multi1011 variate clinical time series with missing values.”
BackgroundGraph-Guided Network for Irregularly Sampled Multivariate Time SeriesPreprint
Zhang, X., Zeman, M., Tsiligkaridis, T., & Žitnik, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05357 · Cited by 182
A Review of Deep Learning Methods for Irregularly Sampled Medical Time Series Data
Sun, C., Song, M., Cai, D., Zhang, B., Li, H., & Hong, S. (2026) · Health Data Science · doi:10.34133/hds.0456 · Cited by 104
“…BiCMTS [ 110 ] 0.834 ± 0.002 0.819 ± 0.013 0.847 ± 0.014 0.960 ± 0.005 MIAM [ 95 ] 0.832 ± 0.005 0.822 ± 0.012 0.849 ± 0.010 0.962 ± 0.003 STraTS [ 150 ] 0.832 ± 0.005 0.819 ± 0.010 0.850 ± 0.013 0.961 ± 0.005 End-to-end (raw-data-based methods) IPN [ 22 ] 0.831 ± 0.003 0.823 ± 0.009 0.844 ±…”
InfluentialHan, L., Abdel-Aty, M. A., Yu, R., & Wang, C. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3438616 · Cited by 54
“In this paper, a Transformer model [48], [52] was proposed to extract the non-aggregate risky driving behavior features.”
BackgroundMethodWarpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series
Zhang, J., Zheng, S., Cao, W., Bian, J., & Li, J. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599543 · Cited by 121
Label-Efficient Time Series Representation Learning: A Review
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., & Li, X. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3430236 · Cited by 69
Transformer-based modeling of abnormal driving events for freeway crash risk evaluation
Han, L., Yu, R., Wang, C., & Abdel-Aty, M. A. (2024) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2024.104727 · Cited by 44
Applications of Self-Supervised Learning to Biomedical Signals: A Survey
Pup, F. D., & Atzori, M. (2023) · IEEE Access · doi:10.1109/access.2023.3344531 · Cited by 43
“Ultimately, Tipirneni et al. [192] pretrained the model with a forecasting pretext task.”
MethodA review of time series forecasting and spatio-temporal series forecasting in deep learning
Yu, Q., Yang, G., Wang, X., Shi, Y., Feng, Y., & Liu, A. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07632-w · Cited by 52
Systematic Review of Advanced AI Methods for Improving Healthcare Data Quality in Post COVID-19 Era
Isgut, M., Gloster, L., Choi, K., Venugopalan, J., & Wang, M. D. (2023) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3216531 · Cited by 35
“Other multivariate time series imputation methods published in 2020 and 2021 have been developed using novel approaches, including random drop [84], transformers [85], self-attention [86], and conditional score-based diffusion [87] models.”
MethodChang, P., Li, H., Quan, S. F., Lu, S., Wung, S., Roveda, J. M., & Li, A. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108060 · Cited by 56
“All of these factors contribute to the extreme sparsity of ICU data [8,9].”
BackgroundMethodPre-training in Medical Data: A Survey
Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30
“However, if the pre-trained model consists of abundant auxiliary tasks and has a complex relationship to the target task, using the pre-trained model becomes inefficient and subnormal[217].”
BackgroundMethodEarly prediction of sepsis using double fusion of deep features and handcrafted features
Duan, Y., Huo, J., Chen, M., Hou, F., Yan, G., Li, S., & Wang, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-04425-z · Cited by 34
Nejad, A. S., Aláiz-Rodríguez, R., McCarthy, G., Kelleher, B. P., Grey, A., & Parnell, A. C. (2024) · Computers & Geosciences · doi:10.1016/j.cageo.2024.105601 · Cited by 32
Håkansson, S., Tuci, M., Bolliger, M., Curt, A. E. P., Jutzeler, C. R., & Brüningk, S. C. (2024) · Experimental Neurology · doi:10.1016/j.expneurol.2024.114913 · Cited by 34
Contreras, M., Kapoor, S., Zhang, J., Davidson, A., Ren, Y., Guan, Z., Ozrazgat‐Baslanti, T., Sena, J., Nerella, S., Bihorac, A., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-22634-7 · Cited by 39
“The Transformer and Mamba baselines were adapted from previous clinical prediction architectures 15,36,37 , converting time-series measurements into token embeddings, which were then fused with static data.”
Ghaderi, H., Foreman, B. P., Nayebi, A., Tipirneni, S., Reddy, C. K. A., & Subbian, V. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104401 · Cited by 27
“To obtain the representation of samples, we use the STraTS model from our prior work in [9].”
BackgroundMethodInfluentialSelf-citationLi, T. Z., Still, J. M., Xu, K., Lee, H. H., Cai, L. Y., Krishnan, A. R., Gao, R., Khan, M. S., Antic, S., Kammer, M. N., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_61 · Cited by 33
DNA-T: Deformable Neighborhood Attention Transformer for Irregular Medical Time Series
Huang, J., Yang, B., Yin, K., & Xu, J. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3395446 · Cited by 29
“Moreover, compared to recent attention-based methods [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], DNA-T additionally introduces timestamps, masks, and time intervals of irregular time series into the attention mechanism, improving knowledge used for attention learning.”
ResultTan, Y., Dede, M., Mohanty, V., Dou, J., Hill, H., Bernstam, E., & Chen, K. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104648 · Cited by 19
Chen, L., Hung, K.-H., Tseng, Y., Wang, H., Lu, T., Huang, W., & Tsao, Y. (2023) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2023.3307794 · Cited by 25
“Tipirneni and Reddy [27] bypass the irregular, absent, and sparse nature of EHRs by training the model in a SSL manner and mask out unobserved forecasts in the loss function during training.”
BackgroundContinuous patient state attention model for addressing irregularity in electronic health records
Chauhan, V. K., Thakur, A., O’Donoghue, O., Rohanian, O., Molaei, S., & Clifton, D. A. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02514-2 · Cited by 20
Mehdizavareh, M. H., Khan, A., & Cichosz, S. L. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2025.06.039 · Cited by 18
“To compare MITST with more advanced deep learning models, we implemented and evaluated STraTS [30], a recent Transformer-based architecture specifically designed for irregular clinical time series.”
InfluentialBashiri, F. S., Carey, K. A., Martin, J., Koyner, J. L., Edelson, D. P., Gilbert, E. R., Mayampurath, A. M., Afshar, M., & Churpek, M. M. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae088 · Cited by 18
MedLens: Improve Mortality Prediction Via Medical Signs Selecting and Regression
Ye, X., Wu, J., Mou, C., & Dai, W. (2023) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai57533.2023.10201302 · Cited by 22
“[20] introduce a self-supervised transformer approach for sparse and irregularly sampled clinical time series, improving mortality prediction accuracy.”
BackgroundEarly Prediction of Sepsis Using Time Series Forecasting
Xu, J., Minakova, N., Sanchez, P. O., & Riezler, S. (2023) · IEEE International Conference on eScience · doi:10.1109/e-science58273.2023.10254852 · Cited by 16
“Our project is based on the STraTS architecture [1], and as such, we utilized their preprocessing approach to prepare the data in the required format.”
BackgroundMethodInfluentialHierarchical Pretraining on Multimodal Electronic Health Records
Wang, X., Luo, J., Wang, J., Yin, Z., Cui, S., Zhong, Y., Wang, Y., & Ma, F. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.171 · Cited by 30
TS-TFSIAM: Time-series self-supervised learning with time-frequency SiameseNet
Liu, S., Li, H., Huang, Y., & Wen, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111472 · Cited by 12
Contreras, M., Silva, B., Shickel, B., Davidson, A., Ozrazgat‐Baslanti, T., Ren, Y., Guan, Z., Balch, J. A., Zhang, J., Bandyopadhyay, S., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-62121-1 · Cited by 20
“Finally, the APRICOT-M model, which uses the embedding schema of a clinical time-series model 22 with incorporation of a state-space model (Mamba block, details in Model development and performance section) 23 , was trained to predict probabilities for each acuity outcome, transition, and…”
IVP-VAE: Modeling EHR Time Series with Initial Value Problem Solvers
Xiao, J., Basso, L., Nejdl, W., Ganguly, N., & Sikdar, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29534 · Cited by 21
Yu, R., He, Y., Li, H., Li, S., & Jian, B. (2024) · Accident Analysis & Prevention · doi:10.1016/j.aap.2024.107698 · Cited by 12
Bignoumba, N., Mellouli, N., & Yahia, S. B. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122148 · Cited by 12
Liu, T., Liang, L., Che, C., Liu, Y., & Jin, B. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104826 · Cited by 11
Discovery of generalizable TBI phenotypes using multivariate time-series clustering
Ghaderi, H., Foreman, B. P., Reddy, C. K. A., & Subbian, V. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108997 · Cited by 10
“We utilize SLAC-Time, built on a self-supervised Transformer model known as STraTS [26], for representation learning of N unlabeled samples represented by 𝒟 = {(d 𝑘 , T 𝑘 )} 𝑘=1𝑁”
Self-citationHGV4Risk: Hierarchical Global View-guided Sequence Representation Learning for Risk Prediction
Li, Y., Zhu, Z., Guo, X., Li, S., Yang, Y., & Zhao, Y. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3605895 · Cited by 16
“Aiming at predicting the occurrence probability of the event of interest, whether the patient will develop a disease [36] or whether the user will engage in non-conforming behavior, the risk prediction modeling is commonly used in the world of medicine [9, 11, 23] to help guide clinical…”
MethodWang, Z., Zhang, Y., Pang, Y., Wang, N., Bah, M. J., Li, K., & Zhang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3305542 · Cited by 9
Irregularity-Informed Time Series Analysis: Adaptive Modelling of Spatial and Temporal Dynamics
Zheng, L. N., Li, Z., Dong, C. G., Zhang, W. E., Yue, L., Xu, M., Maennel, O., & Chen, W. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679716 · Cited by 8
“Although there are some works [42, 42, 32] utilizing the Transformer backbone for IRTS, they also ignored the information from irregular patterns such as sampling frequency and sampling density, which may introduce measurement bias from the original value observation due to a wider range of…”
EMIT - Event-Based Masked Auto Encoding for Irregular Time Series
Patel, H., Qiu, R., Irwin, A., Sadiq, S. W., & Wang, S. (2024) · Proceedings · doi:10.1109/icdm59182.2024.00044 · Cited by 12
“For fair evaluation, the hyperparameters of the Transformer model were kept similar to STraTS [43]: 2 encoder blocks and 4 attention heads.”
InfluentialSpatial-Temporal Transformer for Crime Recognition in Surveillance Videos
Boekhoudt, K., & Talavera, E. (2022) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss56176.2022.9959414 · Cited by 8
Bhardwaj, S., Vekariya, V., Singh, B., Vinay, S., Arul, A., & Roopa, M. D. (2024) · Measurement Sensors · doi:10.1016/j.measen.2024.101053 · Cited by 12
Accounting data anomaly detection and prediction based on self-supervised learning
Zhang, Y., & Duan, B. (2025) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2025.1628652 · Cited by 10
Forecasting Treatment Outcomes Over Time Using Alternating Deep Sequential Models
Wu, F. C., Zhao, G., Zhou, Y., Qian, X., Elias, B.-K., & Lehman, L.-W. H. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3331298 · Cited by 11
“To address the issue of sparsity and irregularity in treatment sequences, STraTS [20] employs a self-supervised Transformer architecture for modeling time series.”
MethodT-Rep: Representation Learning for Time Series using Time-Embeddings
Fraikin, A., Bennetot, A., & Allassonnière, S. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.04486 · Cited by 25
“In existing self-supervised models, the time-embedding is used by the encoder to provide positional information (Zerveas et al., 2021; Tipirneni & Reddy, 2022; Haresamudram et al., 2020; Zhang et al., 2022), but is never exploited in pretext tasks.”
MethodA Foundation Model for Intensive Care: Unlocking Generalization across Tasks and Domains at ScalePreprint
Burger, M., Chopard, D., Lichtner, G., Londschien, M., Sergeev, F., Fuchs, M., Yèche, H., Kuznetsova, R., Faltys, M., Gerdes, E., et al. (2025) · medRxiv · doi:10.1101/2025.07.25.25331635 · Cited by 20
Self-attention with temporal prior: can we learn more from the arrow of time?
Kim, K. G., & Lee, B. T. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1397298 · Cited by 9
Adapting to the stream: an instance-attention GNN method for irregular multivariate time series data
Han, K., Koay, A., Ko, R. K. L., Chen, W., & Xu, M. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-40449-z · Cited by 6
Zou, M., An, Y., Kuang, H., & Wang, J. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104408 · Cited by 6
Hybrid CAE-DSVDD for unsupervised vibration-based damage detection in in situ steel truss bridge
Park, S., & Kim, S.-J. (2025) · Structural Health Monitoring · doi:10.1177/14759217251369727 · Cited by 4
Kurasawa, H., Waki, K., Seki, T., Chiba, A., Fujino, A., Hayashi, K., Nakahara, E., Haga, T., Noguchi, T., & Ohe, K. (2024) · JMIR AI · doi:10.2196/56700 · Cited by 7
“Irregularly spaced data require preprocessing techniques such as interpolation, denoising autoencoders, and self-supervised learning [14-17].”
Multi-Modal Fusion of Routine Care Electronic Health Records (EHR): A Scoping Review
Miled, Z. B., Shebesh, J. A., Su, J., Dexter, P. R., Grout, R. W., & Boustani, M. A. (2025) · Information · doi:10.3390/info16010054 · Cited by 12
“This latter technique is similar to the time delay embedding which is used for time series forecasting [96].”
Zhong, S., Wang, L. R., Zhan, Z., Ng, Y. Y., & Fan, X. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3743689 · Cited by 7
“It then passed the summarized observations to an attention mechanism to learn the importance of each observation, which is then used in the final classification layer.”
InfluentialLi, B., Du, B., & Ye, J. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3744251 · Cited by 6
TransEHR: Alignment-free electronic health records continual learning across feature spaces
Zhao, X., Liu, Z., Ji, B., Xi, P., & Peng, S. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129020 · Cited by 5
Embracing data irregularities in multivariate time series
de Barros, M. R., Fontenele, L. P., Silva, M. S., Rissi, T. L., Cabrera, E. F., Tannuri, E. A., Gomi, E. S., Barreira, R. A., & Costa, A. H. R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.114039 · Cited by 4
Liu, X., Zhi, X., Zhou, T., Zhao, L., Tian, L., Gao, R., Luo, J., Cui, W., & Wang, Q. (2025) · Scientific Data · doi:10.1038/s41597-025-05591-8 · Cited by 5
“The time embedding emb it and value embedding emb iv use the continuous value embedding method (CVE) proposed by Tipirneni and Reddy 49 .”
Plausible Vehicle Trajectory Augmentation With Hybrid Machine Learning
Park, J., Jang, H., Kang, B., & Hwang, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3573191 · Cited by 3
“To address these challenges, transformer-based models such as STraTS [17] have been introduced, offering improved performance in modeling complex and dynamic traffic patterns.”
Zeng, Z., Lin, M., Liu, Y., Yao, S., & Gong, X. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01303-6 · Cited by 3
“Tip-irneni and Reddy [15] used similar method in their STraTS model.”
Multi-Level Transfer Learning for irregular clinical time series prediction
Li, X., Teng, F., Ma, M., Qin, Z., Guo, J. K., Xu, J., & Li, T. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113825 · Cited by 7
Ferreira, J. S., Gradvohl, A. L. S., da Silva, A. E. A., Coelho, G. P., & Cinto, T. (2026) · Advances in Space Research · doi:10.1016/j.asr.2026.03.067 · Cited by 3
Artificial intelligence in triage of COVID-19 patients
de Oliveira, Y. T., Rios, I., Araújo, P., Macambira, A. K. L. S., Guimarães, M. D., Sales, L., Júnior, M. R., Nicola, A. M., Nakayama, M., Paschoalick, H. N. D. S., et al. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1495074 · Cited by 3
An Efficient Contrastive Unimodal Pretraining Method for EHR Time Series Data
King, R., Kodali, S., Krueger, C., Yang, T., & Mortazavi, B. J. (2024) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi62660.2024.10913624 · Cited by 3
“There are many advantages to the triplet embedding in [9] over the dense imputation strategy listed above.”
InfluentialMTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification
Zhong, S., Zhuo, W., Song, S., Li, G., Yu, Z., & Chan, S.-H. G. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737058 · Cited by 10
Jeong, J., Oh, Y., Kim, D., & Yoon, H. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.115251 · Cited by 4
Wang, Z., Liu, T., & Yao, B. (2026) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2026.3672337 · Cited by 5
“Tipirneni and Reddy developed a method that integrated a continuous value embedding technique with self-attention to model irregularly sampled clinical time series by treating them as a set of observation triplets (time, variable, and value) [37].”
Picheta, P., & Deniziak, S. R. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132084 · Cited by 2
Chen, Z., Zhang, L., Chen, J., Li, L., & Zhang, J. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109029 · Cited by 2
Privacy-Conscious Internet Behavior for Depression Detection With Cross-Scale Adaptive Transformer
Yang, M., Bai, Y., Zheng, W., & Hu, B. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3572118 · Cited by 3
“Moreover, unsupervised networks like TST [45] and self-supervised networks like STraTS [46] are designed to address irregular time series problems.”
Choy, X. Y., Wang, L. R., Henderson, T. C., Li, K. Z., Ng, Y. Y., & Fan, X. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650842 · Cited by 2
“2) STraTS: We adopt STraTS [12] for vital sign prediction, keeping its self-supervision component.”
BackgroundMethodInfluentialOh, W., Jayaraman, P., Tandon, P., Chaddha, U., Kovatch, P., Charney, A. W., Glicksberg, B. S., & Nadkarni, G. N. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102750 · Cited by 2
A Global View-Guided Autoregressive Residual Network for Irregular Time Series Classification
Zhu, J., Tang, H., Zhang, L., Jin, B., Xu, Y., & Wei, X. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-33383-5_23 · Cited by 2
Rashidi, P., Contreras, M., Silva, B., Shickel, B., Davidson, A., Ozrazgat‐Baslanti, T., Ren, Y., Guan, Z., Balch, J. A., Zhang, J., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4790824/v1 · Cited by 9
“The APRICOT-M model was developed based on the embedding schema of a clinical Transformer architecture [19], incorporating a Mamba block [20].”
MethodInfluentialSigFormer: A Transformer for Signaling Data Augmentation via Location Reconstruction
Liu, M., Jiang, H., Zhu, T., Sun, L., Wang, J., & Sheng, H. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3599925 · Cited by 2
“Tipirneni et al. [43] treat irregularly sampled time series as continuous triplets and introduce a continuous value embedding technique to encode both continuous time and variable values.”
Gille, C., Altares, G., Colette, B., Boudjeltia, K. Z., Mancaş, M., & Vandenbulcke, V. (2025) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc58623.2025.11251723 · Cited by 1
“Finally, to reduce the sequence along the token dimension, a weighted sum of all tokens is performed with coefficients computed through an FFN also following (1), using the Transformer’s output as its input [33].”
InfluentialA Graph-Based Approach for Early and Explainable Health Risk Assessment
Jha, S., & Feng, W.-C. (2025) · IEEE International Conference on Bioinformatics and Bioengineering (BIBE) · doi:10.1109/bibe66822.2025.00054 · Cited by 1
“Attention-based methods, e.g., SeFT [6] and STraTS [7], learn relationships from (value, time, variable) triplets via attention layers.”
Bootstrapping Large Language Models as Clinical Predictors for Multimodal Irregular Time Series
He, Y., Cai, J., & Zhao, G. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356732 · Cited by 1
“First, following the existing triplet embedding strategies [10], [11], we deploy FFNs to respectively encode the continuous values and timestamps of each MICTS token.”
Video-dominant emotion recognition for portable EEG-based devices
Wen, X., Xu, W., Tian, L., Guo, C.-J., & Bai, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-39315-8 · Cited by 2
“In recent years, self-supervised learning has emerged as a promising direction for enhancing EEG feature extraction and representation learning 30,31 .”
Ding, X., Li, Z., Tang, M., Shen, X., & Zhou, L. (2026) · Electronics · doi:10.3390/electronics15040804 · Cited by 1
A Pre-processing Process Using TadGAN-based Time-series Anomaly Detection
Lee, S. H., & Kim, Y. S. (2022) · Journal of the Korean society for quality management · doi:10.7469/jksqm.2022.50.3.459 · Cited by 11
Chen, X., Zhang, K., Chuai, G., Gao, W., Liu, X., Zhang, Y., & Hou, Y. (2025) · Digital Communications and Networks · doi:10.1016/j.dcan.2025.04.003 · Cited by 2
A Three‐Stage Prediction Model Based on Transformer for Chronic Kidney Disease
Lu, Y., Chang, W., Yang, D., & Jiang, Y. (2025) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70322 · Cited by 1
“Convolutional operations [28] across both the temporal and feature dimensions, excelling at capturing the relationship between different features as well as temporal information.”
Mesinovic, M., Molaei, S., Watkinson, P., & Zhu, T. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02328-0 · Cited by 3
RetractedTS-TFMAE: time series self-supervised learning with time–frequency masked autoencoders
Liu, S., Li, H., & Yang, Y. (2026) · Applied Intelligence · doi:10.1007/s10489-026-07145-w · Cited by 1
“To overcome the drawbacks of employing simple RNNs and CNNs for modeling irregu - lar and sparse time series data, STraTS [22] first encodes the time series data into a ternary form.”
Li, B., Du, B., & Ye, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104347 · Cited by 2
LLM4Geopolitics : A Framework Leveraging Large Language Models for Predicting Geopolitical Events
Mouakher, A., Morgado, N., & Ftouhi, F. (2026) · Expert Systems · doi:10.1111/exsy.70258 · Cited by 2
Adelola, M. A., Adewale, O. S., Adewole, D. B., & Iwasokun, G. B. (2026) · International Journal of Research and Innovation in Applied Science · doi:10.51584/ijrias.2026.11050194 · Cited by 1
Bennis, F. C., Aussems, C., Korevaar, J. C., & Hoogendoorn, M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108097 · Cited by 1
Thapa, S., Salman, M., Shah, S. B., Shiwakoti, S., Zhang, Q., Hu, L., Razzak, I., & Naseem, U. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825881 · Cited by 1
Impact of 3D Cartesian Positions and Occlusion on Self-Avatar Full-Body Animation in Virtual Reality
Maiorca, A., Kinart, A., Fletcher, G., Ghasemzadeh, S. A., Ravet, T., Vleeschouwer, C. D., & Dutoit, T. (2025) · IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR) · doi:10.1109/aixvr63409.2025.00044 · Cited by 1
Tech Stack Prediction Using Hybrid ARIMA and LSTM Model
Senthilkumar, R., Naveen, V., BALAJI, M., & Aravinth, P. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-19-2894-9_26 · Cited by 1
DeepStay: Stay Region Extraction from Location Trajectories Using Weak Supervision
Löwens, C., Thyssens, D., Andersson, E. R., Jenkins, C., & Schmidt-Thieme, L. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422584 · Cited by 2
“Since [25], [26] show good results by using forecasting as a pretext task for time series data, we adopted their approach.”
MethodTowards robust and label-efficient time series representation learning
Emadeldeen, A. I. A. E. (2023) · doi:10.32657/10356/170673 · Cited by 1
Neuro-Markovian Approach to Forecasting Patient Vital Signs from Irregular Time-SeriesPreprint
Choy, X. Y., Wang, L. R., Henderson, T. C., Li, K. Z., Ng, Y. Y., & Fan, X. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5065782 · Cited by 1
Rahimi, F., Sirocchi, C., Matschinske, J., Metzler, M., Zierk, J., & Blumenthal, D. B. (2025) · medRxiv · doi:10.64898/2025.12.12.25342142 · Cited by 1
Parveen¹, S., Sanjana¹, S., Sakshi¹, S., & Kumar, E. (2026) · Research Square · doi:10.21203/rs.3.rs-9499930/v1 · Cited by 1
Stack Index Prediction Using Time-Series AnalysisPreprint
Raman, R. C., Mahadevan, R., Perumal, D., Sankar, V., & Rahman, T. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08120
Adversarial Learning for Improved Patient Representations
Shankar, B., & Hargreaves, C. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-32883-1_42
Metadata
- Authors
- Tipirneni, S., & Reddy, C. K.
- Affiliations
- Virginia Tech (US)
- Venue
- ACM Transactions on Knowledge Discovery from Data
- Published
- 2022-07-30
- Volume
- 16
- Issue
- 6
- Pages
- 1-17
- ISSN
- 1556-4681 (ISSN-L)
- 1556-472X
- OpenAlex
- W4283370926
- Cited by
- 130
- References
- 18
- Citations by year
- 2026: 34 · 2025: 40 · 2024: 33 · 2023: 20 · 2022: 3
- FWCI
- 7.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Knowledge Discovery from Data (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- VTechWorks (Virginia Tech) (submittedVersion, cc-by-nc)
- arXiv (Cornell University) (cc-by)
- Funders
- US National Science Foundation (IIS-1838730)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Keywords
- self-supervised learning, transformer models, irregularly sampled time series, missing data, time series forecasting, mortality prediction, limited labeled data, interpretable models, intensive care, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.