Towards robust and label-efficient time series representation learning
Open accessThe DOI matches its Crossref record; 208 of the 209 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Time series data is sequential measurements collected over time from various sources in different applications, e.g., healthcare and manufacturing. With the increased generation of time series data from these applications, their analysis is becoming more important to get insights. Deep learning has shown a potential and proven capability in automatic learning from massive data, by identifying complex patterns and representations directly from data. However, the current deep learning-based models suffer significant limitations. First, they lack the ability to efficiently learn time series temporal relations while utilizing parallel processing. Second, these models require large amounts of labeled data for training, which can be difficult to obtain, especially with complex time series data. Third, the generalization capability of these models is limited, where they suffer performance deterioration when transferring knowledge from a labeled source domain to an out-of-distribution unlabeled target domain. In this thesis, we address these problems and provide solutions for the real-world deployment of deep learning models for time series data. We first propose a novel attention-based deep learning architecture called AttnSleep to classify EEG-based sleep stages, being one of the common time series healthcare data types. Specifically, we propose a powerful feature extractor that learns from different frequency bands in EEG signals. We also propose a temporal context encoder module to learn the temporal dependencies among extracted features using a causal multi-head attention mechanism. Last, we develop a class-aware loss function to address the class-imbalance problem in sleep data without incurring any additional computational costs. Next, we propose two frameworks to address the label scarcity problem in different settings. The first framework, TS-TCC, is a self-supervised learning approach that learns useful representations from unlabeled data. TS-TCC utilizes time series-specific augmentations to generate two views for each sample. We then learn the temporal representations via a novel cross-view temporal prediction task. Furthermore, we propose a contextual contrasting module that further learns discriminative representations. The second framework, CA-TCC, improves the learned representations from TS-TCC in semi-supervised settings, by training the model in four phases. First, we perform self-supervised training with TS-TCC. Then, we fine-tune the pretrained model with the available few labeled samples. Following that, we use the fine-tuned model to assign pseudo labels to the unlabeled set. Finally, we leverage these pseudo labels to realize a class-aware contrastive loss for semi-supervised training. These two frameworks showed significant performance improvement with having few labeled samples compared to traditional supervised training. Last, we tackle the domain shift problem and propose two novel frameworks to address this issue. In the first framework, we introduce an adversarial domain adaptation technique named ADAST, that addresses two challenges, namely the loss of domain-specific information during feature extraction and the ignorance of class information in the target domain during domain alignment. To overcome these challenges, we incorporate an unshared attention mechanism and an iterative self-training strategy with dual distinct classifiers. In the second framework, we attempt to overcome the complexity of adversarial training and present a novel approach called CoTMix to address the domain shift with a simple yet effective contrastive learning strategy. In specific, we propose a cross-domain temporal mixup strategy to create source-dominant and target-dominant domains. These domains serve as augmented views for the source and target domains in contrastive learning. Unlike prior works, CoTMix maps the source and target domains to an intermediate domain. These frameworks showed improved robustness of deep learning models on time series data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The taxonomies proposed by Eldele et al. [19], Deldari et al. [20] and Liu et al. [33] are somewhat similar to our proposed taxonomy, i.e., three taxonomies involve time series self-supervised contrastive learning methods.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [19]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, these two reviews only discuss a small part of self-supervised contrastive learning (SSCL), which requires a more comprehensive literature review.”
From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [19]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
Research works printed but not linked: 24
- Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals. Circulation, 101(23):215-220, 2000. xxi, 47, 48
- Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1):162-169, 2017. doi: 10.21629/ JSEE.2017.01.18. 4 · doi:10.21629/jsee.2017.01.18.4
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR. 2016.90. 12 · doi:10.1109/cvpr.2016.90.12
- Chien-Liang Liu, Wen-Hoar Hsaio, and Yao-Chung Tu. Time series classi- fication with multivariate convolutional neural network. IEEE Transactions on Industrial Electronics, 66(6):4788-4797, 2019. doi: 10.1109/TIE.2018. 2864702. 13 · doi:10.1109/tie.2018.2864702.13
- Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ 5cf68969fb67aa6082363a6d4e6468e2-Paper.pdf. 14
- Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. 14, 44, 65, 141
- Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image coloriza- tion. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision -ECCV 2016, pages 649-666, Cham, 2016. Springer In- ternational Publishing. ISBN 978-3-319-46487-9. 19
- Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
- Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. Multi-task self-supervised learning for human activity detection. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 3(2), jun 2019. doi: 10.1145/3328932. URL https: //doi.org/10.1145/3328932. 20 · DOI · Paper page
- Xue Jiang, Jianhui Zhao, Bo Du, and Zhiyong Yuan. Self-supervised con- trastive learning for eeg-based sleep staging. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021. doi: 10.1109/ IJCNN52387.2021.9533305. 21, 33 · doi:10.1109/ijcnn52387.2021.9533305.21
- Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 06964dce9addb1c5cb5d6e3d9838f733-Paper.pdf. 23, 88
- Xuxin Liu, Fengbin Zhang, Han Liu, and Haoyi Fan. itimes: Investigating semi-supervised time series classification via irregular time sampling. IEEE Transactions on Industrial Informatics, pages 1-9, 2022. doi: 10.1109/TII. 2022.3199374. 24 · doi:10.1109/tii.2022.3199374.24
- Qingchang Zhu, Zhenghua Chen, and Yeng Chai Soh. A novel semisupervised deep learning method for human activity recognition. IEEE Transactions on Industrial Informatics, 15(7):3821-3830, 2019. doi: 10.1109/TII.2018. 2889315. 25 · doi:10.1109/tii.2018.2889315.25
- Curran As- sociates, Inc., 2014. URL https://proceedings.neurips.cc/paper/2014/ file/375c71349b295fbe2dcdca9206f20a06-Paper.pdf. 26
- Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012. 27
- Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Con- ditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, volume 31, pages 1640-1650, 2018. 29, 108, 109, 125
- Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, and Xiaoli Li. Self-supervised learning for label-efficient sleep stage classification: A comprehensive evaluation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:1333-1342, 2023. doi: 10.1109/ TNSRE.2023.3245285. 37, 61 · doi:10.1109/tnsre.2023.3245285.37
- Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Con- trastive representation learning for electroencephalogram classification. In Proceedings of the Machine Learning for Health NeurIPS Workshop, volume 136 of Proceedings of Machine Learning Research, pages 238-253, 2020. URL https://proceedings.mlr.press/v136/mohsenvand20a.html. 63
- Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 7343-7352, 2020. doi: 10.1109/CVPR42600. 2020.00737. 78 · doi:10.1109/cvpr42600.2020.00737.78
- Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-15, 2023. doi: 10.1109/ TPAMI.2023.3308189. 81 · doi:10.1109/tpami.2023.3308189.81
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative ad- versarial nets. In Advances in Neural Information Processing Systems 27, pages 2672-2680, 2014. 101, 102
- Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020. 121
- Curran Asso- ciates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/ file/8edd72158ccd2a879f79cb2538568fdc-Paper.pdf. 137
- Tsung-Yu Hsieh, Suhang Wang, Yiwei Sun, and Vasant Honavar. Explainable multivariate time series classification: A deep neural network which learns to attend to important variables as well as time intervals. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21, page 607-615, 2021. ISBN 9781450382977. 141
Cited by
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
Metadata
- Authors
- Emadeldeen, A. I. A. E.
- Publisher
- Nanyang Technological University
- Published
- 2023-01-01
- OpenAlex
- W4387309943
- Cited by
- 1
- References
- 206
- Citations by year
- 2024: 1
- Type
- dissertation
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- cc-by-nc
- Locations
- DR-NTU (Nanyang Technological University) (submittedVersion, cc-by-nc)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- time series representation learning, attention-based deep learning, multi-head attention, class imbalance, self-supervised learning, semi-supervised learning, limited labeled data, pseudo-labeling, contrastive learning, domain adaptation, adversarial domain adaptation, temporal prediction, temporal relations, few-shot learning, out-of-distribution generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards robust and label-efficient time series representation learning. https://app.cochat.ai/papers/10.32657/10356/170673
- The paper
- Emadeldeen, A. I. A. E. (2023). Towards robust and label-efficient time series representation learning. https://doi.org/10.32657/10356/170673
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.