CoChat

Towards robust and label-efficient time series representation learning

Open access
Emadeldeen, Ahmed Ibrahim Ahmed Eldele · 2023

The DOI matches its Crossref record; 208 of the 209 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Time series data is sequential measurements collected over time from various sources in different applications, e.g., healthcare and manufacturing. With the increased generation of time series data from these applications, their analysis is becoming more important to get insights. Deep learning has shown a potential and proven capability in automatic learning from massive data, by identifying complex patterns and representations directly from data. However, the current deep learning-based models suffer significant limitations. First, they lack the ability to efficiently learn time series temporal relations while utilizing parallel processing. Second, these models require large amounts of labeled data for training, which can be difficult to obtain, especially with complex time series data. Third, the generalization capability of these models is limited, where they suffer performance deterioration when transferring knowledge from a labeled source domain to an out-of-distribution unlabeled target domain. In this thesis, we address these problems and provide solutions for the real-world deployment of deep learning models for time series data. We first propose a novel attention-based deep learning architecture called AttnSleep to classify EEG-based sleep stages, being one of the common time series healthcare data types. Specifically, we propose a powerful feature extractor that learns from different frequency bands in EEG signals. We also propose a temporal context encoder module to learn the temporal dependencies among extracted features using a causal multi-head attention mechanism. Last, we develop a class-aware loss function to address the class-imbalance problem in sleep data without incurring any additional computational costs. Next, we propose two frameworks to address the label scarcity problem in different settings. The first framework, TS-TCC, is a self-supervised learning approach that learns useful representations from unlabeled data. TS-TCC utilizes time series-specific augmentations to generate two views for each sample. We then learn the temporal representations via a novel cross-view temporal prediction task. Furthermore, we propose a contextual contrasting module that further learns discriminative representations. The second framework, CA-TCC, improves the learned representations from TS-TCC in semi-supervised settings, by training the model in four phases. First, we perform self-supervised training with TS-TCC. Then, we fine-tune the pretrained model with the available few labeled samples. Following that, we use the fine-tuned model to assign pseudo labels to the unlabeled set. Finally, we leverage these pseudo labels to realize a class-aware contrastive loss for semi-supervised training. These two frameworks showed significant performance improvement with having few labeled samples compared to traditional supervised training. Last, we tackle the domain shift problem and propose two novel frameworks to address this issue. In the first framework, we introduce an adversarial domain adaptation technique named ADAST, that addresses two challenges, namely the loss of domain-specific information during feature extraction and the ignorance of class information in the target domain during domain alignment. To overcome these challenges, we incorporate an unshared attention mechanism and an iterative self-training strategy with dual distinct classifiers. In the second framework, we attempt to overcome the complexity of adversarial training and present a novel approach called CoTMix to address the domain shift with a simple yet effective contrastive learning strategy. In specific, we propose a cross-domain temporal mixup strategy to create source-dominant and target-dominant domains. These domains serve as augmented views for the source and target domains in contrastive learning. Unlike prior works, CoTMix maps the source and target domains to an intermediate domain. These frameworks showed improved robustness of deep learning models on time series data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.32657/10356/170673

Cites

208 of 209 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 209 references

Research works printed but not linked: 24

  1. Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals. Circulation, 101(23):215-220, 2000. xxi, 47, 48
  2. Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1):162-169, 2017. doi: 10.21629/ JSEE.2017.01.18. 4 · doi:10.21629/jsee.2017.01.18.4
  3. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR. 2016.90. 12 · doi:10.1109/cvpr.2016.90.12
  4. Chien-Liang Liu, Wen-Hoar Hsaio, and Yao-Chung Tu. Time series classi- fication with multivariate convolutional neural network. IEEE Transactions on Industrial Electronics, 66(6):4788-4797, 2019. doi: 10.1109/TIE.2018. 2864702. 13 · doi:10.1109/tie.2018.2864702.13
  5. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ 5cf68969fb67aa6082363a6d4e6468e2-Paper.pdf. 14
  6. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. 14, 44, 65, 141
  7. Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image coloriza- tion. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision -ECCV 2016, pages 649-666, Cham, 2016. Springer In- ternational Publishing. ISBN 978-3-319-46487-9. 19
  8. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
  9. Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. Multi-task self-supervised learning for human activity detection. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 3(2), jun 2019. doi: 10.1145/3328932. URL https: //doi.org/10.1145/3328932. 20 · DOI · Paper page
  10. Xue Jiang, Jianhui Zhao, Bo Du, and Zhiyong Yuan. Self-supervised con- trastive learning for eeg-based sleep staging. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021. doi: 10.1109/ IJCNN52387.2021.9533305. 21, 33 · doi:10.1109/ijcnn52387.2021.9533305.21
  11. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 06964dce9addb1c5cb5d6e3d9838f733-Paper.pdf. 23, 88
  12. Xuxin Liu, Fengbin Zhang, Han Liu, and Haoyi Fan. itimes: Investigating semi-supervised time series classification via irregular time sampling. IEEE Transactions on Industrial Informatics, pages 1-9, 2022. doi: 10.1109/TII. 2022.3199374. 24 · doi:10.1109/tii.2022.3199374.24
  13. Qingchang Zhu, Zhenghua Chen, and Yeng Chai Soh. A novel semisupervised deep learning method for human activity recognition. IEEE Transactions on Industrial Informatics, 15(7):3821-3830, 2019. doi: 10.1109/TII.2018. 2889315. 25 · doi:10.1109/tii.2018.2889315.25
  14. Curran As- sociates, Inc., 2014. URL https://proceedings.neurips.cc/paper/2014/ file/375c71349b295fbe2dcdca9206f20a06-Paper.pdf. 26
  15. Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012. 27
  16. Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Con- ditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, volume 31, pages 1640-1650, 2018. 29, 108, 109, 125
  17. Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, and Xiaoli Li. Self-supervised learning for label-efficient sleep stage classification: A comprehensive evaluation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:1333-1342, 2023. doi: 10.1109/ TNSRE.2023.3245285. 37, 61 · doi:10.1109/tnsre.2023.3245285.37
  18. Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Con- trastive representation learning for electroencephalogram classification. In Proceedings of the Machine Learning for Health NeurIPS Workshop, volume 136 of Proceedings of Machine Learning Research, pages 238-253, 2020. URL https://proceedings.mlr.press/v136/mohsenvand20a.html. 63
  19. Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 7343-7352, 2020. doi: 10.1109/CVPR42600. 2020.00737. 78 · doi:10.1109/cvpr42600.2020.00737.78
  20. Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-15, 2023. doi: 10.1109/ TPAMI.2023.3308189. 81 · doi:10.1109/tpami.2023.3308189.81
  21. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative ad- versarial nets. In Advances in Neural Information Processing Systems 27, pages 2672-2680, 2014. 101, 102
  22. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020. 121
  23. Curran Asso- ciates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/ file/8edd72158ccd2a879f79cb2538568fdc-Paper.pdf. 137
  24. Tsung-Yu Hsieh, Suhang Wang, Yiwei Sun, and Vasant Honavar. Explainable multivariate time series classification: A deep neural network which learns to attend to important variables as well as time intervals. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21, page 607-615, 2021. ISBN 9781450382977. 141

Cited by

See all 1

Metadata

Authors
Emadeldeen, A. I. A. E.
Published
2023-01-01
OpenAlex
W4387309943
Cited by
1
References
206
Citations by year
2024: 1
Type
dissertation
Language
en
Open access
Licence
cc-by-nc
Locations
DR-NTU (Nanyang Technological University) (submittedVersion, cc-by-nc)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
time series representation learning, attention-based deep learning, multi-head attention, class imbalance, self-supervised learning, semi-supervised learning, limited labeled data, pseudo-labeling, contrastive learning, domain adaptation, adversarial domain adaptation, temporal prediction, temporal relations, few-shot learning, out-of-distribution generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards robust and label-efficient time series representation learning. https://app.cochat.ai/papers/10.32657/10356/170673
The paper
Emadeldeen, A. I. A. E. (2023). Towards robust and label-efficient time series representation learning. https://doi.org/10.32657/10356/170673

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.