CoChat
Sign Up
PreprintCited by 34Domain Adaptation and Few-Shot LearningMachine Learning and AlgorithmsTopic Modeling

The DOI matches its DataCite record.

Sense and Learn: Self-Supervision for Omnipresent Sensors

Open access
Saeed, Aaqib, Ungureanu, Victor, Gfeller, Beat·2020·arXiv
DOI 10.48550/arxiv.2009.13233arXiv 2009.13233
Open this paper in CoChat — free accountOpen PDF

Abstract

Learning general-purpose representations from multisensor data produced by the omnipresent sensing systems (or IoT in general) has numerous applications in diverse use cases. Existing purely supervised end-to-end deep learning techniques depend on the availability of a massive amount of well-curated data, acquiring which is notoriously difficult but required to achieve a sufficient level of generalization on a task of interest. In this work, we leverage the self-supervised learning paradigm towards realizing the vision of continual learning from unlabeled inputs. We present a generalized framework named Sense and Learn for representation or feature learning from raw sensory data. It consists of several auxiliary tasks that can learn high-level and broadly useful features entirely from unannotated data without any human involvement in the tedious labeling process. We demonstrate the efficacy of our approach on several publicly available datasets from different domains and in various settings, including linear separability, semi-supervised or few shot learning, and transfer learning. Our methodology achieves results that are competitive with the supervised approaches and close the gap through fine-tuning a network while learning the downstream tasks in most cases. In particular, we show that the self-supervised network can be utilized as initialization to significantly boost the performance in a low-data regime with as few as 5 labeled instances per class, which is of high practical importance to real-world problems. Likewise, the learned representations with self-supervision are found to be highly transferable between related datasets, even when few labeled instances are available from the target domains. The self-learning nature of our methodology opens up exciting possibilities for on-device continual learning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2009.13233

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The proposed method outperforms existing self-supervised methods in human activity recognition on the UCI HAR, MotionSense, and HAPT datasets.

    “We show that our method outperforms existing self-supervised methods in HAR for UCI HAR, MotionSense, and HAPT datasets.”

    From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [20]

    Section 4.3.2 reports improved activity-recognition F-scores on UCI HAR, HAPT, and MotionSense, but compares the method with fully supervised models in a low-data setting. The comparison tables in Sections 4.3.3–4.3.4 include some self-supervised baselines, such as autoencoders; they do not establish the broader claim that the method outperforms existing self-supervised methods generally on all three datasets. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • For both UCI HAR and HAPT, our fine-tuned self-supervised models outperform the fine-tuned models proposed by [19] and [20].

    “Nonetheless, for both UCI HAR and HAPT, our fine-tuned self-supervised models outperform the fine-tuned models proposed by [19] and [20].”

    From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [20]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 59

  1. Bai, S., Kolter, J. Z., & Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01271. · arXiv
  2. Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., et al. (2019). Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046. · arXiv
  3. de Sa, V. R. (1994). Learning classification with unlabeled data. In Advances in neural information processing systems (pp. 112-119).
  4. Deldari, S., Smith, D. V., Xue, H., & Salim, F. D. (2020). Time-series change point detection with self-supervised contrastive predictive coding. arXiv preprint arXiv: 2011.14097. · arXiv
  5. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805. · arXiv
  6. Doersch, C., Gupta, A., & Efros, A. A. (2015). Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision (pp. 1422-1430).
  7. Esteban, C., Hyland, S. L., & Rätsch, G. (2017). Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.02633. · arXiv
  8. Ferrari, A., Micucci, D., Mobilio, M., & Napoletano, P. (2020). On the personalization of classification models for human activity recognition. IEEE Access, 8, 32066-32079. http://dx.doi.org/10.1109/ACCESS.2020.2973425. · DOI
  9. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., et al. (2020). Shortcut learning in deep neural networks. arXiv preprint arXiv:2004. 07780.
  10. Gidaris, S., Singh, P., & Komodakis, N. (2018). Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728. · arXiv
  11. Gjoreski, M., Kalabakov, S., Luštrek, M., Gams, M., & Gjoreski, H. (2019). Cross- dataset deep transfer learning for activity recognition. In Adjunct proceedings of the 2019 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2019 ACM international symposium on wearable computers (pp. 714-718).
  12. Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., et al. (2000). PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 101(23), e215-e220.
  13. Güneş, S., Polat, K., & Yosunkaya, Ş. (2010). Efficient sleep stage recognition system based on EEG signal using k-means clustering based feature weighting. Expert Systems with Applications, 37(12), 7922-7928.
  14. Hammerla, N. Y., Halloran, S., & Plötz, T. (2016). Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint arXiv: 1604.08880. · arXiv
  15. Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., et al. (2019). Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1), 65.
  16. Healey, J. A., & Picard, R. W. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156-166.
  17. Himberg, J., Korpiaho, K., Mannila, H., Tikanmaki, J., & Toivonen, H. T. (2001). Time series segmentation for context recognition in mobile devices. In Proceedings 2001 IEEE international conference on data mining (pp. 203-210). IEEE.
  18. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. · arXiv
  19. Hyvarinen, A., & Morioka, H. (2016). Unsupervised feature extraction by time- contrastive learning and nonlinear ICA. In Advances in neural information processing systems (pp. 3765-3773).
  20. Kemp, B., Zwinderman, A. H., Tuk, B., Kamphuisen, H. A., & Oberye, J. J. (2000). Anal- ysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the EEG. IEEE Transactions on Biomedical Engineering, 47(9), 1185-1194.
  21. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. · arXiv
  22. Korbar, B., Tran, D., & Torresani, L. (2018). Cooperative learning of audio and video models from self-supervised synchronization. In Advances in neural information processing systems (pp. 7763-7774).
  23. Lajnef, T., Chaibi, S., Ruby, P., Aguera, P.-E., Eichenlaub, J.-B., Samet, M., et al. (2015). Learning machines and sleeping brains: automatic sleep stage classification using decision-tree multi-class support vector machines. Journal of Neuroscience Methods, 250, 94-105.
  24. Larsson, G., Maire, M., & Shakhnarovich, G. (2016). Learning representations for automatic colorization. In European conference on computer vision (pp. 577-593). Springer.
  25. Li, Y., Shi, D., Ding, B., & Liu, D. (2014). Unsupervised feature learning for human activity recognition using smartphone sensors. In Mining intelligence and knowledge exploration (pp. 99-107). Springer.
  26. Luo, Y., Cai, X., Zhang, Y., Xu, J., et al. (2018). Multivariate time series imputation with generative adversarial networks. In Advances in neural information processing systems (pp. 1596-1607).
  27. Luo, Y., & Mesgarani, N. (2019). Conv-TasNet: Surpassing ideal time-frequency magni- tude masking for speech separation. ACM Transactions on Audio Speech and Language Processing, 27(8), 1256-1266.
  28. Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2018). Protecting sensory data against sensitive inferences. In Proceedings of the 1st workshop on privacy by design in distributed systems (p. 2). ACM.
  29. Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013). Learning deep physiological models of affect. IEEE Computational Intelligence Magazine, 8(2), 20-33.
  30. Mazankiewicz, A., Böhm, K., & Bergés, M. (2020). Incremental real-time personalization in human activity recognition using domain adaptive batch normalization. Proceed- ings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(4), 1-20.
  31. McMahan, B., Moore, E., Ramage, D., Hampson, S., & y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (pp. 1273-1282). PMLR.
  32. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (pp. 3111-3119).
  33. Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. · arXiv
  34. Nathan Mundhenk, T., Ho, D., & Chen, B. Y. (2018). Improvements to context based self-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 9339-9348).
  35. Noroozi, M., & Favaro, P. (2016). Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision (pp. 69-84). Springer.
  36. Oord, A. v. d., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. · arXiv
  37. Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence, (10), 1175-1191.
  38. Plötz, T., Hammerla, N. Y., & Olivier, P. L. (2011). Feature learning for activity recognition in ubiquitous computing. In Twenty-second international joint conference on artificial intelligence.
  39. Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., et al. (2018). Multimodal deep learning for activity and context recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4), 1-27.
  40. Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91-99).
  41. Saeed, A., Ozcelebi, T., & Lukkien, J. (2019). Multi-task self-supervised learning for human activity detection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2), 61.
  42. Saeed, A., Ozcelebi, T., Lukkien, J., van Erp, J., & Trajanovski, S. (2018). Model adaptation and personalization for physiological stress detection. In 2018 IEEE 5th international conference on data science and advanced analytics (DSAA) (pp. 209-216).
  43. Saeed, A., & Trajanovski, S. (2017). Personalized driver stress detection with multi-task neural networks using physiological signals. arXiv preprint arXiv:1711.06116. · arXiv
  44. Sarkar, P., & Etemad, A. (2020). Self-supervised ecg representation learning for emotion recognition. IEEE Transactions on Affective Computing.
  45. Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Schaal, S., et al. (0000). Time-contrastive networks: Self-supervised learning from video. In Proceedings of international conference in robotics and automation (ICRA).
  46. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al. (2017). Mastering the game of go without human knowledge. Nature, 550(7676), 354.
  47. Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., et al. (2015). Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems (pp. 127-140). ACM.
  48. Supratak, A., Dong, H., Wu, C., & Guo, Y. (2017). DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(11), 1998-2008.
  49. Tagliasacchi, M., Gfeller, B., de Chaumont Quitry, F., & Roblek, D. (2019). Self- supervised audio representation learning for mobile devices. arXiv preprint arXiv: 1905.11796. · arXiv
  50. Thanh-Tung, H., Tran, T., & Venkatesh, S. (2019). Improving generalization and stability of generative adversarial networks. arXiv preprint arXiv:1902.03984. · arXiv
  51. Tian, Y., Krishnan, D., & Isola, P. (2019). Contrastive multiview coding. arXiv preprint arXiv:1906.05849. · arXiv
  52. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., & Bottou, L. (2010). Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11(12).
  53. Yao, S., Zhao, Y., Shao, H., Zhang, C., Zhang, A., Hu, S., et al. (2018). Sensegan: Enabling deep learning for internet of things with a semi-supervised framework. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(3), 1-21.
  54. Yousefi, S., Narui, H., Dayal, S., Ermon, S., & Valaee, S. (2017). A survey on behavior recognition using wifi channel state information. IEEE Communications Magazine, 55(10), 98-104.
  55. Zeghidour, N., & Grangier, D. (2020). Wavesplit: End-to-end speech separation by speaker clustering. arXiv preprint arXiv:2002.08933. · arXiv
  56. Zhai, X., Oliver, A., Kolesnikov, A., & Beyer, L. (2019). S4L: Self-supervised semi-supervised learning. arXiv preprint arXiv:1905.03670. · arXiv
  57. Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017). Mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. · arXiv
  58. Zhang, S., Gong, Y., Huang, J.-B., Lim, J., Wang, J., Ahuja, N., et al. (2016). Tracking persons-of-interest via adaptive discriminative features. In European conference on computer vision (pp. 415-433). Springer.
  59. Zhang, R., Isola, P., & Efros, A. A. (2016). Colorful image colorization. In European conference on computer vision (pp. 649-666). Springer.

Cited by

Sorted by citations

Metadata

Authors
Saeed, A., Ungureanu, V., & Gfeller, B.
Affiliations
  • Eindhoven University of Technology (NL)
  • Google (Switzerland) (CH)
Corresponding author
Saeed, Aaqib
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Volume
6
Pages
100152-100152
ISSN
2666-8270 (ISSN-L)
DOI
10.48550/arxiv.2009.13233
arXiv
2009.13233
OpenAlex
W3088808810
Cited by
34
References
0
Citations by year
2026: 1 · 2025: 6 · 2024: 7 · 2023: 13 · 2022: 4 · 2021: 2 · 2020: 1
FWCI
2.30
Type
journal article
Language
en
Open access
Licence
Locations
  • Machine Learning with Applications (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
self-supervised learning, multi-sensor data, Internet of Things, representation learning, unlabeled data, few-shot learning, semi-supervised learning, transfer learning, fine-tuning, low-data regimes

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review