CoChat
Sign Up
PreprintCited by 27Domain Adaptation and Few-Shot LearningTime Series Analysis and ForecastingMultimodal Machine Learning Applications

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Open access
Deldari, Shohreh, Xue, Hao, Saeed, Aaqib, He, Jiayuan, Smith, Daniel V., Salim, Flora D.·2022·arXiv
DOI 10.48550/arxiv.2206.02353arXiv 2206.02353
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, Self-Supervised Representation Learning (SSRL) has attracted much attention in the field of computer vision, speech, natural language processing (NLP), and recently, with other types of modalities, including time series from sensors. The popularity of self-supervised learning is driven by the fact that traditional models typically require a huge amount of well-annotated data for training. Acquiring annotated data can be a difficult and costly process. Self-supervised methods have been introduced to improve the efficiency of training data through discriminative pre-training of models using supervisory signals that have been freely obtained from the raw data. Unlike existing reviews of SSRL that have pre-dominately focused upon methods in the fields of CV or NLP for a single modality, we aim to provide the first comprehensive review of multimodal self-supervised learning methods for temporal data. To this end, we 1) provide a comprehensive categorization of existing SSRL methods, 2) introduce a generic pipeline by defining the key components of a SSRL framework, 3) compare existing models in terms of their objective function, network architecture and potential applications, and 4) review existing multimodal techniques in each category and various modalities. Finally, we present existing weaknesses and future opportunities. We believe our work develops a perspective on the requirements of SSRL in domains that utilise multimodal and/or temporal data

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.02353

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The surveys by Eldele et al. [19] and Deldari et al. [20] only discuss a small part of self-supervised contrastive learning (SSCL).

    “However, these two reviews only discuss a small part of self-supervised contrastive learning (SSCL), which requires a more comprehensive literature review.”

    From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [20]

    Section 1.3 (p. 4) defines Deldari et al.’s scope as self-supervised representation learning on multimodal and temporal data, and the Conclusion (p. 30) says contrastive methods are one of four categories reviewed. This supports the claim that the survey does not cover all SSCL, but not the stronger characterization that it discusses only a small part; Section 4.2 (p. 14) does discuss multiple contrastive methods. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • Self-supervised learning provides a viable solution by pre-training a feature extractor on large unlabeled data and then optimizing a small classifier with limited labeled data.

    “Self-supervised learning (SSL) provides a viable solution by first pre-training a feature extractor on a large amount of unlabeled data and then optimizing a small classifier with limited labeled data [13].”

    From Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity Recognition (2023) · cites this paper as [13]

    Section 3.1 (pp. 8–9) describes learning an encoder from unlabeled data and transferring it to downstream tasks with limited labeled data, including classification. It does not specify optimizing a small classifier as the second step; the encoder may instead be reused directly or fine-tuned. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The taxonomies proposed by Eldele et al. [19], Deldari et al. [20] and Liu et al. [33] are somewhat similar to our proposed taxonomy, i.e., three taxonomies involve time series self-supervised contrastive learning methods.

    “The taxonomies proposed by Eldele et al. [19], Deldari et al. [20] and Liu et al. [33] are somewhat similar to our proposed taxonomy, i.e., three taxonomies involve time series self-supervised contrastive learning methods.”

    From Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects (2024) · cites this paper as [20]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 149

  1. [1][1] Futoon M Abushaqra, Hao Xue, Yongli Ren, and Flora D Salim. 2021. PIETS: Parallelised Irregularity Encoders for Forecasting with Heterogeneous Time-Series. In 2021 IEEE International Conference on Data Mining (ICDM) . IEEE, 976-981. -
  2. [2][2] Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. arXiv preprint arXiv:2104.11178 (2021). - · arXiv
  3. [3][3] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. 2020. Self-Supervised MultiModal Versatile Networks. NeurIPS 2, 6 (2020), 7. -
  4. [4][4] Yasin Almalioglu, Mehmet Turan, Muhamad Risqi U. Saputra, Pedro P.B. de Gusmão, Andrew Markham, and Niki Trigoni. 2022. SelfVIO: Self-supervised deep monocular Visual-Inertial Odometry and depth estimation. Neural Networks 150 (2022), 119-136. -
  5. [5][5] HumamAlwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. 2020. Self-Supervised Learning by Cross-Modal Audio-Video Clustering. In Advances in Neural Information Processing Systems (NeurIPS) . -
  6. [6][6] Elad Amrani, Rami Ben-Ari, Tal Hakim, and Alex Bronstein. 2019. Learning to detect and retrieve objects from unlabeled videos. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . IEEE, 3713-3717. -
  7. [7][7] Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. 2021. Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning. Proceedings of the AAAI Conference on Artificial Intelligence 35, 8 (May 2021). -
  8. [8][8] Relja Arandjelovic and Andrew Zisserman. 2017. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision . -
  9. [9][9] Relja Arandjelovic and Andrew Zisserman. 2018. Objects that sound. In Proceedings of the European conference on computer vision (ECCV) . 435-451. -
  10. [10][10] Yuki Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. 2020. Labelling unlabelled videos from scratch with multi-modal self-supervision. Advances in Neural Information Processing Systems 33 (2020), 4660-4671. -
  11. [11][11] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2016. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems 29 (2016), 892-900. -
  12. [12][12] Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019. Effectiveness of self-supervised pre-training for speech recognition. arXiv preprint arXiv:1911.03912 (2019). - · arXiv
  13. [13][13] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555 (2022). - · arXiv
  14. [14][14] Alexei Baevski, Steffen Schneider, and Michael Auli. 2020. vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . -
  15. [15][15] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems (NeurIPS) . -
  16. [16][16] Oresti Banos, Alberto Calatroni, Miguel Damas, Hector Pomares, Daniel Roggen, Ignacio Rojas, and Claudia Villalonga. 2021. Opportunistic activity recognition in IoT sensor ecosystems via multimodal transfer learning. Neural Processing Letters 53, 5 (2021), 3169-3197. -
  17. [17][17] Hubert Banville, Omar Chehab, Aapo Hyvärinen, Denis-Alexander Engemann, and Alexandre Gramfort. 2021. Uncovering the structure of clinical EEG signals with self-supervised learning. Journal of Neural Engineering 18, 4 (2021), 046020. -
  18. [18][18] Adrien Bardes, Jean Ponce, and Yann LeCun. 2022. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. In International Conference on Learning Representations . https://openreview.net/forum?id=xm6YD62D1Ub -
  19. [19][19] George Barnum, Sabera Talukder, and Yisong Yue. 2020. On the Benefits of Early Fusion in Multimodal Representation Learning. arXiv preprint arXiv:2011.07191 (2020). - · arXiv
  20. [20][20] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798-1828. -
  21. [21][21] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. 2007. Greedy layer-wise training of deep networks. In Advances in neural information processing systems . 153-160. -
  22. [22][22] Mathieu Blondel, André FT Martins, and Vlad Niculae. 2020. Learning with Fenchel-Young losses. J. Mach. Learn. Res. 21, 35 (2020), 1-69. -
  23. [23][23] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV) . 132-149. -
  24. [24][24] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS) . -
  25. [25][25] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging Properties in Self-Supervised Vision Transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV . IEEE, 9630-9640. -
  26. [26][26] Andrew N Carr, Quentin Berthet, Mathieu Blondel, Olivier Teboul, and Neil Zeghidour. 2021. Self-Supervised Learning of Audio Representations From Permutations With Differentiable Ranking. IEEE Signal Processing Letters 28 (2021), 708-712. -
  27. [27][27] Keumgang Cha, Junghoon Seo, and Yeji Choi. 2021. Contrastive Multiview Coding With Electro-Optics for SAR Semantic Segmentation. IEEE Geoscience and Remote Sensing Letters 19 (2021), 1-5. -
  28. [28][28] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. 2010. Large Scale Online Learning of Image Similarity Through Ranking. Journal of Machine Learning Research 11, 3 (2010). -
  29. [29][29] Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. 2021. Multimodal clustering networks for self-supervised learning from unlabeled videos. In ICCV . -
  30. [30][30] Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, and Niki Trigoni. 2019. Selective sensor fusion for neural visual-inertial odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10542-10551. -
  31. [31][31] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning . PMLR, 1597-1607. -
  32. [32][32] Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15750-15758. -
  33. [33][33] Joseph Y Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi. 2020. Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.04871 (2020). - · arXiv
  34. [34][34] Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-Wen Li, and Hung-yi Lee. 2021. Audio albert: A lite bert for self-supervised learning of audio representation. In 2021 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 344-350. -
  35. [35][35] Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , Vol. 1. IEEE, 539-546. -
  36. [37][37] Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. 2020. Debiased Contrastive Learning. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc. -
  37. [38][38] Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems 26 (2013). -
  38. [39][39] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. 2019. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems 32 (2019). -
  39. [40][40] Virginia R de Sa. 1994. Learning classification with unlabeled data. Advances in neural information processing systems (1994), 112-112. -
  40. [42][42] Karan Desai and Justin Johnson. 2021. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11162-11173. -
  41. [43][43] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of the Conf. of the North American Chapter of the Assoc. for Computational Linguistics: Human Lang. Tech., NAACL-HLT 2019 . -
  42. [44][44] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. 2016. Adversarial feature learning. arXiv preprint arXiv:1605.09782 (2016). - · arXiv
  43. [45][45] Jeff Donahue and Karen Simonyan. [n.d.]. Large scale adversarial representation learning. NeurIPS 32 ([n. d.]). -
  44. [46][46] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. 2014. Discriminative unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems 27 (2014). -
  45. [47][47] Yueqi Duan, Lei Chen, Jiwen Lu, and Jie Zhou. 2019. Deep embedding learning with discriminative sampling policy. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 4964-4973. -
  46. [48][48] Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. 2021. Time-Series Representation Learning via Temporal and Contextual Contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 . -
  47. [49][49] Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. 2021. Self-Supervised Representation Learning: Introduction, Advances and Challenges. arXiv preprint arXiv:2110.09327 (2021). · arXiv
  48. [50][50] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. 2021. Whitening for Self-Supervised Representation Learning. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139) . PMLR, 3015-3024. -
  49. [51][51] Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, and Xavier Serra. 2020. Coala: Co-aligned autoencoders for learning semantically enriched audio representations. arXiv preprint arXiv:2006.08386 (2020). - · arXiv
  50. [52][52] Xavier Favory, Konstantinos Drossos, Tuomas Virtanen, and Xavier Serra. 2021. Learning contextual tag embeddings for cross-modal alignment of audio and tags. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 596-600. -
  51. [53][53] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. 2017. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3636-3645.
  52. [55][55] Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems . 4650-4661. -
  53. [56][56] Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. 2019. Analyzing and improving representations with the soft nearest neighbor loss. In 3. International conference on machine learning . PMLR, 2012-2020. -
  54. [57][57] Nan Gao, Hao Xue, Wei Shao, Sichen Zhao, Kyle Kai Qin, Arian Prabowo, Mohammad Saiedur Rahaman, and Flora D Salim. 2022. Generative adversarial networks for spatio-temporal data: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 13, 2 (2022), 1-25. -
  55. [58][58] Ruohan Gao, Rogerio Feris, and Kristen Grauman. 2018. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV) . 35-53. -
  56. [59][59] John M. Giorgi, Osvald Nitski, Bo Wang, and Gary D. Bader. 2021. DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. In Proc. of the Annual Meeting of the Assoc. for Computational Linguistics and the International Joint Conf. on Natural Lang. Processing, ACL/IJCNLP . -
  57. [60][60] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems (NeurIPS) . -
  58. [61][61] Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019. Deep Multimodal Representation Learning: A Survey. IEEE Access 7 (2019). -
  59. [62][62] Michael Gutman and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of 13th International Conference on Artificial Intelligence and Statistics . 297-304. -
  60. [63][63] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2021. AudioCLIP: Extending CLIP to Image, Text and Audio. arXiv preprint arXiv:2106.13043 (2021). - · arXiv
  61. [64][64] Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , Vol. 2. IEEE, 1735-1742. -
  62. [65][65] Tengda Han, Weidi Xie, and Andrew Zisserman. 2020. Self-supervised co-training for video representation learning. Advances in Neural Information Processing Systems 33 (2020), 5679-5690. -
  63. [66][66] Harish Haresamudram, Apoorva Beedu, Varun Agrawal, Patrick L Grady, Irfan Essa, Judy Hoffman, and Thomas Plötz. 2020. Masked reconstruction based self-supervision for human activity recognition. In Proceedings of the 2020 International Symposium on Wearable Computers . 45-49. -
  64. [67][67] Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2021. Contrastive predictive coding for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1-26. -
  65. [68][68] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9729-9738.
  66. [69][69] Olivier Henaff. 2020. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning . PMLR. -
  67. [70][70] Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A fast learning algorithm for deep belief nets. Neural computation 18, 7 (2006). -
  68. [71][71] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations . -
  69. [72][72] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv preprint arXiv:2106.07447 (2021). - · arXiv
  70. [73][73] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. 2021. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 9598-9608. -
  71. [74][74] Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. 2022. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 2785-2795. -
  72. [77][77] Longlong Jing, Yucheng Chen, Ling Zhang, Mingyi He, and Yingli Tian. 2020. Self-supervised modal and view invariant feature learning. arXiv preprint arXiv:2005.14169 (2020). - · arXiv
  73. [78][78] Longlong Jing and Yingli Tian. 2020. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence (2020). -
  74. [79][79] Longlong Jing, Ling Zhang, and Yingli Tian. 2021. Self-supervised feature learning by cross-modality and cross-view correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1581-1591. -
  75. [80][80] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems 33 (2020), 21798-21809. -
  76. [81][81] Sameer Khurana, Antoine Laurent, and James Glass. 2020. Cstnet: Contrastive speech translation network for self-supervised speech representation learning. arXiv preprint arXiv:2006.02814 (2020). - · arXiv
  77. [82][82] Seonhoon Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak. 2021. Self-supervised Pre-training and Contrastive Representation Learning for Multiple-choice Video QA. Proceedings of the AAAI Conference on Artificial Intelligence 35, 14 (May 2021), 13171-13179. -
  78. [83][83] Youngji Kim, Sungho Yoon, Sujung Kim, and Ayoung Kim. 2021. Unsupervised balanced covariance learning for visual-inertial sensor fusion. IEEE Robotics and Automation Letters 6, 2 (2021), 819-826.
  79. [84][84] Dani Kiyasseh, Tingting Zhu, and David A Clifton. 2020. Clocs: Contrastive learning of cardiac signals. arXiv preprint arXiv:2005.13249 (2020). - · arXiv
  80. [85][85] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. 2015. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop , Vol. 2. Lille, 0. -
  81. [86][86] Bruno Korbar, Du Tran, and Lorenzo Torresani. 2018. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems 31 (2018). -
  82. [87][87] Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Junaid Qadir, and Björn W Schuller. 2020. Deep representation learning in speech processing: Challenges, recent advances, and future trends. arXiv preprint arXiv:2001.00378 (2020). - · arXiv
  83. [88][88] Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. 2020. Contrastive representation learning: A framework and review. IEEE Access (2020). -
  84. [90][90] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. 2021. Prototypical Contrastive Learning of Unsupervised Representations. In International Conference on Learning Representations . https://openreview.net/forum?id=KmykpuSrjcq -
  85. [91][91] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. [n.d.]. Contrastive clustering. In 2021 AAAI . -
  86. [92][92] Andy T Liu, Shang-Wen Li, and Hung-yi Lee. 2021. Tera: Self-supervised learning of transformer encoder representation for speech. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 2351-2366. -
  87. [94][94] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32 (2019). -
  88. [95][95] Huanru Henry Mao. 2020. A survey on self-supervised pre-training for sequential transfer learning in neural networks. arXiv preprint arXiv:2007.00800 (2020). - · arXiv
  89. [96][96] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-End Learning of Visual Representations From Uncurated Instructional Videos. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . -
  90. [97][97] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 2630-2640. -
  91. [98][98] Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. 2020. Automatic shortcut removal for self-supervised representation learning. In International Conference on Machine Learning . PMLR, 6927-6937. -
  92. [99][99] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. 2016. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision . Springer, 527-544. -
  93. [100][100] Jovana Mitrovic, Brian McWilliams, and Melanie Rey. 2020. Less can be more in contrastive learning. In Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops (Proceedings of Machine Learning Research, Vol. 137) . PMLR, 70-75. -
  94. [101][101] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. 2021. Audio-visual instance discrimination with cross-modal agreement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 12475-12486. -
  95. [102][102] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. 2018. Learnable pins: Cross-modal embeddings for person identity. In Proceedings of the European Conference on Computer Vision (ECCV) . 71-88. -
  96. [103][103] Tung Nguyen, Rui Shu, Tuan Pham, Hung Bui, and Stefano Ermon. 2021. Temporal Predictive Coding For Model-Based Planning In Latent Space. In International Conference on Machine Learning . -
  97. [104][104] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition . 4004-4012. -
  98. [105][105] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint (2018). -
  99. [106][106] Felix Ott, David Rügamer, Lucas Heublein, Bernd Bischl, and Christopher Mutschler. 2022. Cross-Modal Common Representation Learning with Triplet Loss Functions. arXiv preprint arXiv:2202.07901 (2022). - · arXiv
  100. [107][107] Andrew Owens and Alexei A Efros. 2018. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV) . -
  101. [108][108] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. 2016. Ambient sound provides supervision for visual learning. In European conference on computer vision . Springer, 801-816. -
  102. [109][109] Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. [n.d.]. Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association 2019 . -
  103. [110][110] AJ Piergiovanni, Anelia Angelova, and Michael S Ryoo. 2020. Evolving losses for unsupervised video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 133-142. -
  104. [111][111] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge Belongie, and Yin Cui. 2021. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6964-6974. -
  105. [112][112] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning . PMLR. -
  106. [113][113] Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. 2019. Continual Unsupervised Representation Learning. Advances in Neural Information Processing Systems 32 (2019), 7647-7657. -
  107. [114][114] Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. 2020. BYOL works even without batch statistics. arXiv preprint arXiv:2010.10241 (2020). - · arXiv
  108. [115][115] Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive Learning with Hard Negative Samples. In International Conference on Learning Representations . https://openreview.net/forum?id=CR1XOQ0UTh- -
  109. [116][116] Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. 2019. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems 32 (2019). -
  110. [117][117] Aaqib Saeed, David Grangier, and Neil Zeghidour. 2021. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 3875-3879. -
  111. [118][118] Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. 2019. Multi-task Self-Supervised Learning for Human Activity Detection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 2 (2019), 61. -
  112. [123][123] Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 815-823. -
  113. [124][124] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. 2021. Data-Efficient Reinforcement Learning with Self-Predictive Representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event . -
  114. [125][125] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. 2018. Time-Contrastive Networks: Self-Supervised Learning from Video. Proceedings of International Conference in Robotics and Automation (ICRA) (2018). -
  115. [127][127] Yuge Shi, Brooks Paige, Philip Torr, and Siddharth N. 2021. Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models. In International Conference on Learning Representations . https://openreview.net/forum?id=vhKe9UFbrJo -
  116. [128][128] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. 2015. Discriminative learning of deep convolutional feature point descriptors. In Proceedings of the IEEE International Conference on Computer Vision . 118-126. Manuscript submitted to ACM -
  117. [129][129] Linda Smith and Michael Gasser. 2005. The development of embodied cognition: Six lessons from babies. Artificial life 11, 1-2 (2005), 13-29. -
  118. [130][130] Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in neural information processing systems . -
  119. [131][131] Jonathan C Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, and David A Ross. 2020. Learning video representations from textual web supervision. arXiv preprint arXiv:2007.14937 (2020). - · arXiv
  120. [132][132] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7464-7473. -
  121. [133][133] Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah Goodman. 2021. DABS: a Domain-Agnostic Benchmark for SelfSupervised Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) . -
  122. [134][134] Li Tao, Xueting Wang, and Toshihiko Yamasaki. 2020. Self-supervised video representation learning using inter-intra contrastive framework. In Proceedings of the 28th ACM International Conference on Multimedia . 2193-2201. -
  123. [135][135] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. 2021. Divide and contrast: Self-supervised learning from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 10063-10074. -
  124. [136][136] Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020. Contrastive Multiview Coding. In Computer Vision - ECCV 2020 , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 776-794. -
  125. [137][137] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020. What makes for good views for contrastive learning? NeurIPS (2020). https://ai.googleblog.com/2020/08/understanding-view-selection-for.html -
  126. [138][138] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. 2020. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578 (2020). - · arXiv
  127. [139][139] Martine Toering, Ioannis Gatopoulos, Maarten Stol, and Vincent Tao Hu. 2022. Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 108-118. -
  128. [140][140] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. 2021. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. In International Conference on Learning Representations . https://openreview.net/forum?id=8qDwejCuCN
  129. [141][141] Giulio Tononi, Olaf Sporns, and Gerald M Edelman. 1999. Measures of degeneracy and redundancy in biological networks. National Academy of Sciences 96, 6 (1999), 3257-3262. -
  130. [142][142] Feng Wang and Huaping Liu. 2021. Understanding the Behaviour of Contrastive Loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2495-2504. -
  131. [143][143] Jun Wang, Max W Y Lam, Dan Su, and Dong Yu. 2021. Contrastive Separative Coding for Self-Supervised Representation Learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 3865-3869.
  132. [144][144] Luyu Wang, Pauline Luc, Adria Recasens, Jean-Baptiste Alayrac, and Aaron van den Oord. 2021. Multimodal Self-Supervised Learning of General Audio Representations. arXiv preprint arXiv:2104.12807 (2021). - · arXiv
  133. [145][145] Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning . PMLR, 9929-9939. -
  134. [146][146] Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil M Robertson. 2019. Ranked list loss for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . -
  135. [147][147] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. 2018. Learning and using the arrow of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 8052-8060. -
  136. [148][148] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. 2021. Time Series Data Augmentation for Deep Learning: A Survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , Zhi-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4653-4660. Survey Track. -
  137. [149][149] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. 2017. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision . -
  138. [150][150] Qinfeng Xiao, Jing Wang, Jianan Ye, Hongjun Zhang, Yuyan Bu, Yiqiong Zhang, and Hao Wu. 2021. Self-Supervised Learning for Sleep Stage Classification with Predictive and Discriminative Contrastive Coding. In International Conf. on Acoustics, Speech and Signal Processing (ICASSP) . -
  139. [151][151] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2017. Rethinking spatiotemporal feature learning for video understanding. arXiv preprint arXiv:1712.04851 1, 2 (2017), 5. - · arXiv
  140. [152][152] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 1686-1697. -
  141. [153][153] Xitong Yang, Palghat Ramesh, Radha Chitta, Sriganesh Madhvanath, Edgar A Bernal, and Jiebo Luo. 2017. Deep multimodal representation learning from temporal data. In Proceedings of the IEEE conference on computer vision and pattern recognition . 5447-5455. -
  142. [154][154] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. 2021. A Survey on Deep Semi-supervised Learning. arXiv preprint arXiv:2103.00550 (2021). - · arXiv
  143. [155][155] Asano YM., Rupprecht C., and Vedaldi A. 2020. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations . https://openreview.net/forum?id=Hyx-jyBFPr -
  144. [156][156] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. 2021. Multimodal Contrastive Training for Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6995-7004. -
  145. [157][157] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2021. TS2Vec: Towards Universal Representation of Time Series. arXiv preprint arXiv:2106.10466 (2021). Proceedings of the - · arXiv
  146. [158][158] Salah Zaiem, Titouan Parcollet, and Slim Essid. 2021. Pretext Tasks selection for multitask self-supervised speech representation learning. arXiv preprint arXiv:2107.00594 (2021). - · arXiv
  147. [159][159] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, ICML , Marina Meila and Tong Zhang (Eds.), Vol. 139. PMLR. -
  148. [160][160] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A Transformer-Based Framework for Multivariate Time Series Representation Learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . -
  149. [161][161] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. 2020. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6688-6697. -

Cited by

Sorted by citations

Related

Authors
Deldari, S., Xue, H., Saeed, A., He, J., Smith, D. V., & Salim, F. D.
Affiliations
  • RMIT University (AU)
  • Philips (Netherlands) (NL)
  • UNSW Sydney (AU)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.02353
arXiv
2206.02353
OpenAlex
W4281910964
Cited by
26
References
0
Citations by year
2026: 1 · 2025: 6 · 2024: 9 · 2023: 9 · 2022: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Commonwealth Scientific and Industrial Research Organisation (DATA61)
  • RMIT University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
self-supervised representation learning, multimodal learning, temporal data, time series, sensor data, pre-training, network architecture, cross-modal learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review