Masked reconstruction based self-supervision for human activity recognition
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
The ubiquitous availability of wearable sensing devices has rendered large scale collection of movement data a straightforward endeavor. Yet, annotation of these data remains a challenge and as such, publicly available datasets for human activity recognition (HAR) are typically limited in size as well as in variability, which constrains HAR model training and effectiveness. We introduce masked reconstruction as a viable self-supervised pre-training objective for human activity recognition and explore its effectiveness in comparison to state-of-the-art unsupervised learning techniques. In scenarios with small labeled datasets, the pre-training results in improvements over end-to-end learning on two of the four benchmark datasets. This is promising because the pre-training objective can be integrated "as is" into state-of-the-art recognition pipelines to effectively facilitate improved model robustness, and thus, ultimately, leading to better recognition performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our work differs from [8] in that [8] masks only individual samples throughout the accelerometer timeseries, while ours masks entire segments consisting of consecutive samples.”
From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [8]
In the citing paper
“We show that our method outperforms existing self-supervised methods in HAR for UCI HAR, MotionSense, and HAPT datasets.”
From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [8]
In the citing paper
“| Haresamudram et al. [8] (SS) | - | 0.880 | - |”
From Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion (2021) · cites this paper as [8]
In the citing paper
“On RealWorld and PAMAP datasets, CPC [26] and Masked Reconstruction [25] achieve the best performance, respectively.”
From Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity Recognition (2023) · cites this paper as [25]
In the citing paper
“Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC, and SimCLR, to name but a few.”
From Assessing the State of Self-Supervised Human Activity Recognition Using Wearables (2022) · cites this paper as [26]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“For pre-training, we used the Noam optimizer [34] with a warmup of 4 , 000 steps and trained for 150 epochs.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“In NLP, representations are learned in form of word and sentence vectors that have been computed using contextual feature embeddings such as Word2vec [19], GloVe [26] and more recently, BERT [5].”
Method(2014) · Cited by 33,419
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 1×
“We implemented the proposed method and related baselines using PyTorch [24].”
MethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“In NLP, representations are learned in form of word and sentence vectors that have been computed using contextual feature embeddings such as Word2vec [19], GloVe [26] and more recently, BERT [5].”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
(2010) · International Conference on Machine Learning · Cited by 13,995
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“This is in contrast to denoising autoencoders, where the entire perturbed frame is reconstructed [35].”
ResultContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · cited 1×
“For example, in computer vision, inpainting [25], predicting image rotations [10], colorization [15], temporal order verification [20], or odd sequence detection [7] are used as pretext tasks.”
MethodDeep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“…unsupervised learning techniques, i.e., convolutional autoencoders (CAE) [12] and multi-task self-supervision [30], and, for reference only given that the main focus of this paper is on unsupervised approaches, to supervised learning pipelines (DeepConvLSTM [23] and Trans-former classifier).”
BackgroundUnsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · cited 2×
“For example, in computer vision, inpainting [25], predicting image rotations [10], colorization [15], temporal order verification [20], or odd sequence detection [7] are used as pretext tasks.”
Method(2013) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 1,959
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668 · cited 5×
“Our work is focused on deriving effective data representations to be used as part of the standard activity recognition chain (ARC) [3].”
BackgroundMethodInfluentialwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition
Dong, L., Xu, S., & Xu, B. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462506 · Cited by 1,100 · cited 1×
“Transformer encoders have been utilized to classify activities in HAR [17], yet their main application is still elsewhere (e.g., in ASR [6, 21], and generic time-series forecasting [16, 32, 37]).”
MethodEnhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., & Yan, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00235 · Cited by 1,053 · cited 2×
“, in ASR [6, 21], and generic time-series forecasting [16, 32, 37]).”
BackgroundShuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · cited 1×
“For example, in computer vision, inpainting [25], predicting image rotations [10], colorization [15], temporal order verification [20], or odd sequence detection [7] are used as pretext tasks.”
MethodPreprocessing techniques for context recognition from accelerometer data
Figo, D., Diniz, P. C., Ferreira, D. R., & Cardoso, J. M. P. (2010) · Personal and Ubiquitous Computing · doi:10.1007/s00779-010-0293-9 · Cited by 578 · cited 1×
“For the former, typically heuristics are employed to extract time-domain features, such as statistical moments, or spectral representations [8].”
MethodUSC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors
Zhang, M., & Sawchuk, A. A. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370438 · Cited by 554 · cited 8×
“Our experimentswere based on four benchmark datasets (accelerometer plus gyroscope): Mobiactv2 [4], Motionsense [18], USC-HAD [38], and UCI-HAR [1].”
MethodInfluentialAttend and Diagnose: Clinical Time Series Analysis Using Attention Models
Song, H., Rajan, D., Thiagarajan, J., & Spanias, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11635 · Cited by 468
Colorization as a Proxy Task for Visual Understanding
Larsson, G., Maire, M., & Shakhnarovich, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.96 · Cited by 458 · cited 1×
“For example, in computer vision, inpainting [25], predicting image rotations [10], colorization [15], temporal order verification [20], or odd sequence detection [7] are used as pretext tasks.”
MethodSelf-Supervised Video Representation Learning with Odd-One-Out Networks
Fernando, B., Bilen, H., Gavves, E., & Gould, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.607 · Cited by 421 · cited 1×
“For example, in computer vision, inpainting [25], predicting image rotations [10], colorization [15], temporal order verification [20], or odd sequence detection [7] are used as pretext tasks.”
MethodSelf-attention for raw optical Satellite Time Series Classification
Rußwurm, M., & Körner, M. (2020) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2020.06.006 · Cited by 355
Multi-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346 · cited 6×
“Results for [30] comparable to original publication, yet not identical due to original implementation not being released, and details being omitted in [30].”
BackgroundMethodInfluentialProtecting sensory data against sensitive inferences
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2018) · Spiral (Imperial College London) · Cited by 166
Protecting Sensory Data against Sensitive Inferences
Malekzadeh, M., Clegg, R. G., Cavallaro, A., & Haddadi, H. (2018) · Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems · doi:10.1145/3195258.3195260 · Cited by 154 · cited 1×
“Ourexperimentswerebasedonfourbenchmarkdatasets(accelerom-eterplusgyroscope):Mobiactv2[4],Motionsense[18],USC-HAD [38],andUCI-HAR[1].”
Hammerla, N. Y., Kirkham, R., Andras, P., & Ploetz, T. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494353 · Cited by 138 · cited 1×
“Other approaches include utilizing classic dimensionality reduction techniques such as PCA [27] and distribution-based approaches [11].”
MethodTransformers with convolutional context for ASR
Mohamed, A., Okhonko, D., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11660 · Cited by 137 · cited 5×
“For example,Wang et al.[36] target automated speech recognition (ASR).”
BackgroundInfluentialUsing unlabeled data in a sparse-coding framework for human activity recognition
Bhattacharya, S., Nurmi, P., Hammerla, N., & Plötz, T. (2014) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2014.05.006 · Cited by 121
Self-citationActivity Recognition in Pervasive Intelligent Environments
(2011) · Atlantis Ambient and Pervasive Intelligence · doi:10.2991/978-94-91216-05-3 · Cited by 119
Human Activity Recognition from Wearable Sensor Data Using Self-Attention
Saif, M., Tanjid Hasan Tonmoy, M., Kishor, K. B., Mahbubur Rahman, A. K. M., Ashraful Amin, M., Mohammad, S., Muhammad, A. H. K., & Amin, A. A. (2020) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia200236 · Cited by 93
Human Daily Activity and Fall Recognition Using a Smartphone’s Acceleration Sensor
Chatzaki, C., Pediaditis, M., Vavoulas, G., & Tsiknakis, M. (2017) · Communications in Computer and Information Science · doi:10.1007/978-3-319-62704-5_7 · Cited by 91 · cited 1×
“Our experimentswere based on four benchmark datasets (accelerometer plus gyroscope): Mobiactv2 [4], Motionsense [18], USC-HAD [38], and UCI-HAR [1].”
MethodOn the role of features in human activity recognition
Haresamudram, H., Anderson, D. V., & Plötz, T. (2019) · Proceedings of the 23rd International Symposium on Wearable Computers · doi:10.1145/3341163.3347727 · Cited by 90 · cited 5×
“For USC-HAD, we followed the protocol from [12] with data from participants 1 − 10 for training, from participants 11 and 12 for validation, and from participants 13 and 14 for testing.”
BackgroundMethodResultInfluentialSelf-citationActivity Recognition and Healthier Food Preparation
Plötz, T., Moynihan, P., Pham, C., & Olivier, P. (2011) · Atlantis Ambient and Pervasive Intelligence · doi:10.2991/978-94-91216-05-3_14 · Cited by 39 · cited 1×
“Previous work has shown that unlabeled sensor data can effectively be used for deriving feature representations [2, 12, 27] that then can be integrated into activity recognition chains [3, 28].”
BackgroundSelf-citationActivity2Vec: Learning ADL Embeddings from Sensor Data with a Sequence-to-Sequence ModelPreprint
Ghods, A., & Cook, D. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05597 · Cited by 18 · cited 1×
“[9] learned embeddings for activities of daily living (ADL) using a sequence-to-sequence model.”
Method
Research works printed but not linked: 9
- Chatzaki Charikleia, 2016, International Conference on Information and Communication Technologies for Ageing Well and e-Health. Springer, 100--118
- Devlin Jacob, 2018, Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 · arXiv
- Kingma Diederik P, 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 · arXiv
- 10.5555/3104322.3104425 · doi:10.5555/3104322.3104425
- Paszke Adam, Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc
- Thomas Plötz Nils Y Hammerla and Patrick L Olivier. 2011. Feature learning for activity recognition in ubiquitous computing. In Twenty-second international joint conference on artificial intelligence. Thomas Plötz Nils Y Hammerla and Patrick L Olivier. 2011. Feature learning for activity recognition in ubiquitous computing. In Twenty-second international joint conference on artificial intelligence.
- Song Huan, 2018, Thirty-second AAAI conference on artificial intelligence.
- Wang Weiran, 2020, ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6889--6893
- Neo Wu Bradley Green Xue Ben and Shawn O'Banion. 2020. Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case. arXiv preprint arXiv:2001.08317 (2020). Neo Wu Bradley Green Xue Ben and Shawn O'Banion. 2020. Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case. arXiv preprint arXiv:2001.08317 (2020). · arXiv
Cited by
Self-supervised learning for human activity recognition using 700,000 person-days of wearable data
Yuan, H., Chan, S., Creagh, A. P., Tong, C., Acquah, A., Clifton, D. A., & Doherty, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01062-3 · Cited by 166
“The test performance for the AoT stayed at the random chance level, and the test performance for permutation dropped roughly 10 percentage points without weighted sampling.”
Contrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3463506 · Cited by 150
“While [32] utilizes both accelerometer and gyroscope data, it delivers mixed results relative to DeepConvLSTM [59] when fine-tuning the pre-trained weights using limited labeled data.”
BackgroundMethodResultInfluentialSelf-citationHuman activity recognition using wearable sensors by heterogeneous convolutional neural networks
Han, C., Zhang, L., Tang, Y., Huang, W., Min, F., & He, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116764 · Cited by 130
Stochastic Recognition of Physical Activity and Healthcare Using Tri-Axial Inertial Wearable Sensors
Jalal, A., Batool, M., & Kim, K. (2020) · Applied Sciences · doi:10.3390/app10207122 · Cited by 123
“[22] introduced a masked reconstruction-based BERT model for human activity recognition activities.”
BackgroundColloSSL: Collaborative Self-Supervised Learning for Human Activity Recognition
Jain, Y., Tang, C. I., Min, C., Kawsar, F., & Mathur, A. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3517246 · Cited by 113
“[21] investigated the idea of masking sensor data at certain timesteps and training the feature encoder to reconstruct the missing data.”
Background
Metadata
- Authors
- Haresamudram, H., Beedu, A., Agrawal, V., Grady, P. L., Essa, I., Hoffman, J., & Plötz, T.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Proceedings of the 2020 International Symposium on Wearable Computers
- Publisher
- ACM
- Published
- 2020-09-04
- Pages
- 45-49
- OpenAlex
- W3083367988
- Cited by
- 146
- References
- 34
- Citations by year
- 2026: 11 · 2025: 33 · 2024: 34 · 2023: 29 · 2022: 22 · 2021: 14 · 2020: 3
- FWCI
- 5.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Wearable Computers (publishedVersion, public-domain)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- human activity recognition, masked reconstruction, self-supervised pre-training, small labeled datasets, model robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.