A Semisupervised Recurrent Convolutional Attention Model for Human Activity Recognition
The DOI matches its Crossref record; 31 of the 33 references it cites were found in a registry or an index.
Abstract
Recent years have witnessed the success of deep learning methods in human activity recognition (HAR). The longstanding shortage of labeled activity data inherently calls for a plethora of semisupervised learning methods, and one of the most challenging and common issues with semisupervised learning is the imbalanced distribution of labeled data over classes. Although the problem has long existed in broad real-world HAR applications, it is rarely explored in the literature. In this paper, we propose a semisupervised deep model for imbalanced activity recognition from multimodal wearable sensory data. We aim to address not only the challenges of multimodal sensor data (e.g., interperson variability and interclass similarity) but also the limited labeled data and class-imbalance issues simultaneously. In particular, we propose a pattern-balanced semisupervised framework to extract and preserve diverse latent patterns of activities. Furthermore, we exploit the independence of multi-modalities of sensory data and attentively identify salient regions that are indicative of human activities from inputs by our recurrent convolutional attention networks. Our experimental results demonstrate that the proposed model achieves a competitive performance compared to a multitude of state-of-the-art methods, both semisupervised and supervised ones, with 10% labeled training data. The results also show the robustness of our method over imbalanced, small training data sets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Contrastive learning methods generally involve instance discrimination, which means that the instances are regarded as negatives of each other, and thus their representations are pulled away from each other during the training process.”
From Negative Selection by Clustering for Contrastive Learning in Human Activity Recognition (2023) · cites this paper as [20]
Citations
Cites
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
(2013) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 1,959
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 5×
“After eliminating the impact of “others”, the overall classification performance on MHEALTH and PAMAP2 is boosted.”
BackgroundResultInfluentialPerformance evaluation of some clustering algorithms and validity indices
Maulik, U., & Bandyopadhyay, S. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1114856 · Cited by 1,396 · cited 1×
“where Ki denotes the number of clusters of Ci and it is adaptively decided by its covariance [23].”
BackgroundEnd-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096
(2015) · Cited by 989
mHealthDroid: A Novel Framework for Agile Development of Mobile Health Applications
Banos, O., Garcia, R., Holgado-Terriza, J. A., Damas, M., Pomares, H., Rojas, I., Saez, A., & Villalonga, C. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-13105-4_14 · Cited by 583
Ensembles of Deep LSTM Learners for Activity Recognition using Wearables
Guan, Y., & Plötz, T. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3090076 · Cited by 511 · cited 3×
“Although some approaches [10], [16] have been proposed to solve the imbalance issue, their performance deteriorates due to the small data size in semisupervised learning.”
BackgroundMethodWang, H., & Wang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.387 · Cited by 463
Learning to combine foveal glimpses with a third-order Boltzmann machine
Larochelle, H., & Hinton, G. E. (2010) · neural information processing systems · Cited by 436
Multimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281
Semi-Supervised Learning for Imbalanced Sentiment Classification
Li, S., Wang, Z., Zhou, G., & Lee, S. Y. M. (2012) · Cited by 220
Exploring semi-supervised and active learning for activity recognition
Stikic, M., Van Laerhoven, K., & Schiele, B. (2008) · 2008 12th IEEE International Symposium on Wearable Computers · doi:10.1109/iswc.2008.4911590 · Cited by 197
PD Disease State Assessment in Naturalistic Environments Using Deep Learning
Hammerla, N., Fisher, J., Andras, P., Rochester, L., Walker, R., & Ploetz, T. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9484 · Cited by 179 · cited 1×
“[14], [15] resort to deep generative models such as Restricted Boltzmann Machines (RBMs) and”
MethodTri-net for Semi-Supervised Deep Learning
Chen, D.-D., Wang, W., Gao, W., & Zhou, Z.-H. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/278 · Cited by 142
Semi-supervised learning with very few labeled training examples
Zhou, Z., Zhan, D., & Yang, Q. (2007) · Cited by 139
Activity Recognition Based on Semi-supervised Learning
Guan, D., Yuan, W., Lee, Y.-K., Gavrilov, A., & Lee, S. (2007) · IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA) · doi:10.1109/rtcsa.2007.17 · Cited by 110
Semi-supervised convolutional neural networks for human activity recognition
Zeng, M., Yu, T., Wang, X., Nguyen, L. T., Mengshoel, O. J., & Lane, I. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8257967 · Cited by 85 · cited 1×
“[14], [15] resort to deep generative models such as Restricted Boltzmann Machines (RBMs) and”
MethodSelf-citationGuo, H., Chen, L., Peng, L., & Chen, G. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971708 · Cited by 83 · cited 2×
“Most existing works follow a supervised learning approach [3], [4], thus requiring a significant amount of training data to recognize meaningful activities.”
BackgroundResultI-POMDP: An infomax model of eye movement
Butko, N. J., & Movellan, J. R. (2008) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2008.4640819 · Cited by 70
Multi-graph Based Semi-supervised Learning for Activity Recognition
Stikic, M., Larlus, D., & Schiele, B. (2009) · 2009 International Symposium on Wearable Computers · doi:10.1109/iswc.2009.24 · Cited by 68
Learning from less for better: semi-supervised activity recognition via shared structure discovery
Yao, L., Nie, F., Sheng, Q. Z., Gu, T., Li, X., & Wang, S. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2971648.2971701 · Cited by 58
Self-citationA neural network algorithm for semi-supervised node label learning from unbalanced data
Frasca, M., Bertoni, A., Ré, M., & Valentini, G. (2013) · Neural Networks · doi:10.1016/j.neunet.2013.01.021 · Cited by 57
Distributionally Robust Semi-Supervised Learning for People-Centric Sensing
Chen, K., Yao, L., Zhang, D., Chang, X., Long, G., & Wang, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013321 · Cited by 49
Self-citationMulti-agent Attentional Activity Recognition
Chen, K., Yao, L., Zhang, D., Guo, B., & Yu, Z. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/186 · Cited by 41
Self-citationChen, K., Yao, L., Wang, X., Zhang, D., Gu, T., Yu, Z., & Yang, Z. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489767 · Cited by 36
Self-citationUncertainty sampling methods for one-class classifiers
Juszczak, P., & Duin, R. P. W. (2003) · Cited by 30
Research works printed but not linked: 1
- cheng, 2016, Semi-supervised multimodal deep learning for RGB-D object recognition, Proc Int Joint Conf Artif Intell (IJCAI), 3345
Cited by
Deep Learning for Sensor-based Human Activity Recognition
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2022) · ACM Computing Surveys · doi:10.1145/3447744 · Cited by 690
“[28] applied co-training with multiple classifiers on different modalities of the data.”
BackgroundSelf-citationDeep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2020) · arXiv · doi:10.48550/arxiv.2001.07416 · Cited by 314
“[28] applied co-training with multiple classifiers on different modalities of the data.”
BackgroundSelf-citationSenseFi: A library and benchmark on deep-learning-empowered WiFi human sensing
Yang, J., Chen, X., Zou, H., Lu, C. X., Wang, D., Sun, S., & Xie, L. (2023) · Patterns · doi:10.1016/j.patter.2023.100703 · Cited by 209
Learning Adaptive Spatial-Temporal Context-Aware Correlation Filters for UAV Tracking
Yuan, D., Chang, X., Li, Z., & He, Z. (2022) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3486678 · Cited by 159
Self-citationHuman Activity Recognition: Review, Taxonomy and Open Challenges
Arshad, M. H., Bilal, M., & Gani, A. (2022) · Sensors · doi:10.3390/s22176463 · Cited by 206
Metadata
- Authors
- Chen, K., Yao, L., Zhang, D., Wang, X., Chang, X., & Nie, F.
- Affiliations
- UNSW Sydney (AU)
- University of Technology Sydney (AU)
- Monash University (AU)
- Northwestern Polytechnical University (CN)
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2020-05
- Volume
- 31
- Issue
- 5
- Pages
- 1747-1756
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2963521811
- Cited by
- 368
- References
- 33
- Citations by year
- 2026: 6 · 2025: 34 · 2024: 31 · 2023: 126 · 2022: 141 · 2021: 22 · 2020: 7
- FWCI
- 19.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Figshare (submittedVersion)
- UTS ePRESS (University of Technology Sydney) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
- Keywords
- human activity recognition, semi-supervised learning, class imbalance, limited labeled data, inter-class similarity
- MeSH terms
- Supervised Machine Learning, Humans, Human Activities, Pattern Recognition, Automated, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.