Learning to Recognize Human Activities Using Soft Labels
Open accessThe DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
Abstract
Human activity recognition system is of great importance in robot-care scenarios. Typically, training such a system requires activity labels to be both completely and accurately annotated. In this paper, we go beyond such restriction and propose a learning method that allow labels to be incomplete and uncertain. We introduce the idea of soft labels which allows annotators to assign multiple, and weighted labels to data segments. This is very useful in many situations, e.g., when the labels are uncertain, when part of the labels are missing, or when multiple annotators assign inconsistent labels. We formulate the activity recognition task as a sequential labeling problem. Latent variables are embedded in the model in order to exploit sub-level semantics for better estimation. We propose a max-margin framework which incorporate soft labels for learning the model parameters. The model is evaluated on two challenging datasets. To simulate the uncertainty in data annotation, we randomly change the labels for transition segments. The results show significant improvement over the state-of-the-art approach.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Efficient Graph-Based Image Segmentation
Felzenszwalb, P. F., & Huttenlocher, D. P. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000022288.19776.77 · Cited by 6,176
Graph Implementations for Nonsmooth Convex Programs
Grant, M. C., & Boyd, S. P. (2007) · Lecture Notes in Control and Information Sciences · doi:10.1007/978-1-84800-155-8_7 · Cited by 2,666 · cited 1×
“Many existing solvers are available for solving these problems [21], [22].”
BackgroundSupport vector machine learning for interdependent and structured output spaces
Tsochantaridis, I., Hofmann, T., Joachims, T., & Altun, Y. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015341 · Cited by 1,243 · cited 1×
“The models are trained using the Structured-SVM framework [24].”
MethodCutting-plane training of structural SVMs
Joachims, T., Finley, T., & Yu, C.-N. J. (2009) · Machine Learning · doi:10.1007/s10994-009-5108-8 · Cited by 958 · cited 2×
“For optimization we use the one-slack algorithm (dual) described in [22].”
Accurate activity recognition in a home setting
van Kasteren, T., Noulas, A., Englebienne, G., & Kröse, B. (2008) · Proceedings of the 10th international conference on Ubiquitous computing · doi:10.1145/1409635.1409637 · Cited by 874 · cited 1×
“been widely used among literatures [1], [4], [5], [6], [7], [8], [9].”
MethodSelf-citationLearning human activities and object affordances from RGB-D videos
Koppula, H. S., Gupta, R., & Saxena, A. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913478446 · Cited by 683 · cited 9×
“[1] model both activities and objects affordance as randomvariables.”
BackgroundMethodInfluentialLearning structural SVMs with latent variables
Yu, C.-N. J., & Joachims, T. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553523 · Cited by 610 · cited 2×
“[15] extend the Structured SVM framework to allow for latent variables (Latent Structured-SVM) and show that the model with latent variables has the capacity to model more complex data.”
Hidden Conditional Random Fields for Gesture Recognition
Wang, S. B., Quattoni, A., Morency, L.-P., Demirdjian, D., & Darrell, T. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.132 · Cited by 451 · cited 1×
“been widely used among literatures [1], [4], [5], [6], [7], [8], [9].”
MethodPosterior Regularization for Structured Latent Variable Models
Ganchev, K., Graça, J., Gillenwater, J., & Taskar, B. (2010) · Cited by 440
The Concave-Convex Procedure (CCCP)
Yuille, A., & Rangarajan, A. (2001) · Cited by 397
Cited by
Facial Expression Recognition with Inconsistently Annotated Datasets
Zeng, J., Shan, S., & Chen, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_14 · Cited by 341
“al.[11] dealt with the noisy labels from multiple annotators by proposing a loss that incorporates the “soft labeling” in a max-margin learning framework.”
Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data
Wilson, G., Doppa, J. R., & Cook, D. J. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403228 · Cited by 141
“Hu et al. [18] learn human activity recognition models from video data that may have multiple, uncertain, or incomplete labels.”
BackgroundCALDA: Improving Multi-Source Time Series Domain Adaptation With Contrastive Adversarial Learning
Wilson, G., Doppa, J. R., & Cook, D. J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3298346 · Cited by 83
“[31] propose using a different form of weak supervision for human activity recognition from video data, using multiple incomplete or uncertain labels.”
BackgroundHandling annotation uncertainty in human activity recognition
Kwon, H., Abowd, G. D., & Plötz, T. (2019) · Proceedings of the 23rd International Symposium on Wearable Computers · doi:10.1145/3341163.3347744 · Cited by 52
“[1, 12] represented labels as uniform contributions from multiple annotators.”
BackgroundMethodSwitching Structured Prediction for Simple and Complex Human Activity Recognition
Arzani, M. M., Fathy, M., Azirani, A. A., & Adeli, E. (2020) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2960481 · Cited by 44
One-Shot Learning for Surveillance Anomaly Recognition using Siamese 3D CNN
Ullah, A., Muhammad, K., Haydarov, K., Haq, I. U., Lee, M., & Baik, S. W. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207595 · Cited by 39
“A max-margin based learning framework using soft labelling approach is presented by Hu et al. [20].”
MethodSource Free Semi-Supervised Transfer Learning for Diagnosis of Mental Disorders on fMRI Scans
Hu, Y. F., Huang, Z.-A., Liu, R., Xue, X., Sun, X., Song, L., & Tan, K. C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3298332 · Cited by 53
“, a one-hot vector) in supervised training [50], LS enables soft label representation (i.”
BackgroundAutomated Smart Home Assessment to Support Pain Management: Multiple Methods Analysis
Fritz, R., Wilson, M., Dermody, G., Schmitter‐Edgecombe, M., & Cook, D. J. (2020) · Journal of Medical Internet Research · doi:10.2196/23943 · Cited by 33
“Based on prior work developing techniques to train ML models using small sample sizes [35,36], we determined this amount of data exceeds ML training requirements for models using longitudinal data and that this amount of data would likely allow the model to capture aspects of pain.”
Kwon, H., Abowd, G. D., & Plötz, T. (2021) · Sensors · doi:10.3390/s21248337 · Cited by 30
“Previous work showed that handling outlier samples in HAR datasets is important and, if successful, can significantly improve the performance of the resulting HAR model [91–93].”
BackgroundVisual Perception for Multiple Human–Robot Interaction From Motion Behavior
Benli, E., Motai, Y., & Rogers, J. G. (2019) · IEEE Systems Journal · doi:10.1109/jsyst.2019.2958747 · Cited by 16
“RGB-D sensors are used for human activity recognition using soft labels [27], navigation assistance to guide people with vision loss [28], and human–object interaction [29].”
BackgroundFacial Emotion Recognition with Noisy Multi-task Annotations
Zhang, S., Huang, Z., Paudel, D. P., & Gool, L. V. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00007 · Cited by 20
“Learning with multiple noisy labels has also been explored by estimating ground truth with crowdsourcing [65] or soft labels [17].”
BackgroundUncertainty-Aware Pedestrian Crossing Prediction via Reinforcement Learning
Dai, S., Liu, J., & Cheung, N. M. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3400391 · Cited by 14
“Reference [48] also learns soft labels but with a novel annotation strategy allowing the annotator to assign multiple weighted labels.”
BackgroundOAFPM: optimized ANFIS using frequent pattern mining for activity recognition
Marimuthu, P., Perumal, V., & Vijayakumar, V. (2019) · The Journal of Supercomputing · doi:10.1007/s11227-019-02802-z · Cited by 11
Zhang, Y., Srivastava, A. K., & Cook, D. J. (2020) · IET Smart Grid · doi:10.1049/iet-stg.2019.0249 · Cited by 11
“In the literature, researchers have started to investigate more methods for recognizing activities from many types of sensor sources, including video and audio data [24, 25], wearable data [26, 27], and ambient data [28, 29].”
Learn2Agree: Fitting with Multiple Annotators Without Objective Ground Truth
Wang, C., Gao, Y., Fan, C., Hu, J., Lam, T. L., Lane, N. D., & Bianchi‐Berthouze, N. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-39539-0_13 · Cited by 10
“When the number of annotations per X i is modest, efforts are put into creating models that utilize all the annotations with multi-score learning (Meng et al., 2011) or soft labels (Hu et al., 2016).”
Arzani, M. M., Fathy, M., Aghajan, H. K., Azirani, A. A., Raahemifar, K., & Adeli, E. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202208 · Cited by 9
“One of the integral parts of all such interactive robotics applications is human activity recognition (HAR) [2-7].”
BackgroundMethodSkeleton-based structured early activity prediction
Arzani, M. M., Fathy, M., Azirani, A. A., & Adeli, E. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-08875-w · Cited by 5
“Markov networks were successfully applied for HAR [2, 25, 54], however, most of them only incorporate short or mid-range dependency in their models, while in many applications based on our experience long-range dependencies are also important and can enhance the recognition accuracy.”
BackgroundMethodInfluentialAutomatic Label Assignment for Object Detection
Wang, H., Jia, T., Wang, Q., & Zuo, W. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3578021 · Cited by 3
“…of these three losses following the same configuration in For further evaluating the e ff ectiveness of the proposed positive sample selection strategy, we compare it to the another two methods [40], [41], which both handle the label assignment issue using soft labels based on the graph structure.”
InfluentialHandling of Labeling Uncertainty in Smart Homes using Generalizable Fuzzy Features
Demongivert, C., Bouchard, K., Gaboury, S., Lussier, M., Ngankam, H. K., Couture, M., Bier, N., & Giroux, S. (2021) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3462203.3475909 · Cited by 5
“[14] is also a good display of the various incompatible standards that can be applied for describing the same kind of activity boundaries.”
BackgroundOut-of-distribution in Human Activity Recognition
Roy, D., Komini, V., & Girdzijauskas, Š. (2022) · Swedish Artificial Intelligence Society Workshop (SAIS) · doi:10.1109/sais55783.2022.9833052 · Cited by 3
Classifying falls using out-of-distribution detection in human activity recognition
Roy, D., Komini, V., & Girdzijauskas, Š. (2023) · AI Communications · doi:10.3233/aic-220205 · Cited by 2
“Hue et al. [13] explore annotation uncertainty and mitigate the issue by a soft-labeling strategy.”
Smart discovery of periodic-frequent human routines for home automation
Guinea, A. S., Boytsov, A., Mouline, L., & Traon, Y. L. (2019) · EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) · doi:10.1145/3360774.3360784 · Cited by 1
“Recently, different approaches have been proposed in an attempt to alleviate this issue, allowing the use of uncertainty in the labels [17] or the recognition based on a limited amount of labeled data [42].”
BackgroundRetrieval-Based Annotation for Multi-Channel Time Series Data of Human Activities
Rueda, F. M., Nair, N. R., Spiekermann, R., Altermann, E., Oberdiek, P., Reining, C., & Fink, G. A. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-09117-8_4 · Cited by 1
Data-Driven Estimation of the False Positive Rate of the Bayes Binary Classifier via Soft Labels
Jeong, M., Cardone, M., & Dytso, A. (2024) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit57864.2024.10619564 · Cited by 1
Wang, C., Gao, Y., Fan, C., Hu, J., Lam, T. L., Lane, N. D., & Bianchi‐Berthouze, N. (2021) · arXiv (Cornell University) · Cited by 1
Human Activity Recognition Reimagined: The CGRNHAR Self-Supervised Approach
Challa, A. K. R., Raavi, V. K., Bandari, H., Gude, D. K., & Kumar, S. (2025) · Global AI Summit - International Conference on Artificial Intelligence and Emerging Technology (AI Summit) · doi:10.1109/aisummit66170.2025.11411187
Bandari, H., Gude, D. K., Raavi, V. K., Challa, A. K. R., & Kumar, S. (2025) · Global AI Summit - International Conference on Artificial Intelligence and Emerging Technology (AI Summit) · doi:10.1109/aisummit66170.2025.11411581
“Furthermore, methods of Hu et al. [19] and Khosla et al. [20] aimed at reducing dependency on data, but they have never considered privacy risks, which HAR data sets can dramatically entail (i.e., sensitive personal data).”
Kaushik, P., Soni, J., Bhawna, Varshney, N. K., Tyagi, A., & Garg, K. K. (2026) · IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI) · doi:10.1109/iatmsi68868.2026.11465588
“Hu et al. (2016) suggested soft-label learning methods to enhance the accuracy of recognizing human activity based on the pattern analysis and machine intelligence approaches [7].”
ERC4AI: A BERT‐Based Model for Ethical Requirements Classification in AI Systems
Ximenes, E. D. F., de Cerqueira, J. A. S., Abrahamsson, P., & Canedo, E. D. (2026) · Journal of Software Evolution and Process · doi:10.1002/smr.70156 · Cited by 1
Metadata
- Authors
- Hu, N., Englebienne, G., Lou, Z., & Krose, B.
- Affiliations
- University of Amsterdam (NL)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2017-10-01
- Volume
- 39
- Issue
- 10
- Pages
- 1973-1984
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2541965240
- Cited by
- 29
- References
- 27
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 2 · 2023: 4 · 2022: 1 · 2021: 4 · 2020: 6 · 2019: 4 · 2018: 1 · 2017: 1
- FWCI
- 1.36
- Type
- journal article
- Language
- en
- Open access
- green
- research.utwente.nl
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- University of Twente Research Information (publishedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- University of Twente Research Information (submittedVersion, other-oa)
- Funders
- European project ACCOMPANY (287624)
- European project MONARCH (601033)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- human activity recognition, soft labels, incomplete labels, multiple annotators, label inconsistency, sequence labeling, latent variables, max-margin learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.