Handling annotation uncertainty in human activity recognition
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Developing systems for Human Activity Recognition (HAR) using wearables typically relies on datasets that were manually annotated by human experts with regards to precise timings of instances of relevant activities. However, obtaining such data annotations is often very challenging in the predominantly mobile scenarios of Human Activity Recognition. As a result, labels often carry a degree of uncertainty-label jitter-with regards to: i) correct temporal alignments of activity boundaries; and ii) correctness of the actual label provided by the human annotator. In this work, we present a scheme that explicitly incorporates label jitter into the model training process. We demonstrate the effectiveness of the proposed method through a systematic experimental evaluation on standard recognition tasks for which our method leads to significant increases of mean F1 scores.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous work showed that handling outlier samples in HAR datasets is important and, if successful, can significantly improve the performance of the resulting HAR model [91,92,93].”
From Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables (2021) · cites this paper as [92]
Citations
Cites
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Introduction to Statistical Pattern Recognition
(1990) · Elsevier eBooks · doi:10.1016/c2009-0-27872-x · Cited by 11,031
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images
Nguyen, A., Yosinski, J., & Clune, J. (2014) · arXiv · doi:10.48550/arxiv.1412.1897 · Cited by 3,258
Zadeh, L. A. (1997) · Fuzzy Sets and Systems · doi:10.1016/s0165-0114(97)00077-8 · Cited by 2,897
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 4×
“For this evaluation, we used the Opportunity dataset [6], and a state-of-the-art Con-vLSTM classifier [24], which not only automatically learns the feature representation but also models long-term temporal contexts.”
MethodInfluentialA tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668 · cited 1×
“When training, the model predicted the target label probability at the last timestep in the window [5, 24].”
MethodWhen does label smoothing help?
Müller, R., Kornblith, S., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1906.02629 · Cited by 859
The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 2×
“For this evaluation, we used the Opportunity dataset [6], and a state-of-the-art Con-vLSTM classifier [24], which not only automatically learns the feature representation but also models long-term temporal contexts.”
MethodTransition-Aware Human Activity Recognition Using Smartphones
Reyes-Ortiz, J.-L., Oneto, L., Samà, A., Parra, X., & Anguita, D. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.07.085 · Cited by 740 · cited 2×
“Qualitative analysis: According to [27], when the majority of the activities are frequent and short, the trained model shows fluctuating predictions due to the ambiguity in activity boundaries.”
BackgroundWearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom
Bachlin, M., Plotnik, M., Roggen, D., Maidan, I., Hausdorff, J. M., Giladi, N., & Troster, G. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2036165 · Cited by 693 · cited 3×
“For Daphnet Gait (DG) [2], the experiment protocol in [8] was utilized.”
BackgroundMethodPrinciples of Risk Minimization for Learning Theory
Vapnik, V. N. (1991) · neural information processing systems · Cited by 692
Regularizing Neural Networks by Penalizing Confident Output Distributions
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1701.06548 · Cited by 655
Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606
Self-citationCombining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · cited 1×
“For 50salads [30], we followed the accelerometer-only protocol from the original paper as suggested by the authors.”
MethodPerformance metrics for activity recognition
Ward, J. A., Lukowicz, P., & Gellersen, H. W. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1889681.1889687 · Cited by 182 · cited 2×
“We evaluated the proposed SAT-TR, SAT-OV, and SAT with the mean F1 score and the Wilson score interval (95% confidence) [35], and the misalignment analysis [34].”
BackgroundMethodHammerla, N. Y., Kirkham, R., Andras, P., & Ploetz, T. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494353 · Cited by 138 · cited 1×
“As classifiers we used Random Forests (RF) with ECDF features [9], Convolutional Neural Network (CNN), and Convolutional Long short-term memory (ConvLSTM) neural networks [24], to investigate the effect of SAT across different classifier types.”
MethodSelf-citationA comparison of clinical and objective measures of freezing of gait in Parkinson's disease
Morris, T. R., Cho, C., Dilda, V., Shine, J. M., Naismith, S. L., Lewis, S. J. G., & Moore, S. T. (2012) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2012.03.001 · Cited by 114
Learning convolutional action primitives for fine-grained action recognition
Lea, C., Vidal, R., & Hager, G. D. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487305 · Cited by 106
Fuzzy human motion analysis: A review
Lim, C. H., Vats, E., & Chan, C. S. (2014) · Pattern Recognition · doi:10.1016/j.patcog.2014.11.016 · Cited by 94
Real world activity recognition with multiple goals
Hao Hu, D., Pan, S. J., Zheng, V. W., Liu, N. N., & Yang, Q. (2008) · Proceedings of the 10th international conference on Ubiquitous computing · doi:10.1145/1409635.1409640 · Cited by 71
Spatiotemporal Coding in ANVIL
Kipp, M. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/25wozp3pdwif · Cited by 69
Bezdek, J. C. (1992) · IEEE Communications Magazine · doi:10.1109/35.156801 · Cited by 62
Freezing of Gait detection in Parkinson's disease using accelerometer based smart clothes
Niazmand, K., Tonn, K., Zhao, Y., Fietzek, U. M., Schroeteler, F., Ziegler, K., Ceballos-Baumann, A. O., & Lueth, T. C. (2011) · 2011 IEEE Biomedical Circuits and Systems Conference (BioCAS) · doi:10.1109/biocas.2011.6107762 · Cited by 58
Plotz, T., Chen, C., Hammerla, N. Y., & Abowd, G. D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.15 · Cited by 51
Self-citationWearables in the wet lab: a laboratory system for capturing and guiding experiments
Scholl, P. M., Wille, M., & Van Laerhoven, K. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2807547 · Cited by 45 · cited 1×
“In Wetlab [29], the synchronization misalignment between video and accelerometer stream was observed (up to 5 minutes in some sessions).”
BackgroundLearning to Recognize Human Activities Using Soft Labels
Hu, N., Englebienne, G., Lou, Z., & Krose, B. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2621761 · Cited by 29 · cited 3×
“[1, 12] represented labels as uniform contributions from multiple annotators.”
BackgroundMethodTagging human activities in video by crowdsourcing
Nguyen-Dinh, L.-V., Waldburger, C., Roggen, D., & Tröster, G. (2013) · International Conference on Multimedia Retrieval · doi:10.1145/2461466.2461508 · Cited by 26
Creating and Exploring Semantic Annotation for Behaviour Analysis
Yordanova, K., & Krüger, F. (2018) · Sensors · doi:10.3390/s18092778 · Cited by 20
Harnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition
Aung, A. M., & Whitehill, J. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00033 · Cited by 19
Automatic correction of annotation boundaries in activity datasets by class separation maximization
Kirkham, R., Khan, A., Bhattacharya, S., Hammerla, N. Y., Mellor, S., Roggen, D., & Ploetz, T. (2013) · Ubiquitous Computing · doi:10.1145/2494091.2495988 · Cited by 18
Training human activity recognition for labels with inaccurate time stamps
Toda, T., Inoue, S., Tanaka, S., & Ueda, N. (2014) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2638728.2641297 · Cited by 18
Multimodal multisensor activity annotation tool
Barz, M., Moniri, M. M., Weber, M., & Sonntag, D. (2016) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2968219.2971459 · Cited by 17
Nasir, M., Baucom, B., Georgiou, P., & Narayanan, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178298 · Cited by 9 · cited 1×
“Other than uniform weights on each annotator, [20] weighted labels according to each annotator’s reliability.”
BackgroundFusion of Valence and Arousal Annotations through Dynamic Subjective Ordinal Modelling
Ruiz, A., Martínez, O., Binefa, X., & Sukno, F. M. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.48 · Cited by 6
A hierarchical representation for human activity recognition with noisy labels
Hu, N., Englebienne, G., Lou, Z., & Kröse, B. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353719 · Cited by 3
Research works printed but not linked: 1
- Xie L., 2016, Disturblabel: Regularizing cnn on the loss layer.. In CVPR. 4753--4762.
Cited by
Human Activity Recognition: Review, Taxonomy and Open Challenges
Arshad, M. H., Bilal, M., & Gani, A. (2022) · Sensors · doi:10.3390/s22176463 · Cited by 206
Quantifying Parkinson’s disease motor severity under uncertainty using MDS-UPDRS videos
Lu, M., Zhao, Q., Poston, K. L., Sullivan, E. V., Pfefferbaum, A., Shahid, M., Katz, M., Montaser‐Kouhsari, L., Schulman, K. A., Milstein, A. S., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102179 · Cited by 108
Gjoreski, M., Gams, M., Luštrek, M., Genc, P., Garbas, J.-U., & Hassan, T. (2020) · IEEE Access · doi:10.1109/access.2020.2986810 · Cited by 106
IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3678545 · Cited by 61
“This hinders the development of robust and generalizable models.”
Self-citationX-CHAR: A Concept-based Explainable Complex Human Activity Recognition Model
Jeyakumar, J. V., Sarker, A., Garcia, L. A., & Srivastava, M. B. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3580804 · Cited by 58
Metadata
- Authors
- Kwon, H., Abowd, G. D., & Plötz, T.
- Affiliations
- Georgia Institute of Technology (US)
- Venue
- Proceedings of the 23rd International Symposium on Wearable Computers
- Publisher
- ACM
- Published
- 2019-09-09
- Pages
- 109-117
- OpenAlex
- W2971670291
- Cited by
- 52
- References
- 36
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 6 · 2023: 6 · 2022: 14 · 2021: 11 · 2020: 3
- FWCI
- 2.28
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium on Wearable Computers (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- human activity recognition, annotation uncertainty, wearable sensors, temporal alignment, label noise
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.