Deep Auto-Set: A Deep Auto-Encoder-Set Network for Activity Recognition Using Wearables
Abstract
Automatic recognition of human activities from time-series sensor data (referred to as HAR) is a growing area of research in ubiquitous computing. Most recent research in the field adopts supervised deep learning paradigms to automate extraction of intrinsic features from raw signal inputs and addresses HAR as a multi-class classification problem where detecting a single activity class within the duration of a sensory data segment suffices. However, due to the innate diversity of human activities and their corresponding duration, no data segment is guaranteed to contain sensor recordings of a single activity type. In this paper, we express HAR more naturally as a set prediction problem where the predictions are sets of ongoing activity elements with unfixed and unknown cardinality. For the first time, we address this problem by presenting a novel HAR approach that learns to output activity sets using deep neural networks. Moreover, motivated by the limited availability of annotated HAR datasets as well as the unfortunate immaturity of existing unsupervised systems, we complement our supervised set learning scheme with a prior unsupervised feature learning process that adopts convolutional auto-encoders to exploit unlabeled data. The empirical experiments on two widely adopted HAR datasets demonstrate the substantial improvement of our proposed methodology over the baseline models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent work towards unsupervised pre-training has gone beyond the early introduction using Restricted Boltzmann Machines (RBMs), involving (variants of) autoencoders and self-supervision.
“Recent work towards such unsupervised pre-training has gone beyond the early introduction using Restricted Boltzmann Machines (RBMs) [63], involving (variants of) autoencoders [31, 74], and self-supervision [32, 69].”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [74]
Most recent unsupervised pre-training approaches do not specifically target temporality, a characteristic inherent to body-worn sensor data.
“While they result in effective representations, most of these approaches do not specifically target a characteristic inherent to body-worn sensor data - temporality.”
From Contrastive Predictive Coding for Human Activity Recognition (2021) · cites this paper as [74]
Autoencoders are most commonly used for feature extraction and dimensionality reduction.
“As such, autoencoders are most commonly used for feature extraction and dimensionality reduction [120,122,123,124,125,126,133,134,135,136,137,138,139,140,141].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [120]
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
(2017) · Cited by 11,625 · see citing works
Proceedings of the 19th International Joint Conference on Artificial Intelligence
Pujol, J. M., Delgado, J., Ramon Sangüesa, A., & Flache, A. (2005) · International Joint Conference on Artificial Intelligence · Cited by 5,771 · see citing works
Activity recognition using cell phone accelerometers
Kwapisz, J. R., Weiss, G. M., & Moore, S. A. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964918 · Cited by 2,912 · cited 2× · see citing works
“• WISDM Actitracker dataset [12]: This dataset contains 1,098,207 triaxial accelerometer readings gathered from 36 users which reflect activity patterns of walking, jogging, sitting, standing, and climbing stairs.”
BackgroundMethodDeep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 7× · see citing works
“In this architecture, all convolution (and deconvolution) layers apply afilter ofwidth 5 (as in [16]) and”
BackgroundMethodInfluentialWhy Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010) · International Conference on Artificial Intelligence and Statistics · doi:10.5555/1756006.1756025 · Cited by 2,579 · see citing works
A tutorial on human activity recognition using body-worn inertial sensors
Bulling, A., Blanke, U., & Schiele, B. (2014) · ACM Computing Surveys · doi:10.1145/2499621 · Cited by 1,668 · see citing works
The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 2× · see citing works
“• Opportunity dataset [6]: This dataset comprises annotated recordings from a wide variety of on-body sensors configured on four subjects while carrying out morning activities.”
BackgroundMethodHuman Activity Recognition and Pattern Discovery
Kim, E., Helal, S., & Cook, D. J. (2010) · IEEE Pervasive Computing · doi:10.1109/mprv.2010.7 · Cited by 704 · see citing works
Proceedings of the 24th International Conference on Artificial Intelligence
Yang, Q., & Wooldridge, M. (2015) · International Conference on Artificial Intelligence · Cited by 649 · see citing works
Analyzing features for activity recognition
Huỳnh, T., & Schiele, B. (2005) · Proceedings of the 2005 joint conference on Smart objects and ambient intelligence: innovative context-aware services: usages and technologies · doi:10.1145/1107548.1107591 · Cited by 336 · see citing works
Centinela: A human activity recognition system based on acceleration and vital sign data
Lara, Ó. D., Pérez, A. J., Labrador, M. A., & Posada, J. (2011) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2011.06.004 · Cited by 241 · see citing works
Deep Activity Recognition Models with Triaxial Accelerometers
Alsheikh, M. A., Selim, A., Niyato, D., Doyle, L., Lin, S., & Tan, H.-P. (2015) · arXiv · doi:10.48550/arxiv.1511.04664 · Cited by 126 · see citing works
Motion primitive-based human activity recognition using a bag-of-features approach
Zhang, M., & Sawchuk, A. A. (2012) · International Health Informatics Symposium · doi:10.1145/2110363.2110433 · Cited by 125 · see citing works
Yao, R., Lin, G., Shi, Q., & Ranasinghe, D. C. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.12.024 · Cited by 108 · see citing works
Self-citationMultimodal recognition of reading activity in transit using body-worn sensors
Bulling, A., Ward, J. A., & Gellersen, H. W. (2012) · ACM Transactions on Applied Perception · doi:10.1145/2134203.2134205 · Cited by 95 · see citing works
Sequence Learning with Passive RFID Sensors for Real-Time Bed-Egress Recognition in Older People
Wickramasinghe, A., Ranasinghe, D. C., Fumeaux, C., Hill, K., & Visvanathan, R. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2576285 · Cited by 72 · see citing works
Self-citationTorres, R. L. S., Shi, Q., van den Hengel, A., & Ranasinghe, D. C. (2017) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2017.04.002 · Cited by 13 · see citing works
Self-citationJoint Learning of Set Cardinality and State DistributionPreprint
Rezatofighi, S. H., Milan, A., Shi, Q., Dick, A., & Reid, I. D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04093 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625 · see citing works
“As such, autoencoders are most commonly used for feature extraction and dimensionality reduction [18, 27, 53, 73, 74, 80, 103, 137, 171, 172, 194, 234, 283, 287, 296, 298].”
BackgroundMethodDeep Learning for Sensor-based Human Activity Recognition
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2022) · ACM Computing Surveys · doi:10.1145/3447744 · Cited by 690 · see citing works
“to predict a label for each time step. Therefore, the FCNs could not only use the information of the corresponding time step itself but also utilize the information of its neighboring time steps. In [148], Varamin et al. designed a multi-label architecture to simultaneously predict the number of ongoing activities and…”
BackgroundDeep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2020) · arXiv · doi:10.48550/arxiv.2001.07416 · Cited by 314 · see citing works
“to predict a label for each time step. Therefore, the FCNs could not only use the information of the corresponding time step itself but also utilize the information of its neighboring time steps. In [148], Varamin et al. designed a multi-label architecture to simultaneously predict the number of ongoing activities and…”
BackgroundContrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3463506 · Cited by 150 · see citing works
“[74] define HAR as a set prediction problems within the autoencoder setup.”
BackgroundMethodAbedin, A., Ehsanpour, M., Shi, Q., Rezatofighi, H., & Ranasinghe, D. C. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3448083 · Cited by 107 · see citing works
Self-citationSuh, S., Rey, V. F., & Lukowicz, P. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110143 · Cited by 107 · see citing works
Thakur, D., Biswas, S., Ho, E. S. L., & Chattopadhyay, S. (2022) · IEEE Access · doi:10.1109/access.2022.3140373 · Cited by 104 · see citing works
Energy-aware human activity recognition for wearable devices: A comprehensive review
Contoli, C., Freschi, V., & Lattanzi, E. (2024) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2024.101976 · Cited by 43 · see citing works
A Multitask Deep Learning Approach for Sensor-Based Human Activity Recognition and Segmentation
Duan, F., Zhu, T., Wang, J., Chen, L., Ning, H., & Wan, Y. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3273673 · Cited by 34 · see citing works
Kwon, H., Abowd, G. D., & Plötz, T. (2021) · Sensors · doi:10.3390/s21248337 · Cited by 30 · see citing works
Zilelioglu, H., Khodabandelou, G., Chibani, A., & Amirat, Y. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3267243 · Cited by 30 · see citing works
Towards deep clustering of human activities from wearables
Abedin, A., Motlagh, F., Shi, Q., Rezatofighi, H., & Ranasinghe, D. (2020) · Proceedings of the 2020 International Symposium on Wearable Computers · doi:10.1145/3410531.3414312 · Cited by 27 · see citing works
Self-citationInvestigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Plötz, T. (2023) · 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom) · doi:10.1109/percom56429.2023.10099197 · Cited by 27 · see citing works
Adversarial Deep Feature Extraction Network for User Independent Human Activity Recognition
Suh, S., Rey, V. F., & Lukowicz, P. (2022) · 2022 IEEE International Conference on Pervasive Computing and Communications (PerCom) · doi:10.1109/percom53586.2022.9762387 · Cited by 25 · see citing works
Robust Feature Representation Using Multi-Task Learning for Human Activity Recognition
Azadi, B., Haslgrübler, M., Anzengruber-Tánase, B., Sopidis, G., & Ferscha, A. (2024) · Sensors · doi:10.3390/s24020681 · Cited by 20 · see citing works
Thakur, D., Roy, S., Biswas, S., Ho, E. S. L., Chattopadhyay, S., & Shetty, S. S. (2023) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri58017.2023.00032 · Cited by 19 · see citing works
Faridee, A. Z. M., Khan, M. A. A. H., Pathak, N., & Roy, N. (2019) · EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) · doi:10.1145/3360774.3360831 · Cited by 17 · see citing works
HAR-GCNN: Deep Graph CNNs for Human Activity Recognition From Highly Unlabeled Mobile Sensor Data
Mohamed, A., Lejarza, F., Cahail, S., Claudel, C. G., & Thomaz, E. (2022) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops53856.2022.9767259 · Cited by 15 · see citing works
Cengiz, A. B., Birant, K. U., Cengiz, M., Birant, D., & Baysari, K. (2022) · Symmetry · doi:10.3390/sym14102022 · Cited by 15 · see citing works
ARC-Net: Activity Recognition Through Capsules
Damirchi, H., Khorrambakht, R., & Taghirad, H. D. (2020) · International Conference on Machine Learning and Applications · doi:10.1109/icmla51294.2020.00215 · Cited by 15 · see citing works
Haresamudram, H., Essa, I., & Plötz, T. (2024) · Sensors · doi:10.3390/s24041238 · Cited by 14 · see citing works
Comparing self-supervised learning techniques for wearable human activity recognition
Ek, S., Presotto, R., Civitarese, G., Portet, F., Lalanda, P., & Bettini, C. (2025) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-024-00182-9 · Cited by 15 · see citing works
“Other studies proposed generative meth-ods (Varamin et al. 2018; Haresamudram et al. 2020; Zhang et al. 2022) and demonstrated competitive performances.”
HAR-GCNN: Deep Graph CNNs for Human Activity Recognition From Highly Unlabeled Mobile Sensor Data
Mohamed, A., Lejarza, F., Cahail, S., Claudel, C. G., & Thomaz, E. (2022) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops53856.2022.9767342 · Cited by 13 · see citing works
Zhang, L., Zhang, W., & Japkowicz, N. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412851 · Cited by 13 · see citing works
Human Action Recognition Using ConvBiLSTM-GRU In Indoor Environment
Sain, M. K., Singha, J., Saini, S., & Semwal, V. B. (2023) · IEEE Global Conference on Artificial Intelligence and Internet of Things (GCAIoT) · doi:10.1109/gcaiot61060.2023.10385107 · Cited by 12 · see citing works
Federated Self-Supervised Learning in Heterogeneous Settings: Limits of a Baseline Approach on HAR
Ek, S., Rombourg, R., Portet, F., & Lalanda, P. (2022) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops53856.2022.9767369 · Cited by 11 · see citing works
Novel Human Activity Recognition by graph engineered ensemble deep learning model
Ghalan, M., & Aggarwal, R. K. (2024) · IFAC Journal of Systems and Control · doi:10.1016/j.ifacsc.2024.100253 · Cited by 11 · see citing works
Deep Learning Methods applied to Intrusion Detection: Survey, Taxonomy and Challenges
Lifandali, O., & Abghour, N. (2021) · International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa53625.2021.9682357 · Cited by 9 · see citing works
Tan, T.-H., Shih, J.-Y., Liu, S.-H., Alkhaleefah, M., Chang, Y.-L., & Gochoo, M. (2023) · Sensors · doi:10.3390/s23063354 · Cited by 8 · see citing works
An Optimal Feature Selection Method for Human Activity Recognition Using Multimodal Sensory Data
Haider, T., Khan, M. H., & Farid, M. S. (2024) · Information · doi:10.3390/info15100593 · Cited by 9 · see citing works
BSTCA-HAR: Human Activity Recognition Model Based on Wearable Mobile Sensors
Yuan, Y., Huang, L., Tan, X., Yang, F., & Yang, S. (2024) · Applied Sciences · doi:10.3390/app14166981 · Cited by 5 · see citing works
Kulkarni, A., Seetharam, A., & Ramesh, A. (2019) · EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) · doi:10.1145/3360774.3360803 · Cited by 5 · see citing works
Unsupervised deep learning of bright-field images for apoptotic cell classification
Zheng, Z., Sun, B., He, S., Wang, G., Bi, C., & Chen, T. (2023) · Signal Image and Video Processing · doi:10.1007/s11760-023-02592-1 · Cited by 4 · see citing works
A Comprehensive Review of Deep Learning for Activity Recognition
Thakur, D., & Fortino, G. (2024) · Internet of things · doi:10.1007/978-3-031-60027-2_4 · Cited by 2 · see citing works
DeepER: A Deep Learning based Emergency Resolution Time Prediction System
Bejarano, G., Kulkarni, A., Luo, X., Seetharam, A., & Ramesh, A. (2020) · IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) · doi:10.1109/ithings-greencom-cpscom-smartdata-cybermatics50389.2020.00090 · Cited by 2 · see citing works
Guided-GAN: Adversarial Representation Learning for Activity Recognition with WearablesPreprint
Abedin, A., Rezatofighi, H., & Ranasinghe, D. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05732 · Cited by 5 · see citing works
Self-citationA two-phase hybrid clustering framework exploring transitional activities in HAR
Woo, M., Harby, A. A., Zulkernine, F. H., & Abdulsalam, H. M. (2025) · Discover Artificial Intelligence · doi:10.1007/s44163-025-00503-6 · Cited by 1 · see citing works
A Multiscale Windows Deep Learning Aproach For Sensor-Based Human Activity Recogniton
Duan, F., Zhu, T., Wang, J., & Liu, Z. (2022) · International Conference on Data Intelligence and Security (ICDIS) · doi:10.1109/icdis55630.2022.00074 · Cited by 1 · see citing works
S-THAD: a framework for sensor-based temporal human activity detection from continuous data streams
Ehatisham-ul-Haq, M., Azam, M. A., Murtaza, F., Amin, Y., & Fu, Q. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-02931-5 · Cited by 1 · see citing works
Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition
Haresamudram, H., Essa, I., & Ploetz, T. (2022) · arXiv · Cited by 27 · see citing works
Zhang, L., Zhang, W., & Japkowicz, N. (2020) · arXiv (Cornell University) · doi:10.13016/m2vboa-m2ab
Haresamudram, H., Essa, I. A., & Plötz, T. (2024) · International Conference on Activity and Behavior Computing (ABC) · doi:10.1109/abc61795.2024.10651688
Augmentation of Human Activity Data: Convert, Generate, Transform
Nair, N. R., Matei, A., Krön, D., Rueda, F. M., Reining, C., & Fink, G. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78192-6_5
EXACT: A Meta-Learning Framework for Precise Exercise Segmentation in Physical Therapy
Wang, H. D., Bae, S., Sun, X., Thatigotla, Y., & Ma, M. (2025) · ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS) · doi:10.1145/3716550.3722027
Huan, R., Zhou, Y., Dong, G., Chen, P., Liang, R., & Chen, C. (2026) · Pattern Analysis and Applications · doi:10.1007/s10044-026-01750-8
- Authors
- Varamin, A. A., Abbasnejad, E., Shi, Q., Ranasinghe, D. C., & Rezatofighi, H.
- Affiliations
- The University of Adelaide, Adelaide SA, Australia
- The University of Adelaide (AU)
- Venue
- Proceedings of the 15th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services
- Publisher
- ACM
- Published
- 2018-11-05
- Pages
- 246-253
- OpenAlex
- W2897943474
- Cited by
- 44
- References
- 20
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 9 · 2023: 7 · 2022: 9 · 2021: 8 · 2020: 5 · 2019: 2
- FWCI
- 1.13
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- human activity recognition, set prediction, deep neural network, convolutional autoencoder, unsupervised feature learning, unlabeled data, time-series sensor data, activity duration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
