Attention-Based Deep Learning Framework for Human Activity Recognition With User Adaptation
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Sensor-based human activity recognition (HAR) requires to predict the action of a person based on sensor-generated time series data. HAR has attracted major interest in the past few years, thanks to the large number of applications enabled by modern ubiquitous computing devices. While several techniques based on hand-crafted feature engineering have been proposed, the current state-of-the-art is represented by deep learning architectures that automatically obtain high level representations and that use recurrent neural networks (RNNs) to extract temporal dependencies in the input. RNNs have several limitations, in particular in dealing with long-term dependencies. We propose a novel deep learning framework, TrASenD, based on a purely attention-based mechanism, that overcomes the limitations of the state-of-the-art. We show that our proposed attention-based architecture is considerably more powerful than previous approaches, with an average increment, of more than 7% on the F1 score over the previous best performing model. Furthermore, we consider the problem of personalizing HAR deep learning models, which is of great importance in several applications. We propose a simple and effective transfer-learning based strategy to adapt a model to a specific user, providing an average increment of 6% on the F1 score on the predictions for that user. Our extensive experimental evaluation proves the significantly superior capabilities of our proposed framework over the current state-of-the-art and the effectiveness of our user adaptation technique.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Future research directions include the optimization of the personalization process to minimize the feedback required from the user, for example by using data augmentation or curriculum training techniques [5].”
BackgroundDeep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 2×
“We then consider DeepConvLSTM [28] which is a CNN+LSTM approach, and its new attentive version proposed in [26] that we call DeepConvLSTM-Att.”
MethodDeep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 1×
“PAMAP2 [32, 33].”
Nweke, H. F., Teh, Y. W., Al-garadi, M. A., & Alo, U. R. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.056 · Cited by 930
Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714 · cited 3×
“As non-deep learning baselines we considered a Random Forest (RF) classifier (one of the most used and most effective shallow classifiers for HAR [41]) on the same raw frequency domain features fed to the deep learning approaches (denoted with RF-FF ), and then on the most used handcrafted frequency…”
BackgroundMethodDeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667 · cited 8×
“We start with a description of the architectural template defined by the DeepSense framework [47] 2 .”
BackgroundMethodInfluentialDeep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606 · cited 1×
“, [18], [34], [42]) use CNNs directly on the raw data coming from the sensors (each dimension of the data”
MethodPreprocessing techniques for context recognition from accelerometer data
Figo, D., Diniz, P. C., Ferreira, D. R., & Cardoso, J. M. P. (2010) · Personal and Ubiquitous Computing · doi:10.1007/s00779-010-0293-9 · Cited by 578 · cited 1×
“Several previously proposed approaches tackled the problem by hand-crafting features [12, 41].”
BackgroundUSC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors
Zhang, M., & Sawchuk, A. A. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370438 · Cited by 554 · cited 1×
“USC-HAD [50].”
Ensembles of Deep LSTM Learners for Activity Recognition using Wearables
Guan, Y., & Plötz, T. (2017) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3090076 · Cited by 511
Bianchi, V., Bassoli, M., Lombardo, G., Fornacciari, P., Mordonini, M., & De Munari, I. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2920283 · Cited by 466 · cited 1×
“Despite being a well studied and mature problem, HAR has been a very active research area in recent years, due to the rise of ubiquitous computing enabled by smartphones, wearables, and Internet-of-Things devices [27, 48, 6].”
BackgroundTransfer learning for activity recognition: a survey
Cook, D., Feuz, K. D., & Krishnan, N. C. (2013) · Knowledge and Information Systems · doi:10.1007/s10115-013-0665-3 · Cited by 447 · cited 1×
“In particular transfer learning has been leveraged to compensate for the amount of labeled data when training a model for activity recognition in different environments/circumstances [22, 11].”
BackgroundCreating and benchmarking a new dataset for physical activity monitoring
Reiss, A., & Stricker, D. (2012) · Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments · doi:10.1145/2413097.2413148 · Cited by 295 · cited 1×
“PAMAP2 [32, 33].”
Ravi, D., Wong, C., Lo, B., & Yang, G.-Z. (2016) · 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN) · doi:10.1109/bsn.2016.7516235 · Cited by 269 · cited 1×
“On the other hand, Model Driven approaches [e.g. 21, 31, 39] first preprocess the data to get a grid-like structure, and then use CNNs.”
MethodAttnSense: Multi-level Attention Mechanism For Multimodal Human Activity Recognition
Ma, H., Li, W., Zhang, X., Gao, S., & Lu, S. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/431 · Cited by 266 · cited 3×
“Another recently proposed architecture based on the DeepSense framework, which adopts a similar attention strategy to SADeepSense is AttnSense [24].”
MethodPD Disease State Assessment in Naturalistic Environments Using Deep Learning
Hammerla, N., Fisher, J., Andras, P., Rochester, L., Walker, R., & Ploetz, T. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9484 · Cited by 179 · cited 1×
“Finally, other deep learning techniques used for HAR are autoencoders [44, 2], and Restricted Boltzmann Machines [14, 20, 30].”
MethodOn attention models for human activity recognition
Murahari, V. S., & Plötz, T. (2018) · Proceedings of the 2018 ACM International Symposium on Wearable Computers · doi:10.1145/3267242.3267287 · Cited by 157 · cited 1×
“We then consider DeepConvLSTM [28] which is a CNN+LSTM approach, and its new attentive version proposed in [26] that we call DeepConvLSTM-Att.”
MethodDeep Learning for RFID-Based Activity Recognition
Li, X., Zhang, Y., Marsic, I., Sarcevic, A., & Burd, R. S. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994569 · Cited by 150 · cited 1×
“, [25], [36], [45], [48], [58]) first preprocess the data to get a gridlike structure, and then use CNNs.”
BackgroundTowards multimodal deep learning for activity recognition on mobile devices
Radu, V., Lane, N. D., Bhattacharya, S., Mascolo, C., Marina, M. K., & Kawsar, F. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct · doi:10.1145/2968219.2971461 · Cited by 147 · cited 3×
“0:9, and = 10 8. 5.4 Results Table 2: F1 score results of the considered models on three HAR datasets. Model Dataset HHAR PAMAP2 USC-HAD RF-FF 0.569 0.512 0.417 RF-HC [41] 0.575 0.501 0.474 MultiRBM [30] 0.647 0.589 0.598 LSTM 0.663 0.583 0.612 DeepConvLSTM [28] 0.701 0.633 0.658 DeepConvLSTM-Att [26] 0.735 0.647 0.68…”
MethodInfluentialYao, S., Zhao, Y., Shao, H., Liu, D., Liu, S., Hao, Y., Piao, A., Hu, S., Lu, S., & Abdelzaher, T. F. (2019) · IEEE INFOCOM 2019 - IEEE Conference on Computer Communications · doi:10.1109/infocom.2019.8737500 · Cited by 52
User adaptation of convolutional neural network for human activity recognition
Matsui, S., Inoue, N., Akagi, Y., Nagino, G., & Shinoda, K. (2017) · 2017 25th European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081308 · Cited by 46
Research works printed but not linked: 35
- 10.1109/JSEN.2020.2987306 · DOI
- 10.1109/JSEN.2014.2370945 · DOI · Paper page
- 10.1109/TSMC.2017.2705582 · DOI · Paper page
- 10.1109/JSEN.2020.2992879 · DOI
- abadi, 2016, TensorFlow: A system for large-scale machine learning, Proc USENIX Symp on Operating System Design and Implementation, 265
- 10.1007/s10015-017-0422-x · DOI · Paper page
- ioffe, 2015, Batch normalization: Accelerating deep network training by reducing internal covariate shift, Proc Int Conf Mach Learn, 37, 448
- 10.1145/2987354.2987355 · DOI · Paper page
- 10.1109/JSEN.2019.2946095 · DOI · Paper page
- 10.1109/SIBGRAPI.2011.41 · doi:10.1109/sibgrapi.2011.41
- 10.1109/JSEN.2019.2956901 · DOI
- vaswani, 2017, Attention is all you need, Proc NIPS, 1
- 10.1109/JSEN.2020.2967054 · DOI
- 10.1109/MC.2018.2381131 · DOI · Paper page
- xu, 2015, Show, attend and tell: Neural image caption generation with visual attention, Proc 32nd Int Conf Mach Learn, 37, 2048
- 10.1007/978-3-319-47121-1_3 · DOI
- chaudhari, 2019, An attentive survey of attention models, arXiv 1904 02874
- 10.3115/v1/W14-4012 · DOI · Paper page
- rokni, 2018, Personalized human activity recognition using convolutional neural networks, Proc AAAI, 1
- chung, 2014, Empirical evaluation of gated recurrent neural networks on sequence modeling, Proc NIPS Workshop Deep Learn, 2
- cooijmans, 2017, Recurrent batch normalization, Proc Int Conf Learn Represent, 1
- hochreiter, 2001, Gradient flow in recurrent nets: The difficulty of learning long-term dependencies, A Field Guide to Dynamical Recurrent Neural Networks
- 10.1109/JSEN.2020.2985374 · DOI · Paper page
- almaslukh, 2017, An effective deep autoencoder approach for online smartphone-based human activity recognition, Int J Comput Sci Netw Secur, 17, 160
- ba, 2016, Layer normalization, arXiv 1607 06450
- 10.1109/JSEN.2020.2964278 · DOI
- bahdanau, 2015, Neural machine translation by jointly learning to align and translate, Proc 3rd Int Conf Learn Represent (ICLR), 265
- 10.1007/978-3-030-33110-8_17 · DOI
- srivastava, 2014, Dropout: A simple way to prevent neural networks from overfitting, J Mach Learn Res, 15, 1929
- 10.1109/IJCNN.2017.7966182 · DOI · Paper page
- 10.1109/JSEN.2020.2978772 · DOI · Paper page
- sathyanarayana, 2016, Impact of physical activity on sleep: A deep learning based exploration, arXiv 1607 07034
- 10.1109/EMBC.2018.8512533 · DOI · Paper page
- siirtola, 2018, Personalizing human activity recognition models using incremental learning, Proc 26th Eur Symp Artif Neural Netw Comput Intell Mach Learn, 1
- 10.1109/78.650093 · DOI · Paper page
Cited by
Latent Independent Excitation for Generalizable Sensor-based Cross-Person Activity Recognition
Qian, H., Pan, S. J., & Miao, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17416 · Cited by 85
“With the auxiliary information representing the target domain, it is possible to fine-tune the models to adapt to the target domain (Buffelli and Vandin 2020; Rokni, Nourollahi, and Ghasemzadeh 2018; Mazankiewicz, B¨ohm, and Berg´es 2020).”
BackgroundMethodMiao, S., Chen, L., Hu, R., & Luo, Y. (2022) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3550331 · Cited by 36
Ahmed, M. E., Yu, H., Vassallo, M., & Koufaki, P. (2025) · Smart Health · doi:10.1016/j.smhl.2025.100555 · Cited by 10
Metadata
- Authors
- Buffelli, D., & Vandin, F.
- Venue
- IEEE Sensors Journal
- Published
- 2021-06-15
- Volume
- 21
- Issue
- 12
- Pages
- 13474-13483
- ISSN
- 1530-437X (ISSN-L)
- 1558-1748
- 2379-9153
- OpenAlex
- W3033787231
- Cited by
- 68
- References
- 24
- FWCI
- 4.23
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- Universit degli Studi di Padova (STARS2017)
- Ministero dellIstruzione dellUniversit e della Ricerca (PRIN Project n. 20174LF3T8 AHeAD, Departments of Excellence (Law 232/2016))
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.