Feature Representation and Data Augmentation for Human Activity Classification Based on Wearable IMU Sensor Data Using a Deep LSTM Neural Network
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Wearable inertial measurement unit (IMU) sensors are powerful enablers for acquisition of motion data. Specifically, in human activity recognition (HAR), IMU sensor data collected from human motion are categorically combined to formulate datasets that can be used for learning human activities. However, successful learning of human activities from motion data involves the design and use of proper feature representations of IMU sensor data and suitable classifiers. Furthermore, the scarcity of labelled data is an impeding factor in the process of understanding the performance capabilities of data-driven learning models. To tackle these challenges, two primary contributions are in this article: first; by using raw IMU sensor data, a spectrogram-based feature extraction approach is proposed. Second, an ensemble of data augmentations in feature space is proposed to take care of the data scarcity problem. Performance tests were conducted on a deep long term short term memory (LSTM) neural network architecture to explore the influence of feature representations and the augmentations on activity recognition accuracy. The proposed feature extraction approach combined with the data augmentation ensemble produces state-of-the-art accuracy results in HAR. A performance evaluation of each augmentation approach is performed to show the influence on classification accuracy. Finally, in addition to using our own dataset, the proposed data augmentation technique is evaluated against the University of California, Irvine (UCI) public online HAR dataset and yields state-of-the-art accuracy results at various learning rates.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


















Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In sensor-based HAR leveraging the temporal nature of IMU signals allows for the application of augmentation methods grounded in signal processing [32,34,99,132,141,178,193].”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [178]
Experimental — accuracy not yet measuredIn the citing paper
“In general, using the dataset augmented with synthetic training samples yields higher classification accuracy when compared to using the original dataset [36,56,261,262].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [261]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 1×
“The synthetic minority over-sampling technique (SMOTE) by [49] is the most popular of these.”
MethodHe, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · cited 1×
“This phenomenon is common in binary classification [19,20] where only two classes are involved.”
BackgroundTwo-dimensional pca: a new approach to appearance-based face representation and recognition
Yang, J., Zhang, D., Frangi, A. F., & Yang, J.-Y. (2004) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2004.1261097 · Cited by 3,563
Activity recognition using cell phone accelerometers
Kwapisz, J. R., Weiss, G. M., & Moore, S. A. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964918 · Cited by 2,912 · cited 2×
“…standard deviation [36,37], root mean square [38,39], zero or mean crossing rate [40], derivative, peak counts [41,42] Human activity recognition [37,38], speech recognition [39], eye movement analysis [42] Frequency-domain Discrete fast Fourier transforms coefficient, spectral energy [7,43]…”
BackgroundThe recognition of human movement using temporal templates
Bobick, A. F., & Davis, J. W. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.910878 · Cited by 2,811 · cited 1×
“Various feature extraction approaches in image processing include using the spatio-temporal interest point (STIP) [27] detector, motion–energy images (MEI) and motion history images (MHI) [28].”
BackgroundOpensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 1×
“Time-domain Mean [35], variance, standard deviation [36,37], root mean square [38,39], zero or mean crossing rate [40], derivative, peak counts [41,42] Human activity recognition [37,38], speech recognition [39], eye movement analysis [42]”
BackgroundLaptev, I. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-1838-7 · Cited by 2,629
(2013) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 1,959
Activity recognition from accelerometer data
Ravi, N., Dandekar, N., Mysore, P., & Littman, M. L. (2005) · Innovative Applications of Artificial Intelligence · Cited by 1,775
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 1×
“At the centre of this technological development is the desire for such miniature devices to be used in applications such as ambient assisted living (AAL) [1], physiological medical diagnostics [2], localization and navigation [3–5], mobile and wireless context-driven decision support systems [6], and security monitori…”
BackgroundCost-sensitive boosting for classification of imbalanced data
Sun, Y., Kamel, M. S., Wong, A. K. C., & Wang, Y. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2007.04.009 · Cited by 1,423
Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
Salamon, J., & Bello, J. P. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2657381 · Cited by 1,364 · cited 1×
“Nevertheless, data augmentation has been successfully applied in wearable sensor data in studies such as [13,14] showing positive state-of-the-art results, and sound recognition [15].”
BackgroundHuman activity recognition with smartphone sensors using deep learning neural networks
Ronao, C. A., & Cho, S.-B. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.04.032 · Cited by 1,215 · cited 4×
“However, studies [8] report that certain activities which produce relatively stationary data are challenging to discriminate amongst.”
BackgroundMethodInfluentialUnderstanding Data Augmentation for Classification: When to Warp?
Wong, S. C., Gatt, A., Stamatescu, V., & McDonnell, M. D. (2016) · 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2016.7797091 · Cited by 1,007
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & De La Torre, F. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.47 · Cited by 790
Experimental perspectives on learning from imbalanced data
Van Hulse, J., Khoshgoftaar, T. M., & Napolitano, A. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273614 · Cited by 786 · cited 1×
“This phenomenon is common in binary classification [19,20] where only two classes are involved.”
BackgroundUm, T. T., Pfister, F. M. J., Pichler, D., Endo, S., Lang, M., Hirche, S., Fietzek, U., & Kulić, D. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136817 · Cited by 729 · cited 2×
“In [14], a sizeable number of data augmentation methods for time series data are mentioned and implemented.”
BackgroundEye Movement Analysis for Activity Recognition Using Electrooculography
Bulling, A., Ward, J. A., Gellersen, H., & Tröster, G. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.86 · Cited by 718
Human fall detection on embedded platform using depth maps and wireless accelerometer
Kwolek, B., & Kepski, M. (2014) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2014.09.005 · Cited by 650 · cited 1×
“In [21], both the Kinect sensor and a wearable motion-sensing device are used to detect falls.”
MethodData Augmentation for Time Series Classification using Convolutional Neural Networks
Guennec, A. L., Malinowski, S., & Tavenard, R. (2016) · HAL · Cited by 604
Um, T. T. (n.d.) · mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) · Cited by 602
Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R., & Havinga, P. (2010) · University of Twente Research Information · Cited by 563
Jimenez, A. R., Seco, F., Prieto, J. C., & Guevara, J. (2010) · 2010 7th Workshop on Positioning, Navigation and Communication · doi:10.1109/wpnc.2010.5649300 · Cited by 469
A high integrity IMU/GPS navigation loop for autonomous land vehicle applications
Sukkarieh, S., Nebot, E. M., & Durrant-Whyte, H. F. (1999) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.768189 · Cited by 439
A deep learning approach to on-node sensor data analytics for mobile or wearable devices
Ravi, D., Wong, C., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2633287 · Cited by 439 · cited 9×
“Frequency-domain Discrete fast Fourier transforms coefficient, spectral energy [7,43] Human activity recognition [7]”
BackgroundMethodInfluentialDeep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning
Guo, X., Singh, S. P., Lee, H., Lewis, R. L., & Wang, X. (2014) · neural information processing systems · Cited by 368
A survey of depth and inertial sensor fusion for human action recognition
Chen, C., Jafari, R., & Kehtarnavaz, N. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-015-3177-1 · Cited by 356 · cited 1×
“Apart from using only inertial sensors for motion analysis, recent studies [26] have considered the fusion of both inertial sensors and camera vision sensors to improve on the human activity recognition accuracy.”
BackgroundAnalyzing features for activity recognition
Huynh, T., & Schiele, B. (2005) · Proceedings of the 2005 joint conference on Smart objects and ambient intelligence: innovative context-aware services: usages and technologies · doi:10.1145/1107548.1107591 · Cited by 339
Yang, J.-Y., Wang, J.-S., & Chen, Y.-P. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.08.002 · Cited by 335 · cited 1×
“Time-domain Mean [35], variance, standard deviation [36,37], root mean square [38,39], zero or mean crossing rate [40], derivative, peak counts [41,42] Human activity recognition [37,38], speech recognition [39], eye movement analysis [42]”
BackgroundSmartphones and Health Promotion: A Review of the Evidence
Bert, F., Giacometti, M., Gualano, M. R., & Siliquini, R. (2014) · Journal of Medical Systems · doi:10.1007/s10916-013-9995-7 · Cited by 319 · cited 1×
“In order to achieve the latter, wearable devices such as smart watches or smartphones are equipped with inertial sensors to generate data and user analysis software applications for feedback purposes [32].”
BackgroundMonitoring Activities of Daily Living in Smart Homes: Understanding human behavior
Debes, C., Merentitis, A., Sukhanov, S., Niessen, M., Frangiadakis, N., & Bauer, A. (2016) · IEEE Signal Processing Magazine · doi:10.1109/msp.2015.2503881 · Cited by 301 · cited 5×
“Fall detection in elderly people [21–23] is seen to be the most popular and attractive application in AAL studies.”
BackgroundInfluentialA High Reliability Wearable Device for Elderly Fall Detection
Pierleoni, P., Belli, A., Palma, L., Pellegrini, M., Pernini, L., & Valenti, S. (2015) · IEEE Sensors Journal · doi:10.1109/jsen.2015.2423562 · Cited by 276 · cited 1×
“[23] proposed a fall detection algorithm which was implemented in a wearable device.”
MethodActivity Recognition and Monitoring Using Multiple Sensors on Different Body Positions
Maurer, U., Smailagic, A., Siewiorek, D. P., & Deisher, M. (2006) · doi:10.21236/ada534437 · Cited by 266
Feature Selection and Activity Recognition from Wearable Sensors
Pirttikangas, S., Fujinami, K., & Nakajima, T. (2006) · Lecture Notes in Computer Science · doi:10.1007/11890348_39 · Cited by 229 · cited 1×
“Time-domain Mean [35], variance, standard deviation [36,37], root mean square [38,39], zero or mean crossing rate [40], derivative, peak counts [41,42] Human activity recognition [37,38], speech recognition [39], eye movement analysis [42]”
BackgroundNon-GPS navigation for security personnel and first responders
Ojeda, L., & Borenstein, J. (2007) · Journal of Navigation · doi:10.1017/s0373463307004286 · Cited by 226
The Use of Deep Learning to Predict Stroke Patient Mortality
Cheon, S.-H., Kim, J., & Lim, J. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16111876 · Cited by 164
Quantitative Assessment of Parkinsonian Tremor Based on an Inertial Measurement Unit
Dai, H., Zhang, P., & Lueth, T. (2015) · Sensors · doi:10.3390/s151025055 · Cited by 159 · cited 2×
“In [24], a quantitative assessment of Parkinson’s tremor is conducted using least-square-estimation models.”
BackgroundMethodInertial Sensor-Based Robust Gait Analysis in Non-Hospital Settings for Neurological Disorders
Tunca, C., Pehlivan, N., Ak, N., Arnrich, B., Salur, G., & Ersoy, C. (2017) · Sensors · doi:10.3390/s17040825 · Cited by 142
Accelerometry Based Classification of Walking Patterns Using Time-frequency Analysis
Wang, N., Ambikairajah, E., Lovell, N. H., & Celler, B. G. (2007) · 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society · doi:10.1109/iembs.2007.4353438 · Cited by 117 · cited 1×
“Time frequency domain Wavelet coefficients [44] Blink detection [42]”
BackgroundHighly Portable, Sensor-Based System for Human Fall Monitoring
Mao, A., Ma, X., He, Y., & Luo, J. (2017) · Sensors · doi:10.3390/s17092096 · Cited by 98 · cited 1×
“Mao et al. [22] proposed a fall detection method based on the acceleration and Euler angle data extracted from a wearable micro electro-mechanical system (MEMS) sensor to represent the orientation of the users’ body using Kalman solutions.”
MethodFeature extraction using PCA and Kernel-PCA for face recognition
Ebied, H. M. (2012) · International Conference on Informatics and Systems · Cited by 86
Multilayer Joint Gait-Pose Manifolds for Human Gait Motion Modeling
Ding, M., & Fan, G. (2015) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2373393 · Cited by 60 · cited 1×
“Human gait modeling studies [29] have also come up with descriptive motion models which can be used to aid the recognition of persons and activities.”
BackgroundGesture spotting using wrist worn microphone and 3-axis accelerometer
Ward, J. A., Lukowicz, P., & Tröster, G. (2005) · doi:10.1145/1107548.1107578 · Cited by 56 · cited 1×
“Time-domain Mean [35], variance, standard deviation [36,37], root mean square [38,39], zero or mean crossing rate [40], derivative, peak counts [41,42] Human activity recognition [37,38], speech recognition [39], eye movement analysis [42]”
BackgroundMatsuzaka, Y., & Uesawa, Y. (2019) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00065 · Cited by 49
Wearable Inertial Sensors: Applications, Challenges, and Public Test Benches
Bruno, B., Mastrogiovanni, F., & Sgorbissa, A. (2015) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2015.2448279 · Cited by 40 · cited 1×
“Fitness-oriented applications focus on this need by being able to automatically identify human activities, log them and do statistical analyses for body fitness assessments [31].”
BackgroundClassifying Diverse Physical Activities Using “Smart Garments”
Esfahani, M. I. M., & Nussbaum, M. A. (2019) · Sensors · doi:10.3390/s19143133 · Cited by 37
Muñoz-Organero, M., Powell, L., Heller, B., Harpin, V., & Parker, J. (2019) · Sensors · doi:10.3390/s19132935 · Cited by 36
Human motion based intent recognition using a deep dynamic neural model
Yu, Z., & Lee, M. (2015) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2015.01.001 · Cited by 33 · cited 1×
“The possibility of understanding human intention using motion data has also been studied in [25] as a build-up to achieve AAL.”
BackgroundXu, J. Y., Chang, H.-I., Chien, C., Kaiser, W. J., & Pottie, G. J. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2282812 · Cited by 13 · cited 1×
“At the centre of this technological development is the desire for such miniature devices to be used in applications such as ambient assisted living (AAL) [1], physiological medical diagnostics [2], localization and navigation [3–5], mobile and wireless context-driven decision support systems [6], and security monitori…”
Background
Research works printed but not linked: 5
- Eyobu, O.S., Kim, Y., and Han, D.S. (2017, January 21–23). Activity Recognition for Infant Health Care Based on Wireless Inertial Measurement Unit Motion Data. Proceedings of the Korean Institute of Communication Sciences Conference, Jeju, Korea.
- Ohashi, H., Al-Nasser, M., Ahmed, S., Akiyama, T., Sato, T., Nguyen, P., Nakamura, K., and Dengel, A. (2017, January 6–11). Augmenting Wearable Sensor Data with Physical Constraint for DNN-Based Human-Action Recognition. Presented at ICML 2017 Times Series Workshop, Sydney, Australia.
- Bächlin, M., Förster, K., and Tröster, G. (October, January 30). SwimMaster: A wearable assistant for swimmer. Proceedings of the 11th International Conference on Ubiquitous Computing, Orlando, FL, USA.
- Cui, Z., Chen, W., and Chen, Y. (arXiv, 2016). Multi-scale convolutional neural networks for time series classification, arXiv.
- DeVries, T., and Taylor, G.W. (arXiv, 2017). Dataset Augmentation in Feature Space, arXiv.
Cited by
Time Series Data Augmentation for Deep Learning: A Survey
Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/631 · Cited by 727
An empirical survey of data augmentation for time series classification with neural networks
Iwana, B. K., & Uchida, S. (2021) · PLOS ONE · doi:10.1371/journal.pone.0254841 · Cited by 725
“For most datasets, this is not a problem, but ones with long time series, such as HandOutlines with 2,709 time steps, can take extraordinary amounts of time compared to the transformation-based methods.”
MethodInfluentialQiu, S., Zhao, H., Jiang, N., Wang, Z., Liu, L., An, Y., Zhao, H., Miao, X., Liu, R., & Fortino, G. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.006 · Cited by 671
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625
A CNN-LSTM Approach to Human Activity Recognition
Mutegeki, R., & Han, D. S. (2020) · 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic48513.2020.9065078 · Cited by 452
“Eyobu and Han [8] employed an LSTM network to classify a dataset with frequency domain features extracted, as opposed to the raw signal which differs from our approach.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Steven Eyobu, O., & Han, D. S.
- Affiliations
- School of Electronics Engineering, Kyungpook National University, 80 Daehak-ro, Buk-gu, Daegu 41566, Korea
- Kyungpook National University (KR)
- Corresponding author
- Han, Dong Seog
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2018-08-31
- Volume
- 18
- Issue
- 9
- Pages
- 2892
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC6165524
- OpenAlex
- W2889505988
- Cited by
- 259
- References
- 50
- Citations by year
- 2026: 15 · 2025: 27 · 2024: 36 · 2023: 45 · 2022: 32 · 2021: 55 · 2020: 31 · 2019: 17 · 2018: 1
- FWCI
- 13.01
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Institute for Information and communications Technology Promotion (R7124-16- 0004)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Keywords
- human activity recognition, inertial measurement unit, spectrogram feature extraction, data augmentation, feature representation, LSTM neural network, labeled data scarcity, UCI HAR dataset
- MeSH terms
- Wearable Electronic Devices, Deep Learning, Humans, Human Activities, Movement, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.