Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 26Context-Aware Activity Recognition SystemsHuman Pose and Action RecognitionGait Recognition and Analysis

The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; 1 of the 12 sources its claims cite were checked for support.

A tiny inertial transformer for human activity recognition via multimodal knowledge distillation and explainable AI

Open access
Lamaakal, Ismail, Yahyati, Chaymae, Maleh, Yassine, El Makkaoui, Khalid, Ouahbi, Ibrahim, El-Latif, Ahmed A. Abd, Zomorodi, Mariam, El-Rahiem, Basma Abd·2025·Scientific Reports

Published version of a preprint

Abstract

Human activity recognition (HAR) is essential for applications such as healthcare monitoring, fitness tracking, and smart environments, yet deploying accurate and interpretable models on resource-constrained devices remains challenging. In this paper, we propose XTinyHAR, a lightweight, transformer-based unimodal framework trained via cross-modal knowledge distillation from a multimodal teacher. Our model incorporates temporal positional embeddings and attention rollout to enhance sequential feature extraction and interpretability. Evaluated on UTD-MHAD and MM-Fit, XTinyHAR achieves test accuracies of 98.71% and 98.55% with F1-scores that match these results and Cohen's Kappa above 0.98. The model remains lightweight (2.45 MB) with fast inference (3.1 ms on CPU, 1.2 ms on GPU) and low computational cost (11.3M FLOPs). Extensive ablation studies confirm the contribution of each component, and subject-wise evaluations demonstrate strong generalization across users. These results highlight XTinyHAR's potential as a high-performance, interpretable, and deployable solution for real-time HAR on edge devices. Our codes are available at: https://github.com/Ism-ail11/XTinyHAR.

Overview

CoChat Summary

## TL;DR

XTinyHAR is a lightweight inertial-only transformer for human activity recognition trained by distilling knowledge from a multimodal skeleton+inertial teacher. On UTD-MHAD and MM-Fit, it reports 98.71% and 98.55% test accuracy, respectively, while staying small and fast enough for edge deployment.

## Key findings

  • XTinyHAR achieves test accuracies of 98.71% on UTD-MHAD and 98.55% on MM-Fit; the abstract also reports F1-scores that match these results and Cohen’s Kappa above 0.98.
  • The model is reported as 2.45 MB with 11.3M FLOPs, and inference time is 3.1 ms on CPU and 1.2 ms on GPU.
  • In an ablation on UTD-MHAD, removing knowledge distillation reduces accuracy from 98.71% to 96.84%; removing positional embeddings reduces it to 97.42%; removing attention rollout reduces it to 98.11%.
  • Distilling from a multimodal teacher outperforms a unimodal teacher: student accuracy improves by 1.5–2.4% absolute, with 98.71% vs 96.98% on UTD-MHAD and 98.55% vs 96.13% on MM-Fit.
  • Subject-wise evaluation reports accuracies ranging from 97.8% to 99.2% on UTD-MHAD and 96.8% to 98.5% on MM-Fit, with standard deviations of 0.54% and 0.52%, respectively.

## Methods

  • The paper proposes XTinyHAR, an inertial-only student transformer trained via offline multimodal-to-unimodal knowledge distillation from a multimodal teacher (ST-ConvT) that uses skeleton and inertial streams.
  • Experiments are run on UTD-MHAD (8 subjects, 27 actions, 861 labeled samples) and MM-Fit (more than 70 participants, 12 exercises), using subject-independent evaluation; UTD-MHAD uses leave-one-subject-out, and MM-Fit uses a fixed subject-wise split.
  • Inputs are segmented into 3-second windows with 50% overlap, inertial signals are resampled to 50 Hz and z-score normalized, and the student uses a patch-based Transformer with L = 2 encoder layers, D = 128, and P = 20.
  • Main analysis includes classification metrics, ablation studies, subject-wise testing, noise/missing-data robustness, edge-device deployment tests on Raspberry Pi 4B and Jetson Nano, and XAI methods: attention analysis, Integrated Gradients, attention rollout, and attention similarity.

## Limitations

  • The paper states that UTD-MHAD and MM-Fit lack diversity in age and physical ability; in MM-Fit, subjects aged 60+ show a 3.2% accuracy drop.
  • The model is designed for fixed wrist-mounted IMUs; simulated sensor-placement shifts in MM-Fit cause performance degradation of up to 5.6%.
  • The paper notes a lack of personalization mechanisms for individual sensor variation and routines.
  • The text makes evident, and the paper states, that no cross-dataset generalization experiments were conducted (for example, training on UTD-MHAD and testing on MM-Fit).
  • The paper also states that true deployment on ultra-constrained platforms such as Cortex-M microcontrollers was not covered.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41598-025-26297-2

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 An illustration of HAR process using conventional approaches.
Fig. 1 An illustration of HAR process using conventional approaches.CC BY-NC-ND · PubMed Central
Fig. 2 Overview of the proposed Model XTinyHAR. A lightweight inertial transformer (right) is trained via knowledge distillation from a multimodal spatio-temporal ConvTransformer (left), which integrates skeleton and inertial data. The student model learns to replicate the teacher’s predictions using only inertial input.
Fig. 2 Overview of the proposed Model XTinyHAR. A lightweight inertial transformer (right) is trained via knowledge distillation from a multimodal spatio-temporal ConvTransformer (left), which integrates skeleton and inertial data. The student model learns to replicate the teacher’s predictions using only inertial input.CC BY-NC-ND · PubMed Central
Fig. 3 Attention distribution over time for the UTD-MHAD dataset. The student model shows a strong temporal focus that aligns with the teacher, reflecting successful knowledge transfer from multimodal to unimodal input.
Fig. 3 Attention distribution over time for the UTD-MHAD dataset. The student model shows a strong temporal focus that aligns with the teacher, reflecting successful knowledge transfer from multimodal to unimodal input.CC BY-NC-ND · PubMed Central
Fig. 4 Attention distribution over time for the MM-Fit dataset. Despite more complex and varied motions, the student still effectively mimics the teacher’s attention focus.
Fig. 4 Attention distribution over time for the MM-Fit dataset. Despite more complex and varied motions, the student still effectively mimics the teacher’s attention focus.CC BY-NC-ND · PubMed Central
Fig. 5 Integrated gradients heatmap for a sample from UTD-MHAD. The X-axis represents temporal patches (T0–T9), and the Y-axis denotes inertial signal channels. Each cell indicates the relevance score for the corresponding modality and time step in contributing to the model’s decision.
Fig. 5 Integrated gradients heatmap for a sample from UTD-MHAD. The X-axis represents temporal patches (T0–T9), and the Y-axis denotes inertial signal channels. Each cell indicates the relevance score for the corresponding modality and time step in contributing to the model’s decision.CC BY-NC-ND · PubMed Central
Fig. 6 Integrated gradients attribution heatmap for MM-Fit. Highlighted regions correspond to complex fitness motions with strong periodicity. The student model relies on subtle sensor variations that align well with physical movement cycles.
Fig. 6 Integrated gradients attribution heatmap for MM-Fit. Highlighted regions correspond to complex fitness motions with strong periodicity. The student model relies on subtle sensor variations that align well with physical movement cycles.CC BY-NC-ND · PubMed Central
Fig. 7 Attention rollout matrix for UTD-MHAD. Attention becomes increasingly concentrated on semantically meaningful segments across Transformer layers, illustrating a refined temporal abstraction.
Fig. 7 Attention rollout matrix for UTD-MHAD. Attention becomes increasingly concentrated on semantically meaningful segments across Transformer layers, illustrating a refined temporal abstraction.CC BY-NC-ND · PubMed Central
Fig. 8 Attention rollout for MM-Fit. The model develops focus on critical movement repetitions through the network depth, showing a learned sense of action structure.
Fig. 8 Attention rollout for MM-Fit. The model develops focus on critical movement repetitions through the network depth, showing a learned sense of action structure.CC BY-NC-ND · PubMed Central
Fig. 9 Cosine similarity of attention distributions between teacher and student on UTD-MHAD. Strong correspondence in temporal focus indicates successful internalization of teacher reasoning.
Fig. 9 Cosine similarity of attention distributions between teacher and student on UTD-MHAD. Strong correspondence in temporal focus indicates successful internalization of teacher reasoning.CC BY-NC-ND · PubMed Central
Fig. 10 Attention similarity for MM-Fit. Despite the student using only inertial data, it replicates the teacher’s multimodal attention distribution in key segments.
Fig. 10 Attention similarity for MM-Fit. Despite the student using only inertial data, it replicates the teacher’s multimodal attention distribution in key segments.CC BY-NC-ND · PubMed Central
Fig. 11 Training, validation accuracy, and loss curves for UTD-MHAD and MM-Fit datasets.
Fig. 11 Training, validation accuracy, and loss curves for UTD-MHAD and MM-Fit datasets.CC BY-NC-ND · PubMed Central
Fig. 12 Confusion matrix produced by the proposed XTinyHAR model on the UTD-MHAD dataset. Each row represents the true activity class, and each column corresponds to the predicted class. The strong diagonal dominance reflects the model’s high classification accuracy and low misclassification rate across 27 activity classes.
Fig. 12 Confusion matrix produced by the proposed XTinyHAR model on the UTD-MHAD dataset. Each row represents the true activity class, and each column corresponds to the predicted class. The strong diagonal dominance reflects the model’s high classification accuracy and low misclassification rate across 27 activity classes.CC BY-NC-ND · PubMed Central
Fig. 13 Confusion matrix produced by the proposed model on the MM-Fit dataset. The matrix reveals strong diagonal dominance, indicating high per-class classification accuracy. Misclassifications are minimal and occur between classes with similar movement patterns.
Fig. 13 Confusion matrix produced by the proposed model on the MM-Fit dataset. The matrix reveals strong diagonal dominance, indicating high per-class classification accuracy. Misclassifications are minimal and occur between classes with similar movement patterns.CC BY-NC-ND · PubMed Central
Fig. 14 Comparison of the proposed model’s performance across the UTD-MHAD and MM-Fit datasets.
Fig. 14 Comparison of the proposed model’s performance across the UTD-MHAD and MM-Fit datasets.CC BY-NC-ND · PubMed Central
Fig. 15 Accuracy comparison between unimodal and multimodal teacher distillation on two datasets.
Fig. 15 Accuracy comparison between unimodal and multimodal teacher distillation on two datasets.CC BY-NC-ND · PubMed Central
Fig. 16 Impact of temperature T and distillation weight \alpha on XTinyHAR accuracy.
Fig. 16 Impact of temperature T and distillation weight \alpha on XTinyHAR accuracy.CC BY-NC-ND · PubMed Central
Fig. 17 Impact of Gaussian noise on model accuracy. XTinyHAR shows robust performance even under \sigma =0.2 perturbation.
Fig. 17 Impact of Gaussian noise on model accuracy. XTinyHAR shows robust performance even under \sigma =0.2 perturbation.CC BY-NC-ND · PubMed Central
Fig. 18 Comparison of XTinyHAR performance under various perturbation types. Missing data leads to only minor degradation.
Fig. 18 Comparison of XTinyHAR performance under various perturbation types. Missing data leads to only minor degradation.CC BY-NC-ND · PubMed Central
Fig. 19 Energy consumption per inference for XTinyHAR model on CPU and GPU platforms. Results demonstrate the model’s ultra-low energy demand, supporting sustainable on-device HAR.
Fig. 19 Energy consumption per inference for XTinyHAR model on CPU and GPU platforms. Results demonstrate the model’s ultra-low energy demand, supporting sustainable on-device HAR.CC BY-NC-ND · PubMed Central
Fig. 20 XTinyHAR subject-wise accuracy on the UTD-MHAD dataset.
Fig. 20 XTinyHAR subject-wise accuracy on the UTD-MHAD dataset.CC BY-NC-ND · PubMed Central
Fig. 21 XTinyHAR subject-wise accuracy on the MM-Fit dataset.
Fig. 21 XTinyHAR subject-wise accuracy on the MM-Fit dataset.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

1 Valid · 6 Not valid · 11 Not checked · 199 Own results and interpretations

104 more claims

Create a free account to see all 217 claims and to verify the 11 references still open.

See all claims — free account
  • It may state two separate claims as one. 73
  • Probably not a substantial claim. 44

Create a free account to see these claims.

Citations

Cites

Show all 49 references

Research works printed but not linked: 3

  1. Bastola, A., Luitel, N., Wang, H., Paudel, D. P., Poudel, R. & Razi, A. RobustFormer: Noise-Robust Pre-Training for Images and Videos. arXiv preprint arXiv:2411.13040 (2024). · arXiv
  2. Zhu, J. Comparative study of sequence-to-sequence models: From RNNs to transformers. Appl. Comput. Eng. 42(67), 2755–2721 (2023).
  3. Lin, J. et al. MCUNet: Tiny deep learning on IoT devices. Adv. Neural Inf. Process. Syst. (NeurIPS) 33, 11711–11722 (2020).

Cited by

See all 26

Metadata

Authors
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A.
Affiliations
  • Mohamed I University (MA)
  • Université Sultan Moulay Slimane (MA)
  • Menoufia University (EG)
Corresponding author
  • Maleh, Yassine
  • El-Rahiem, Basma Abd
Venue
Scientific Reports
Published
2025-11-27
Volume
15
Issue
1
Pages
42335-42335
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4416740529
Cited by
26
References
49
Citations by year
2026: 25 · 2025: 1
FWCI
9.08
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • Research Square (acceptedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Research, Development, and Innovation Authority (13325-psu-2023-PSNU-R-3-1-EF)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
Keywords
human activity recognition, inertial sensors, lightweight transformer, cross-modal knowledge distillation, explainable AI, attention rollout, edge deployment, resource-constrained devices, real-time activity recognition
MeSH terms
Algorithms, Artificial Intelligence, Humans, Human Activities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A tiny inertial transformer for human activity recognition via multimodal knowledge distillation and explainable AI. https://app.cochat.ai/papers/10.1038/s41598-025-26297-2
The paper
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A. (2025). A tiny inertial transformer for human activity recognition via multimodal knowledge distillation and explainable AI. Scientific Reports. https://doi.org/10.1038/s41598-025-26297-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.