COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 77 references it cites were found in a registry or an index, and 2 are not checked yet; 11 of the 26 sources its claims cite were checked for support.
Abstract
The goal of creating intelligent, human-centered wearable systems for continuous activity understanding faces a fundamental trade-off: Egocentric video-based models capture rich semantic information and have demonstrated strong performance in human activity recognition (HAR), but their high power consumption, privacy concerns, and dependence on lighting limit their feasibility for continuous on-device recognition. In contrast, inertial measurement unit (IMU) sensors offer an energy-efficient, privacy-preserving alternative, yet lack large-scale annotated datasets, leading to weaker generalization. To bridge this gap, we propose COMODO, a cross-modal self-supervised distillation framework that transfers semantic knowledge from video to IMU without requiring labels. COMODO leverages a pretrained and frozen video encoder to construct a dynamic instance queue to align the feature distributions of video and IMU embeddings. This enables the IMU encoder to inherit rich semantic structure from video while maintaining its efficiency for real-world applications. Experiments on multiple egocentric HAR datasets show that COMODO consistently improves downstream performance, matching or surpassing fully supervised models, and demonstrating strong cross-dataset generalization. Benefiting from its simplicity and flexibility, COMODO is compatible with diverse pretrained video and time-series models, offering the potential to leverage more powerful teacher and student foundation models in future ubiquitous computing research. The code is available at this repository: https://github.com/cruiseresearchgroup/COMODO.
Verified 2026-11-20 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
COMODO is a cross-modal self-supervised distillation framework that transfers semantic knowledge from a frozen pretrained video encoder to a lightweight IMU encoder via a dynamic FIFO queue and similarity-distribution alignment, without IMU labels. Across three egocentric HAR datasets, it matches or surpasses fully supervised baselines and improves cross-dataset generalization while retaining IMU inference efficiency.
Key findings
- On IMU-based HAR, COMODO achieved A@1 of 59.13 on Ego4D, 84.92 on EgoExo4D, and 92.48 on MMEA, outperforming IMU2CLIP by 4.3%, 5.46%, and 1.07% A@1, respectively.
- In cross-dataset transfer, COMODO improved over IMU2CLIP, CrossHAR, and supervised fine-tuned Mantis in all cases; e.g., EgoExo4D→Ego4D A@1 was 56.62 vs. 51.75 for IMU2CLIP and 52.58 for Mantis†, and Ego4D→EgoExo4D A@1 was 82.54.
- Efficiency: COMODO (8.1M params) required 1.70G FLOPs and 75.7ms latency for a 5-second window, versus TimeSformer-Base-k400 (121.4M params) at 196.05G FLOPs and 3590.3ms, giving 47× speedup and 115× FLOPs reduction while retaining 78.8% accuracy vs. 89.8%.
- Ablations showed concat pooling outperformed mean pooling (e.g., MMEA A@1 +6.84%), FIFO sampling outperformed random sampling, and optimal queue size was dataset-specific (Ego4D 16384, EgoExo4D 1024, MMEA 2048); COMODO also outperformed L2 and InfoNCE baselines.
Methods
- Design: a frozen pretrained video teacher (default TimeSformer-base; VideoMAE-Base-k400 also tested) and a trainable IMU student (default Mantis; MOMENT-small also tested) are projected to L2-normalized embeddings; a FIFO queue stores video embeddings, and the IMU encoder is trained by minimizing cross-entropy between the softmax similarity distributions of video and IMU embeddings, equivalent to KL alignment.
- Data: three egocentric HAR datasets—Ego4D (31 activities; 253K train pairs), EgoExo4D (8 activities; 109K train pairs), and MMEA (32 activities; 4553 train pairs). Ego4D/EgoExo4D were preprocessed to 200 Hz IMU, 5-second non-overlapping windows, and 224×224 video at 10 FPS; MMEA used 25 Hz IMU and official splits.
- Analysis: trained for 20 epochs with batch size 128, learning rate 3e-4, τ_v=0.1, τ_x=0.05, on a single NVIDIA L40S GPU with seed 42; evaluated by training an RBF-kernel SVM on frozen IMU features and reporting Accuracy@1/@3/@5 against supervised and self-supervised baselines, including cross-dataset transfer.
Limitations
- Stated in the paper: COMODO relies on synchronized video-IMU data pairs during training; precise temporal alignment between heterogeneous modalities remains challenging, though similarity-distribution alignment mitigates it; and effectiveness is tied to the representational capacity of the pretrained video teacher.
- Stated in the paper: the current framework uses one-way knowledge transfer from a frozen teacher rather than mutual distillation, and does not explicitly consider personalization or contextual adaptation.
- Evident from the text: downstream evaluation still uses labels to train an SVM on extracted features, so the full pipeline is not label-free; cross-dataset generalization is tested only across the three egocentric datasets, and standalone IMU datasets cannot be used without paired video.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 2 Not valid · 6 Not checked · 141 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“while IMU2CLIP reports results on Ego4D, the IMU2CLIP paper does not provide official benchmark splits.”
What the sources say
- Partly supportedcites [40] [40] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. 2023. IMU2CLIP: Languagegrounded Motion Sensor Translation with Multimodal Contrastive Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 13246-13253. - sourceExperimental — accuracy not yet measured
The Results section reports Ego4D test-set results. The Methods section says the authors split the data into training, validation, and test sets by video ID, and Table 1 gives split sizes. Our copy does not say whether those splits were provided as official benchmark splits, so it supports the results portion but does not establish the claim about their absence. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Aligning heterogeneous video and IMU signals with different temporal resolutions and feature spaces [12].”
What the sources say
- Partly supportedcites [12] [12] Sourish Gunesh Dhekane and Thomas Ploetz. 2025. Transfer Learning in Sensor-Based Human Activity Recognition: A Survey. ACM Comput. Surv. 57, 8, Article 205 (March 2025), 39 pages. https://doi.org/10.1145/3717608 - sourceExperimental — accuracy not yet measured
Our copy supports the heterogeneity and alignment parts: §4.1 (p. 13) defines heterogeneity as differing feature spaces (𝒳_s ≠ 𝒳_t) and states that m-heterogeneity 'typically occurs when other data modalities like RGB videos ... are used to bolster the IMU-based transfer,' and it describes aligning the two (multi-input-multi-output mapping via QR factorization [22]; LSTM-based projection onto a common feature space [135]). However, no passage attributes a difference in TEMPORAL RESOLUTION specifically to video–IMU pairs; 'sensing frequency' appears only as one generic factor among wearable-sensor heterogeneity sources (§4, p. 12; §4.1, p. 13), tied to device/d-heterogeneity rather than to the video-versus-IMU comparison. So the temporal-resolution element of the citing sentence is not supported by this source. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Designing objectives and optimization to reliably transfer semantic superior video domain knowledge to IMU domain, which has a significant modality and performance gap [19].”
What the sources say
- Supportedcites [19] [19] Xinyu Gong, Sreyas Mohan, Naina Dhingra, Jean-Charles Bazin, Yilei Li, Zhangyang Wang, and Rakesh Ranjan. 2023. Mmg-ego4d: Multimodal generalization in egocentric action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6481-6491. - source
Table 3 and its caption (section 'class 1 class 2 class 3') give the unimodal gap directly: video (MViT-B) 58.89 few-shot / 52.40 Top-1 versus IMU Transformer 40.07 / 29.78, with the caption stating 'Video achieves the best performance, while also consuming more computational resources.' §6.2 (supervised results) adds that dropping video at evaluation costs ~33% accuracy, 'suggesting that the video modality is more informative than the other two modalities', and §4.2/§6.1 frames video-to-IMU transfer as hard because 'the support and query modalities are disjoint' — the modality gap; Table 1 quantifies the modality disparity (video 60x storage, 43x compute vs IMU). The cited clause is therefore supported by our copy; the surrounding framing about designing objectives is the citing authors' own argument. (LLM, conf=0.85)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Continuous wearable video is often untenable due to power consumption and privacy concerns [4, 36, 41, 43].”
What the sources say
- Not checked — no full text of the sourcecites [4] [4] Divyanshu Bhardwaj, Alexander Ponticello, Shreya Tomar, Adrian Dabrowski, and Katharina Krombholz. 2024. In Focus, Out of Privacy: The Wearer's Perspective on the Privacy Dilemma of Camera Glasses. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 577, 18 pages. doi:10.1145/3613904.3642242 - source
- Not checked — no full text of the sourcecites [36] [36] Si Liao, Hanwei He, Huangxun Chen, and Zhice Yang. 2025. Bystander Privacy in Video Sharing Era: Automated Consent Compliance through Platform Censorship. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 902, 16 pages. doi:10.1145/3706598.3713391 - source
- Partly supportedcites [41] [41] Tamara Mujirishvili, Anton Fedosov, Kooshan Hashemifard, Pau Climent-Pérez, and Francisco Florez-Revuelta. 2024. 'I Don't Want to Become a Number': Examining Different Stakeholder Perspectives on a Video-Based Monitoring System for Senior Care with Inherent Privacy Protection (by Design).. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 774, 19 pages. doi:10.1145/3613904.3642164 - sourceExperimental — accuracy not yet measured
Section 2.3 (page unavailable) says cameras in private places raise privacy concerns that hinder their acceptability. It does not say that continuous wearable video is often untenable, or identify power consumption as a reason; its discussion of power efficiency in Section 3.0.1 concerns the hardware chosen for a senior-care monitoring system. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [43] [43] Octav Opaschi and Radu-Daniel Vatavu. 2020. Uncovering Practical Security and Privacy Threats for Connected Glasses with Embedded Video Cameras. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 167 (Dec. 2020), 26 pages. doi:10.1145/3432700 - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Figure 6, COMODO achieves a superior balance between recognition accuracy and computational cost.”
plainstates an associationn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
88 more claims
Create a free account to see all 157 claims and to verify the 4 references still open.
- Probably not a substantial claim. 31
- It may state two separate claims as one. 44
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 Ablation Study · p. 15
“The results in Table 5 reveal that COMODO consistently and substantially outperforms the IMU2CLIP across every teacher-student configuration on all datasets.”
§ 4.6 Ablation Study · p. 16
“| MOMENT 𝑆 | VideoMAE 𝐵 | IMU2CLIP | 41.39 | 65.82 | 76.45 | 69.77 | 94.61 | 98.81 | 78.42 | 91.19 | 94.60 | | | VideoMAE 𝐵 | COMODO | 57.74 | 76.66 | 83.79 | 81.60 | 97.68 | 99.48 | 79.48 | 91.19 | 94.45 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Baseline Implementation Detail · p. 28
“| CrossHAR | 512 | 8 4.97 | 96.55 | 98.11 |”
§ 4.2 Experimental Settings · p. 11
“| CrossHAR | ✗ | 50.96 | 72.66 | 81.53 | 77.14 | 96.95 | 99.31 | 84.97 | 96.55 | 97.74 |”
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · cited 2×
“[43] shows that multimodal students can out-perform unimodal teachers by learning richer cross-modal representations.”
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Zhou, H., Zhang, S.-H., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2020) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2012.07436 · Cited by 9,681
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17325 · Cited by 7,752
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“In this way, our representation learning distillation framework remains unsupervised, with class labels only used to train the SVM on the extracted features.”
Are Transformers Effective for Time Series Forecasting?
Zeng, A., Chen, M.-H., Zhang, L., et al. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2205.13504 · Cited by 4,870
Are Transformers Effective for Time Series Forecasting?
Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26317 · Cited by 3,320
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Cited by
RightFeatKD: Selective Feature-Based Knowledge Distillation
Haque, S. T., Yan, Y., & Ngu, A. H. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31927-2_43
Metadata
- Authors
- Chen, B., Wongso, W., Li, Z., Khaokaew, Y., Xue, H., & Salim, F. D.
- Affiliations
- UNSW Sydney (AU)
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2026-06-15
- Volume
- 10
- Issue
- 2
- Pages
- 1-29
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W4414565371
- Semantic Scholar
- c1b81ba2d90f8ce02b66263c5af3fbe23ae65cdb
- Cited by
- 1
- References
- 63
- Citations by year
- 2026: 1
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- ARC Centre of Excellence for Automated Decision-Making and Society (CE200100005)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- inertial measurement unit, cross-modal knowledge distillation, self-supervised learning, wearable sensors, energy efficiency, privacy preservation, cross-dataset generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.