COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition
Open accessPublished version of a preprint
The DOI matches its Crossref record; 62 of the 77 references it cites were found in a registry or an index, and 2 are not checked yet; 9 of the 26 sources its claims cite were checked for support.
Abstract
The goal of creating intelligent, human-centered wearable systems for continuous activity understanding faces a fundamental trade-off: Egocentric video-based models capture rich semantic information and have demonstrated strong performance in human activity recognition (HAR), but their high power consumption, privacy concerns, and dependence on lighting limit their feasibility for continuous on-device recognition. In contrast, inertial measurement unit (IMU) sensors offer an energy-efficient, privacy-preserving alternative, yet lack large-scale annotated datasets, leading to weaker generalization. To bridge this gap, we propose COMODO, a cross-modal self-supervised distillation framework that transfers semantic knowledge from video to IMU without requiring labels. COMODO leverages a pretrained and frozen video encoder to construct a dynamic instance queue to align the feature distributions of video and IMU embeddings. This enables the IMU encoder to inherit rich semantic structure from video while maintaining its efficiency for real-world applications. Experiments on multiple egocentric HAR datasets show that COMODO consistently improves downstream performance, matching or surpassing fully supervised models, and demonstrating strong cross-dataset generalization. Benefiting from its simplicity and flexibility, COMODO is compatible with diverse pretrained video and time-series models, offering the potential to leverage more powerful teacher and student foundation models in future ubiquitous computing research. The code is available at this repository: https://github.com/cruiseresearchgroup/COMODO.
Verified 2026-10-10 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
COMODO is a cross-modal self-supervised distillation framework that transfers semantic knowledge from a frozen pretrained video encoder to a lightweight IMU encoder via a dynamic FIFO queue and similarity-distribution alignment, without IMU labels. Across three egocentric HAR datasets, it matches or surpasses fully supervised baselines and improves cross-dataset generalization while retaining IMU inference efficiency.
Key findings
- On IMU-based HAR, COMODO achieved A@1 of 59.13 on Ego4D, 84.92 on EgoExo4D, and 92.48 on MMEA, outperforming IMU2CLIP by 4.3%, 5.46%, and 1.07% A@1, respectively.
- In cross-dataset transfer, COMODO improved over IMU2CLIP, CrossHAR, and supervised fine-tuned Mantis in all cases; e.g., EgoExo4D→Ego4D A@1 was 56.62 vs. 51.75 for IMU2CLIP and 52.58 for Mantis†, and Ego4D→EgoExo4D A@1 was 82.54.
- Efficiency: COMODO (8.1M params) required 1.70G FLOPs and 75.7ms latency for a 5-second window, versus TimeSformer-Base-k400 (121.4M params) at 196.05G FLOPs and 3590.3ms, giving 47× speedup and 115× FLOPs reduction while retaining 78.8% accuracy vs. 89.8%.
- Ablations showed concat pooling outperformed mean pooling (e.g., MMEA A@1 +6.84%), FIFO sampling outperformed random sampling, and optimal queue size was dataset-specific (Ego4D 16384, EgoExo4D 1024, MMEA 2048); COMODO also outperformed L2 and InfoNCE baselines.
Methods
- Design: a frozen pretrained video teacher (default TimeSformer-base; VideoMAE-Base-k400 also tested) and a trainable IMU student (default Mantis; MOMENT-small also tested) are projected to L2-normalized embeddings; a FIFO queue stores video embeddings, and the IMU encoder is trained by minimizing cross-entropy between the softmax similarity distributions of video and IMU embeddings, equivalent to KL alignment.
- Data: three egocentric HAR datasets—Ego4D (31 activities; 253K train pairs), EgoExo4D (8 activities; 109K train pairs), and MMEA (32 activities; 4553 train pairs). Ego4D/EgoExo4D were preprocessed to 200 Hz IMU, 5-second non-overlapping windows, and 224×224 video at 10 FPS; MMEA used 25 Hz IMU and official splits.
- Analysis: trained for 20 epochs with batch size 128, learning rate 3e-4, τ_v=0.1, τ_x=0.05, on a single NVIDIA L40S GPU with seed 42; evaluated by training an RBF-kernel SVM on frozen IMU features and reporting Accuracy@1/@3/@5 against supervised and self-supervised baselines, including cross-dataset transfer.
Limitations
- Stated in the paper: COMODO relies on synchronized video-IMU data pairs during training; precise temporal alignment between heterogeneous modalities remains challenging, though similarity-distribution alignment mitigates it; and effectiveness is tied to the representational capacity of the pretrained video teacher.
- Stated in the paper: the current framework uses one-way knowledge transfer from a frozen teacher rather than mutual distillation, and does not explicitly consider personalization or contextual adaptation.
- Evident from the text: downstream evaluation still uses labels to train an SVM on extracted features, so the full pipeline is not label-free; cross-dataset generalization is tested only across the three egocentric datasets, and standalone IMU datasets cannot be used without paired video.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
6 Valid · 2 Not valid · 8 Not checked · 141 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“while IMU2CLIP reports results on Ego4D, the IMU2CLIP paper does not provide official benchmark splits.”
What the sources say
- Partly supportedcites [40] [40] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Damavandi. 2023. IMU2CLIP: Languagegrounded Motion Sensor Translation with Multimodal Contrastive Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 13246-13253. - sourceExperimental — accuracy not yet measured
The Results section reports Ego4D test-set results. The Methods section says the authors split the data into training, validation, and test sets by video ID, and Table 1 gives split sizes. Our copy does not say whether those splits were provided as official benchmark splits, so it supports the results portion but does not establish the claim about their absence. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Continuous wearable video is often untenable due to power consumption and privacy concerns [4, 36, 41, 43].”
What the sources say
- Not checked — no full text of the sourcecites [4] [4] Divyanshu Bhardwaj, Alexander Ponticello, Shreya Tomar, Adrian Dabrowski, and Katharina Krombholz. 2024. In Focus, Out of Privacy: The Wearer's Perspective on the Privacy Dilemma of Camera Glasses. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 577, 18 pages. doi:10.1145/3613904.3642242 - source
- Not checked — no full text of the sourcecites [36] [36] Si Liao, Hanwei He, Huangxun Chen, and Zhice Yang. 2025. Bystander Privacy in Video Sharing Era: Automated Consent Compliance through Platform Censorship. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25) . Association for Computing Machinery, New York, NY, USA, Article 902, 16 pages. doi:10.1145/3706598.3713391 - source
- Partly supportedcites [41] [41] Tamara Mujirishvili, Anton Fedosov, Kooshan Hashemifard, Pau Climent-Pérez, and Francisco Florez-Revuelta. 2024. 'I Don't Want to Become a Number': Examining Different Stakeholder Perspectives on a Video-Based Monitoring System for Senior Care with Inherent Privacy Protection (by Design).. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 774, 19 pages. doi:10.1145/3613904.3642164 - sourceExperimental — accuracy not yet measured
Section 2.3 (page unavailable) says cameras in private places raise privacy concerns that hinder their acceptability. It does not say that continuous wearable video is often untenable, or identify power consumption as a reason; its discussion of power efficiency in Section 3.0.1 concerns the hardware chosen for a senior-care monitoring system. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [43] [43] Octav Opaschi and Radu-Daniel Vatavu. 2020. Uncovering Practical Security and Privacy Threats for Connected Glasses with Embedded Video Cameras. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 4, Article 167 (Dec. 2020), 26 pages. doi:10.1145/3432700 - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As illustrated in Figure 6, COMODO achieves a superior balance between recognition accuracy and computational cost.”
plainstates an associationn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, despite sharing the same architecture and hardware footprint as IMU2CLIP, COMODO consistently achieves higher accuracy (+3.6% mean A@1), demonstrating that similarity distribution distillation enhances semantic representations without introducing additional inference overhead.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is further evidenced by the fact that the combination of the stronger Mantis student with either teacher consistently yields the best and second-best results overall.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
88 more claims
Create a free account to see all 157 claims and to verify the 6 references still open.
- Probably not a substantial claim. 31
- It may state two separate claims as one. 44
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.6 Ablation Study · p. 15
“The results in Table 5 reveal that COMODO consistently and substantially outperforms the IMU2CLIP across every teacher-student configuration on all datasets.”
§ 4.6 Ablation Study · p. 16
“| MOMENT 𝑆 | VideoMAE 𝐵 | IMU2CLIP | 41.39 | 65.82 | 76.45 | 69.77 | 94.61 | 98.81 | 78.42 | 91.19 | 94.60 | | | VideoMAE 𝐵 | COMODO | 57.74 | 76.66 | 83.79 | 81.60 | 97.68 | 99.48 | 79.48 | 91.19 | 94.45 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Baseline Implementation Detail · p. 28
“| CrossHAR | 512 | 8 4.97 | 96.55 | 98.11 |”
§ 4.2 Experimental Settings · p. 11
“| CrossHAR | ✗ | 50.96 | 72.66 | 81.53 | 77.14 | 96.95 | 99.31 | 84.97 | 96.55 | 97.74 |”
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · cited 2×
“[43] shows that multimodal students can out-perform unimodal teachers by learning richer cross-modal representations.”
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Zhou, H., Zhang, S.-H., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2020) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2012.07436 · Cited by 9,681
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17325 · Cited by 7,752
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“In this way, our representation learning distillation framework remains unsupervised, with class labels only used to train the SVM on the extracted features.”
Are Transformers Effective for Time Series Forecasting?
Zeng, A., Chen, M.-H., Zhang, L., et al. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2205.13504 · Cited by 4,870
Are Transformers Effective for Time Series Forecasting?
Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26317 · Cited by 3,320
Continual lifelong learning with neural networks: A review
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.01.012 · Cited by 3,117
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Cited by
RightFeatKD: Selective Feature-Based Knowledge Distillation
Haque, S. T., Yan, Y., & Ngu, A. H. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31927-2_43
Metadata
- Authors
- Chen, B., Wongso, W., Li, Z., Khaokaew, Y., Xue, H., & Salim, F. D.
- Affiliations
- UNSW Sydney (AU)
- Venue
- Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Published
- 2026-06-15
- Volume
- 10
- Issue
- 2
- Pages
- 1-29
- ISSN
- 2474-9567 (ISSN-L)
- OpenAlex
- W4414565371
- Semantic Scholar
- c1b81ba2d90f8ce02b66263c5af3fbe23ae65cdb
- Cited by
- 1
- References
- 63
- Citations by year
- 2026: 1
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- ARC Centre of Excellence for Automated Decision-Making and Society (CE200100005)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- inertial measurement unit, cross-modal knowledge distillation, self-supervised learning, wearable sensors, energy efficiency, privacy preservation, cross-dataset generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition. https://app.cochat.ai/papers/10.1145/3810218
- The paper
- Chen, B., Wongso, W., Li, Z., Khaokaew, Y., Xue, H., & Salim, F. D. (2026). COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. https://doi.org/10.1145/3810218
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.