CoChat

COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition

Open access
Chen, Baiyu, Wongso, Wilson, Li, Zechen, Khaokaew, Yonchanok, Xue, Hao, Salim, Flora D. · 2026 · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Published version of a preprint

The DOI matches its Crossref record; 62 of the 77 references it cites were found in a registry or an index, and 2 are not checked yet; 9 of the 26 sources its claims cite were checked for support.

Abstract

The goal of creating intelligent, human-centered wearable systems for continuous activity understanding faces a fundamental trade-off: Egocentric video-based models capture rich semantic information and have demonstrated strong performance in human activity recognition (HAR), but their high power consumption, privacy concerns, and dependence on lighting limit their feasibility for continuous on-device recognition. In contrast, inertial measurement unit (IMU) sensors offer an energy-efficient, privacy-preserving alternative, yet lack large-scale annotated datasets, leading to weaker generalization. To bridge this gap, we propose COMODO, a cross-modal self-supervised distillation framework that transfers semantic knowledge from video to IMU without requiring labels. COMODO leverages a pretrained and frozen video encoder to construct a dynamic instance queue to align the feature distributions of video and IMU embeddings. This enables the IMU encoder to inherit rich semantic structure from video while maintaining its efficiency for real-world applications. Experiments on multiple egocentric HAR datasets show that COMODO consistently improves downstream performance, matching or surpassing fully supervised models, and demonstrating strong cross-dataset generalization. Benefiting from its simplicity and flexibility, COMODO is compatible with diverse pretrained video and time-series models, offering the potential to leverage more powerful teacher and student foundation models in future ubiquitous computing research. The code is available at this repository: https://github.com/cruiseresearchgroup/COMODO.

Verified 2026-10-10 · 2 references not checked yet

Overview

CoChat Summary

TL;DR

COMODO is a cross-modal self-supervised distillation framework that transfers semantic knowledge from a frozen pretrained video encoder to a lightweight IMU encoder via a dynamic FIFO queue and similarity-distribution alignment, without IMU labels. Across three egocentric HAR datasets, it matches or surpasses fully supervised baselines and improves cross-dataset generalization while retaining IMU inference efficiency.

Key findings

  • On IMU-based HAR, COMODO achieved A@1 of 59.13 on Ego4D, 84.92 on EgoExo4D, and 92.48 on MMEA, outperforming IMU2CLIP by 4.3%, 5.46%, and 1.07% A@1, respectively.
  • In cross-dataset transfer, COMODO improved over IMU2CLIP, CrossHAR, and supervised fine-tuned Mantis in all cases; e.g., EgoExo4D→Ego4D A@1 was 56.62 vs. 51.75 for IMU2CLIP and 52.58 for Mantis†, and Ego4D→EgoExo4D A@1 was 82.54.
  • Efficiency: COMODO (8.1M params) required 1.70G FLOPs and 75.7ms latency for a 5-second window, versus TimeSformer-Base-k400 (121.4M params) at 196.05G FLOPs and 3590.3ms, giving 47× speedup and 115× FLOPs reduction while retaining 78.8% accuracy vs. 89.8%.
  • Ablations showed concat pooling outperformed mean pooling (e.g., MMEA A@1 +6.84%), FIFO sampling outperformed random sampling, and optimal queue size was dataset-specific (Ego4D 16384, EgoExo4D 1024, MMEA 2048); COMODO also outperformed L2 and InfoNCE baselines.

Methods

  • Design: a frozen pretrained video teacher (default TimeSformer-base; VideoMAE-Base-k400 also tested) and a trainable IMU student (default Mantis; MOMENT-small also tested) are projected to L2-normalized embeddings; a FIFO queue stores video embeddings, and the IMU encoder is trained by minimizing cross-entropy between the softmax similarity distributions of video and IMU embeddings, equivalent to KL alignment.
  • Data: three egocentric HAR datasets—Ego4D (31 activities; 253K train pairs), EgoExo4D (8 activities; 109K train pairs), and MMEA (32 activities; 4553 train pairs). Ego4D/EgoExo4D were preprocessed to 200 Hz IMU, 5-second non-overlapping windows, and 224×224 video at 10 FPS; MMEA used 25 Hz IMU and official splits.
  • Analysis: trained for 20 epochs with batch size 128, learning rate 3e-4, τ_v=0.1, τ_x=0.05, on a single NVIDIA L40S GPU with seed 42; evaluated by training an RBF-kernel SVM on frozen IMU features and reporting Accuracy@1/@3/@5 against supervised and self-supervised baselines, including cross-dataset transfer.

Limitations

  • Stated in the paper: COMODO relies on synchronized video-IMU data pairs during training; precise temporal alignment between heterogeneous modalities remains challenging, though similarity-distribution alignment mitigates it; and effectiveness is tied to the representational capacity of the pretrained video teacher.
  • Stated in the paper: the current framework uses one-way knowledge transfer from a frozen teacher rather than mutual distillation, and does not explicitly consider personalization or contextual adaptation.
  • Evident from the text: downstream evaluation still uses labels to train an SVM on extracted features, so the full pipeline is not label-free; cross-dataset generalization is tested only across the three egocentric datasets, and standalone IMU datasets cannot be used without paired video.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3810218

Cites

62 of 77 references verified0 citation errors · 13 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

6 Valid · 2 Not valid · 8 Not checked · 141 Own results and interpretations

88 more claims

Create a free account to see all 157 claims and to verify the 6 references still open.

See all claims — free account
  • Probably not a substantial claim. 31
  • It may state two separate claims as one. 44

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Cited by

See all 1

Metadata

Authors
Chen, B., Wongso, W., Li, Z., Khaokaew, Y., Xue, H., & Salim, F. D.
Affiliations
UNSW Sydney (AU)
Venue
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Published
2026-06-15
Volume
10
Issue
2
Pages
1-29
ISSN
2474-9567 (ISSN-L)
OpenAlex
W4414565371
Cited by
1
References
63
Citations by year
2026: 1
FWCI
2.83
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
ARC Centre of Excellence for Automated Decision-Making and Society (CE200100005)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
inertial measurement unit, cross-modal knowledge distillation, self-supervised learning, wearable sensors, energy efficiency, privacy preservation, cross-dataset generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.