IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
Open accessAbstract
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos -- while preserving the transitivity across these modalities. We explore several new IMU-based applications that IMU2CLIP enables, such as motion-based media retrieval and natural language reasoning tasks with motion data. In addition, we show that IMU2CLIP can significantly improve the downstream performance when fine-tuned for each application (e.g. activity recognition), demonstrating the universal usage of IMU2CLIP as a new pre-trained resource. Our code will be made publicly available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Moon et al. first applied sensor-language alignment to wearable sensors through IMU2CLIP, which uses the Ego4d dataset for training.
“Moon et al. first applied this paradigm to wearable sensors, through IMU2CLIP [135], which uses the Ego4d dataset [65] for training.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [135]
If no annotated data are available for unseen activities, cross-modal contrastive training (e.g., between sensor data and videos/text descriptions) is a good option.
“However, if no annotated data are available for the unseen activities, cross-modal contrastive training (e.g., between sensor data and videos/text descriptions) is a good option [61,82,135,201].”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [135]
A number of existing early efforts have provided proof for multi-modal possibility, through unified representations of multiple data modalities, e.g., IMU2CLIP.
“A number of existing early efforts have provided proof for such a possibility, through unified representations of multiple data modalities, e.g., IMU2CLIP [135], which learns a joint space between IMU, egocentric video, and text.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [135]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 10
Research works printed but not linked: 17
- [1]Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jack- son Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al., "Ego4d: Around the world in 3,000 hours of egocentric video," in CVPR, 2022.
- [5]Xinxing Chen, Kuangen Zhang, Haiyuan Liu, Yuquan Leng, and Chenglong Fu, "A probability distribution model-based approach for foot placement prediction in the early swing phase with a wearable imu sensor," IEEE Transactions on Neural Systems and Rehabilitation En- gineering, vol. 29, pp. 2595–2604, 2021.
- [6]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "Bert: Pre-training of deep bidirec- tional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. · arXiv
- [8]Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li, "CLIP4Clip: An empirical study of clip for end to end video clip retrieval," arXiv preprint arXiv:2104.08860, 2021. · arXiv
- [9]Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith, "Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping," arXiv preprint arXiv:2002.06305, 2020. · arXiv
- [10]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al., "Learning transferable visual models from natural lan- guage supervision," in ICML, 2021.
- [11]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, "A simple framework for contrastive learning of visual representations," in ICML, 2020.
- Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al., "Egocentric video-language pretraining," arXiv preprint arXiv:2206.01670, 2022. · arXiv
- [13]Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello, "Wav2clip: Learning robust audio representations from clip," in ICASSP, 2022.
- [14]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al., "Learning transferable visual models from natural lan- guage supervision," in ICML, 2021.
- [15]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition," in CVPR, 2016.
- [16]Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, "Imagenet: A large-scale hierarchical image database," in CVPR, 2009.
- [17]Nils Reimers and Iryna Gurevych, "Sentence-bert: Sen- tence embeddings using siamese bert-networks," in EMNLP, 2019.
- [19]Luciano Floridi and Massimo Chiriatti, "Gpt-3: Its na- ture, scope, limits, and consequences," Minds and Ma- chines, vol. 30, no. 4, pp. 681–694, 2020.
- [22]Sara Ashry, Tetsuji Ogawa, and Walid Gomaa, "Charm- deep: Continuous human activity recognition model based on deep neural network using imu sensors of smart- watch," IEEE Sensors Journal, vol. 20, no. 15, pp. 8757– 8770, 2020.
- [23]John Duchi, Elad Hazan, and Yoram Singer, "Adaptive subgradient methods for online learning and stochastic optimization," JMLR, 2011.
- [24]Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Van- houcke, et al., "Socratic models: Composing zero-shot multimodal reasoning with language," arXiv preprint arXiv:2204.00598, 2022. · arXiv
- Authors
- Moon, S., Madotto, A., Lin, Z., Dirafzoon, A., Saraf, A., Bearman, A., & Damavandi, B.
- Affiliations
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2210.14395
- OpenAlex
- W4307474240
- Cited by
- 10
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- inertial measurement unit, motion sensors, multimodal contrastive learning, CLIP, egocentric video, activity recognition, pre-training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
