CoChat

Cosmo: contrastive fusion learning with small data for multimodal human activity recognition

Ouyang, Xiaomin, Shuai, Xian, Zhou, Jiayu, Shi, Ivy Wang, Xie, Zhiyuan, Xing, Guoliang, Huang, Jianwei · 2022 · Proceedings of the 28th Annual International Conference on Mobile Computing And Networking

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.

Abstract

Human activity recognition (HAR) is a key enabling technology for a wide range of emerging applications. Although multimodal sensing systems are essential for capturing complex and dynamic human activities in real-world settings, they bring several new challenges including limited labeled multimodal data. In this paper, we propose Cosmo, a new system for contrastive fusion learning with small data in multimodal HAR applications. Cosmo features a novel two-stage training strategy that leverages both unlabeled data on the cloud and limited labeled data on the edge. By integrating novel fusion-based contrastive learning and quality-guided attention mechanisms, Cosmo can effectively extract both consistent and complementary information across different modalities for efficient fusion. Our evaluation on a cloud-edge testbed using two public datasets and a new multimodal HAR dataset shows that Cosmo delivers significant improvement over state-of-the-art baselines in both recognition accuracy and convergence delay.

Overview

Integrity signals

Registry

Crossref · 10.1145/3495243.3560519

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 49 references

Research works printed but not linked: 10

  1. Humam Alwassel , Dhruv Mahajan , Bruno Korbar , Lorenzo Torresani , Bernard Ghanem , and Du Tran . 2020. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33 ( 2020 ). Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. 2020. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33 (2020).
  2. Chongguang Bi , Guoliang Xing , Tian Hao , Jina Huh , Wei Peng , and Mengyan Ma . 2017 . FamilyLog: A mobile system for monitoring family mealtime activities . In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 21--30 . Chongguang Bi, Guoliang Xing, Tian Hao, Jina Huh, Wei Peng, and Mengyan Ma. 2017. FamilyLog: A mobile system for monitoring family mealtime activities. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 21--30.
  3. Rohan Ghosh , Anupam Gupta , Andrei Nakagawa , Alcimar Soares , and Nitish Thakor . 2019. Spatiotemporal filtering for event-based action recognition. arXiv preprint arXiv:1903.07067 ( 2019 ). Rohan Ghosh, Anupam Gupta, Andrei Nakagawa, Alcimar Soares, and Nitish Thakor. 2019. Spatiotemporal filtering for event-based action recognition. arXiv preprint arXiv:1903.07067 (2019). · arXiv
  4. Gene Glass and Kenneth Hopkins . 1996. Statistical methods in education and psychology. Psyccritiques 41, 12 ( 1996 ). Gene Glass and Kenneth Hopkins. 1996. Statistical methods in education and psychology. Psyccritiques 41, 12 (1996).
  5. Guoliang Xing . 2022. Machine Learning Technologies for Advancing Digital Biomarkers for Alzheimer's Disease , Alzheimer's Drug Discovery Foundation . https://www.alzdiscovery.org/research-and-grants/portfolio-details/21130887. Guoliang Xing. 2022. Machine Learning Technologies for Advancing Digital Biomarkers for Alzheimer's Disease, Alzheimer's Drug Discovery Foundation. https://www.alzdiscovery.org/research-and-grants/portfolio-details/21130887.
  6. Kalun Ho , Franz-Josef Pfreundt , Janis Keuper , and Margret Keuper . 2021. Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space. arXiv preprint arXiv:2106.12303 ( 2021 ). Kalun Ho, Franz-Josef Pfreundt, Janis Keuper, and Margret Keuper. 2021. Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space. arXiv preprint arXiv:2106.12303 (2021). · arXiv
  7. Lampros C Kourtis , Oliver B Regele , Justin M Wright , and Graham B Jones . 2019. Digital biomarkers for Alzheimer's disease: the mobile/wearable devices opportunity. NPJ digital medicine 2, 1 ( 2019 ), 1--9. Lampros C Kourtis, Oliver B Regele, Justin M Wright, and Graham B Jones. 2019. Digital biomarkers for Alzheimer's disease: the mobile/wearable devices opportunity. NPJ digital medicine 2, 1 (2019), 1--9.
  8. Aaron van den Oord , Yazhe Li , and Oriol Vinyals . 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 ( 2018 ). Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). · arXiv
  9. Wojciech Zaremba , Ilya Sutskever , and Oriol Vinyals . 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 ( 2014 ). Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014). · arXiv
  10. Fengda Zhang , Kun Kuang , Zhaoyang You , Tao Shen , Jun Xiao , Yin Zhang , Chao Wu , Yueting Zhuang , and Xiaolin Li. 2020. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982 ( 2020 ). Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. 2020. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982 (2020). · arXiv

Cited by

See all 94

Metadata

Authors
Ouyang, X., Shuai, X., Zhou, J., Shi, I. W., Xie, Z., Xing, G., & Huang, J.
Affiliations
  • Chinese University of Hong Kong (HK)
  • Michigan State University (US)
  • Li Po Chun United World College, Hong Kong
  • Chinese University of Hong Kong, Shenzhen (CN)
Venue
Proceedings of the 28th Annual International Conference on Mobile Computing And Networking
Publisher
ACM
Published
2022-10-14
Pages
324-337
ISSN
1543-5679 (ISSN-L)
OpenAlex
W4306250089
Cited by
94
References
47
Citations by year
2026: 17 · 2025: 31 · 2024: 32 · 2023: 14
FWCI
6.98
Type
conference paper
Language
en
Open access
closed
Locations
  • ACM Annual International Conference on Mobile Computing and Networking (MobiCom) (publishedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
Funders
  • Shenzhen Institute of Artificial Intelligence and Robotics for Society
  • GRF Grants of Research Grants Council (RGC) of Hong Kong (14203420)
  • Guangdong Basic and Applied Basic Research Foundation (Project 2021B1515120008)
  • Shenzhen Science and Technology Program (Project JCYJ2021032 4120011032)
  • Alzheimer?s Drug Discovery Foundation (RDADB-201906-2019049)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
human activity recognition, multimodal fusion, contrastive learning, small labeled datasets, cloud-edge collaboration, two-stage training, unlabeled data, convergence delay

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. https://app.cochat.ai/papers/10.1145/3495243.3560519
The paper
Ouyang, X., Shuai, X., Zhou, J., Shi, I. W., Xie, Z., Xing, G., & Huang, J. (2022). Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. https://doi.org/10.1145/3495243.3560519

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.