Cosmo: contrastive fusion learning with small data for multimodal human activity recognition
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Human activity recognition (HAR) is a key enabling technology for a wide range of emerging applications. Although multimodal sensing systems are essential for capturing complex and dynamic human activities in real-world settings, they bring several new challenges including limited labeled multimodal data. In this paper, we propose Cosmo, a new system for contrastive fusion learning with small data in multimodal HAR applications. Cosmo features a novel two-stage training strategy that leverages both unlabeled data on the cloud and limited labeled data on the edge. By integrating novel fusion-based contrastive learning and quality-guided attention mechanisms, Cosmo can effectively extract both consistent and complementary information across different modalities for efficient fusion. Our evaluation on a cloud-edge testbed using two public datasets and a new multimodal HAR dataset shows that Cosmo delivers significant improvement over state-of-the-art baselines in both recognition accuracy and convergence delay.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In the typical deployment of a WHAR model [51], the pre-training phase is performed in the cloud to leverage the cloud computing power and large-scale wearable datasets.The supervised learning phase is performed in the local devices (e.g., smartphones) to personalize the WHAR model while ensuring user privacy.”
From Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity Recognition (2023) · cites this paper as [51]
In the citing paper
“This guidance process is usually short for better user experience, so the amount of available labeled data are very limited.”
From Spatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity Recognition (2023) · cites this paper as [51]
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997
Unsupervised feature selection for multi-cluster data
Cai, D., Zhang, C., & He, X. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835848 · Cited by 1,112
What makes for good views for contrastive learning
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10243 · Cited by 773
Chen, C., Jafari, R., & Kehtarnavaz, N. (2015) · 2015 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2015.7350781 · Cited by 765
Transition-Aware Human Activity Recognition Using Smartphones
Reyes-Ortiz, J.-L., Oneto, L., Samà, A., Parra, X., & Anguita, D. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.07.085 · Cited by 740 · cited 1×
“First, in real-world applications where the sensor data is continuously collected, we can simply remove the irrelevant data through some filtering techniques [14, 44].”
BackgroundStisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704
Research works printed but not linked: 10
- Humam Alwassel , Dhruv Mahajan , Bruno Korbar , Lorenzo Torresani , Bernard Ghanem , and Du Tran . 2020. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33 ( 2020 ). Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. 2020. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems 33 (2020).
- Chongguang Bi , Guoliang Xing , Tian Hao , Jina Huh , Wei Peng , and Mengyan Ma . 2017 . FamilyLog: A mobile system for monitoring family mealtime activities . In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 21--30 . Chongguang Bi, Guoliang Xing, Tian Hao, Jina Huh, Wei Peng, and Mengyan Ma. 2017. FamilyLog: A mobile system for monitoring family mealtime activities. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 21--30.
- Rohan Ghosh , Anupam Gupta , Andrei Nakagawa , Alcimar Soares , and Nitish Thakor . 2019. Spatiotemporal filtering for event-based action recognition. arXiv preprint arXiv:1903.07067 ( 2019 ). Rohan Ghosh, Anupam Gupta, Andrei Nakagawa, Alcimar Soares, and Nitish Thakor. 2019. Spatiotemporal filtering for event-based action recognition. arXiv preprint arXiv:1903.07067 (2019). · arXiv
- Gene Glass and Kenneth Hopkins . 1996. Statistical methods in education and psychology. Psyccritiques 41, 12 ( 1996 ). Gene Glass and Kenneth Hopkins. 1996. Statistical methods in education and psychology. Psyccritiques 41, 12 (1996).
- Guoliang Xing . 2022. Machine Learning Technologies for Advancing Digital Biomarkers for Alzheimer's Disease , Alzheimer's Drug Discovery Foundation . https://www.alzdiscovery.org/research-and-grants/portfolio-details/21130887. Guoliang Xing. 2022. Machine Learning Technologies for Advancing Digital Biomarkers for Alzheimer's Disease, Alzheimer's Drug Discovery Foundation. https://www.alzdiscovery.org/research-and-grants/portfolio-details/21130887.
- Kalun Ho , Franz-Josef Pfreundt , Janis Keuper , and Margret Keuper . 2021. Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space. arXiv preprint arXiv:2106.12303 ( 2021 ). Kalun Ho, Franz-Josef Pfreundt, Janis Keuper, and Margret Keuper. 2021. Estimating the Robustness of Classification Models by the Structure of the Learned Feature-Space. arXiv preprint arXiv:2106.12303 (2021). · arXiv
- Lampros C Kourtis , Oliver B Regele , Justin M Wright , and Graham B Jones . 2019. Digital biomarkers for Alzheimer's disease: the mobile/wearable devices opportunity. NPJ digital medicine 2, 1 ( 2019 ), 1--9. Lampros C Kourtis, Oliver B Regele, Justin M Wright, and Graham B Jones. 2019. Digital biomarkers for Alzheimer's disease: the mobile/wearable devices opportunity. NPJ digital medicine 2, 1 (2019), 1--9.
- Aaron van den Oord , Yazhe Li , and Oriol Vinyals . 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 ( 2018 ). Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018). · arXiv
- Wojciech Zaremba , Ilya Sutskever , and Oriol Vinyals . 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 ( 2014 ). Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014). · arXiv
- Fengda Zhang , Kun Kuang , Zhaoyang You , Tao Shen , Jun Xiao , Yin Zhang , Chao Wu , Yueting Zhuang , and Xiaolin Li. 2020. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982 ( 2020 ). Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. 2020. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08982 (2020). · arXiv
Cited by
Gaya-Morey, F. X., Manresa-Yee, C., & Rubio, J. M. B. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05645-1 · Cited by 137
Artificial Intelligence of Things: A Survey
Siam, M. S. I., Ahn, H., Liu, L., Alam, S., Shen, H., Cao, Z., Shroff, N. B., Krishnamachari, B., Srivastava, M. B., & Zhang, M. (2025) · ACM Transactions on Sensor Networks · doi:10.1145/3690639 · Cited by 123
Harmony: Heterogeneous Multi-Modal Federated Learning through Disentangled Model Training
Ouyang, X., Xie, Z., Fu, H., Cheng, S., Pan, L., Ling, N., Xing, G., Zhou, J., & Huang, J. (2023) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3581791.3596844 · Cited by 109
Self-citationPractically Adopting Human Activity Recognition
Xu, H., Zhou, P., Tan, R., & Li, M. (2023) · Proceedings of the 29th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3570361.3613299 · Cited by 55
A review of machine learning with small and limited data
Naser, M. Z. (2026) · Journal Of Big Data · doi:10.1186/s40537-025-01346-9 · Cited by 92
Metadata
- Authors
- Ouyang, X., Shuai, X., Zhou, J., Shi, I. W., Xie, Z., Xing, G., & Huang, J.
- Affiliations
- Chinese University of Hong Kong (HK)
- Michigan State University (US)
- Li Po Chun United World College, Hong Kong
- Chinese University of Hong Kong, Shenzhen (CN)
- Venue
- Proceedings of the 28th Annual International Conference on Mobile Computing And Networking
- Publisher
- ACM
- Published
- 2022-10-14
- Pages
- 324-337
- ISSN
- 1543-5679 (ISSN-L)
- OpenAlex
- W4306250089
- Cited by
- 94
- References
- 47
- Citations by year
- 2026: 17 · 2025: 31 · 2024: 32 · 2023: 14
- FWCI
- 6.98
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Annual International Conference on Mobile Computing and Networking (MobiCom) (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Shenzhen Institute of Artificial Intelligence and Robotics for Society
- GRF Grants of Research Grants Council (RGC) of Hong Kong (14203420)
- Guangdong Basic and Applied Basic Research Foundation (Project 2021B1515120008)
- Shenzhen Science and Technology Program (Project JCYJ2021032 4120011032)
- Alzheimer?s Drug Discovery Foundation (RDADB-201906-2019049)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- human activity recognition, multimodal fusion, contrastive learning, small labeled datasets, cloud-edge collaboration, two-stage training, unlabeled data, convergence delay
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. https://app.cochat.ai/papers/10.1145/3495243.3560519
- The paper
- Ouyang, X., Shuai, X., Zhou, J., Shi, I. W., Xie, Z., Xing, G., & Huang, J. (2022). Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. https://doi.org/10.1145/3495243.3560519
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.