Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition - And Ways to Overcome Them
Open accessAbstract
Cross-modal contrastive pre-training between natural language and other modalities, e.g., vision and audio, has demonstrated astonishing performance and effectiveness across a diverse variety of tasks and domains. In this paper, we investigate whether such natural language supervision can be used for wearable sensor based Human Activity Recognition (HAR), and discover that--surprisingly--it performs substantially worse than standard end-to-end training and self-supervision. We identify the primary causes for this as: sensor heterogeneity and the lack of rich, diverse text descriptions of activities. To mitigate their impact, we also develop strategies and assess their effectiveness through an extensive experimental evaluation. These strategies lead to significant increases in activity recognition, bringing performance closer to supervised and self-supervised training, while also enabling the recognition of unseen activities and cross modal retrieval of videos. Overall, our work paves the way for better sensor-language learning, ultimately leading to the development of foundational models for HAR using wearables.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Haresamudram et al. demonstrated that cross-modal contrastive pre-training followed by zero-shot prediction is highly challenging for wearables applications due to sensor heterogeneity and lack of rich, detailed descriptions of activities, leading to poor alignment between sensor and language modalities.
“Recently, Haresamudram et al. [78] demonstrated that such a setup is highly challenging when training for wearables applications, due to factors such as sensor heterogeneity and the lack of rich, detailed descriptions of activities, leading to poor alignment between sensor and language modalities.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [78]
Experimental — accuracy not yet measuredThe drop in performance relative to end-to-end training and self-supervision can be reduced by adapting some layers of the network with target data and through text augmentation.
“Yet, it was shown that the drop in performance relative to end-to-end training and self-supervision can be reduced by adapting some layers of the network with target data, and through text augmentation.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [78]
Recognizing activities not seen during training can be beneficial, as it enables the addition of new capabilities to existing recognizers.
“Recognizing activities not seen during training can be beneficial, as it enables the addition of new capabilities to existing recognizers [78].”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [78]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Improving CLIP Training with Language Rewrites
Fan, L., Krishnan, D., Isola, P., Katabi, D., & Tian, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.20088 · Cited by 74 · see citing works
Exploring Contrastive Learning in Human Activity Recognition for Healthcare
Tang, C. I., Perez-Pozuelo, I., Spathis, D., & Mascolo, C. (2020) · arXiv · doi:10.48550/arxiv.2011.11542 · Cited by 67 · see citing works
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text UnderstandingPreprint
Xu, H., Ghosh, G., Huang, P.-Y., Okhonko, D., Aghajanyan, A., Metze, F., Zettlemoyer, L., & Feichtenhofer, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14084 · Cited by 58 · see citing works
Haresamudram, H., Beedu, A., Rabbi, M., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2408.12023 · Cited by 18 · see citing works
Self-citationIMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
Moon, S., Madotto, A., Lin, Z., Dirafzoon, A., Saraf, A., Bearman, A., & Damavandi, B. (2022) · arXiv · doi:10.48550/arxiv.2210.14395 · Cited by 10 · see citing works
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16058 · Cited by 7 · see citing works
TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity RecognitionPreprint
Zhou, Y., Yang, J., Zou, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.08245 · Cited by 6 · see citing works
ContextGPT: Infusing LLMs Knowledge into Neuro-Symbolic Activity Recognition Models
Arrotta, L., Bettini, C., Civitarese, G., & Fiori, M. (2024) · arXiv · doi:10.48550/arxiv.2403.06586 · Cited by 3 · see citing works
SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity RecognitionPreprint
Li, Z., Deldari, S., Chen, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.10624 · Cited by 1 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 36
- Banos, O.; Garcia, R.; Holgado-Terriza, J. A.; Damas, M.; Pomares, H.; Rojas, I.; Saez, A.; and Villalonga, C. 2014. mHealthDroid: a novel framework for agile development of mobile health applications. In International workshop on ambient assisted living, 91-98. Springer.
- Bin Morshed, M.; Haresamudram, H. K.; Bandaru, D.; Abowd, G. D.; and Plötz, T. 2022. A personalized approach for developing a snacking detection system using earbuds in a semi-naturalistic setting. In Proceedings of the 2022 ACM international symposium on wearable computers, 11-16.
- Chatzaki, C.; Pediaditis, M.; Vavoulas, G.; and Tsiknakis, M. 2016. Human daily activity and fall recognition using a smartphone's acceleration sensor. In International Confer- ence on Information and Communication Technologies for Ageing Well and e-Health, 100-118. Springer.
- Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020. A simple framework for contrastive learning of visual repre- sentations. In International conference on machine learning, 1597-1607.
- PMLR.
- Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei- Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248-255.
- Elizalde, B.; Deshmukh, S.; Al Ismail, M.; and Wang, H. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1-5. IEEE.
- Girdhar, R.; El-Nouby, A.; Liu, Z.; Singh, M.; Alwala, K. V.; Joulin, A.; and Misra, I. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15180-15190.
- Grauman, K.; Westbury, A.; Byrne, E.; Chavis, Z.; Furnari, A.; Girdhar, R.; Hamburger, J.; Jiang, H.; Liu, M.; Liu, X.; et al. 2022. Ego4d: Around the world in 3,000 hours of ego- centric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18995-19012.
- Haresamudram, H.; Essa, I.; and Plötz, T. 2022. Assessing the state of self-supervised human activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3): 1-47.
- Haresamudram, H.; Essa, I.; and Plötz, T. 2023. Investigat- ing enhancements to contrastive predictive coding for hu- man activity recognition. In 2023 IEEE International Con- ference on Pervasive Computing and Communications (Per- Com), 232-241. IEEE.
- Imran, S. A.; Khan, M. N. H.; Biswas, S.; and Islam, B. 2024. LLaSA: Large Multimodal Agent for Human Ac- tivity Analysis Through Wearable Sensors. arXiv preprint arXiv:2406.14498. · arXiv
- Koskimäki, H.; Siirtola, P.; and Röning, J. 2017. Myogym: introducing an open gym data set for activity recognition collected using myo armband. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiq- uitous Computing and Proceedings of the 2017 ACM Inter- national Symposium on Wearable Computers, 537-546.
- Kwon, H.; Tong, C.; Haresamudram, H.; Gao, Y.; Abowd, G. D.; Lane, N. D.; and Ploetz, T. 2020. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. Proceedings of the ACM on In- teractive, Mobile, Wearable and Ubiquitous Technologies, 4(3): 1-29.
- Ma, Y.; Xu, G.; Sun, X.; Yan, M.; Zhang, J.; and Ji, R. 2022. X-clip: End-to-end multi-grained contrastive learn- ing for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, 638-647.
- Malekzadeh, M.; Clegg, R. G.; Cavallaro, A.; and Haddadi, H. 2018. Protecting sensory data against sensitive infer- ences. In Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems, 1-6.
- Matsuki, M.; Lago, P.; and Inoue, S. 2019. Characterizing word embeddings for zero-shot sensor-based human activity recognition. Sensors, 19(22): 5043.
- Mishra, R.; Gupta, A.; Gupta, H. P.; and Dutta, T. 2020. A sensors based deep learning model for unseen locomotion mode identification using multiple semantic matrices. IEEE Transactions on Mobile Computing, 21(3): 799-810.
- Mu, N.; Kirillov, A.; Wagner, D.; and Xie, S. 2022. Slip: Self-supervision meets language-image pre-training. In Eu- ropean conference on computer vision, 529-544. Springer.
- Ordóñez, F. J.; and Roggen, D. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1): 115.
- Plötz, T. 2023. If only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios. In 2023 IEEE International Conference on Pervasive Comput- ing and Communications Workshops and other Affiliated Events (PerCom Workshops), 565-570. IEEE.
- Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, 8748-8763.
- PMLR.
- Reiss, A.; and Stricker, D. 2012. Introducing a new bench- marked dataset for activity monitoring. In 2012 16th in- ternational symposium on wearable computers, 108-109.
- Saeed, A.; Ozcelebi, T.; and Lukkien, J. 2019. Multi-task self-supervised learning for human activity detection. Pro- ceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2): 1-30.
- Shen, S.; Li, C.; Hu, X.; Xie, Y.; Yang, J.; Zhang, P.; Gan, Z.; Wang, L.; Yuan, L.; Liu, C.; et al. 2022. K-lite: Learn- ing transferable visual models with external knowledge. Advances in Neural Information Processing Systems, 35: 15558-15573.
- Stisen, A.; Blunck, H.; Bhattacharya, S.; Prentow, T. S.; Kjærgaard, M. B.; Dey, A.; Sonne, T.; and Jensen, M. M. 2015. Smart devices are different: Assessing and mitigat- ingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems, 127-140.
- Sztyler, T.; and Stuckenschmidt, H. 2016. On-body localiza- tion of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), 1- 9. IEEE.
- Thomee, B.; Shamma, D. A.; Friedland, G.; Elizalde, B.; Ni, K.; Poland, D.; Borth, D.; and Li, L.-J. 2016. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2): 64-73.
- Tong, C.; Ge, J.; and Lane, N. D. 2021. Zero-shot learning for imu-based activity recognition using video embeddings. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(4): 1-23.
- Verma, G.; Choi, M.; Sharma, K.; Watson-Daniels, J.; Oh, S.; and Kumar, S. 2024. Mysterious Projections: Mul- timodal LLMs Gain Domain-Specific Visual Capabilities Without Richer Cross-Modal Projections. arXiv preprint arXiv:2402.16832. · arXiv
- Willetts, M.; Hollowell, S.; Aslett, L.; Holmes, C.; and Do- herty, A. 2018. Statistical machine learning of sleep and physical activity phenotypes from sensor data in 96,220 UK Biobank participants. Scientific reports, 8(1): 1-10.
- Wu, H.-H.; Seetharaman, P.; Kumar, K.; and Bello, J. P. 2022. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), 4563-4567. IEEE.
- Wu, Y.; Chen, K.; Zhang, T.; Hui, Y.; Berg-Kirkpatrick, T.; and Dubnov, S. 2023. Large-scale contrastive language- audio pretraining with feature fusion and keyword-to- caption augmentation. In ICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP), 1-5. IEEE.
- Xia, K.; Li, W.; Gan, S.; and Lu, S. 2024. TS2ACT: Few-Shot Human Activity Sensing with Cross-Modal Co- Learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 7(4): 1-22.
- Yang, J.; Li, C.; Zhang, P.; Xiao, B.; Liu, C.; Yuan, L.; and Gao, J. 2022. Unified contrastive learning in image-text- label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19163-19173.
Cited by
Sorted by citations
Haresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Plötz, T. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3729467 · Cited by 33 · see citing works
Self-citation
- Authors
- Haresamudram, H., Beedu, A., Rabbi, M., Saha, S., Essa, I., & Ploetz, T.
- Affiliations
- Georgia Institute of Technology (US)
- Optum (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2025-04-11
- Volume
- 39
- Issue
- 1
- Pages
- 273-281
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4409346737
- Cited by
- 8
- References
- 12
- Citations by year
- 2026: 4 · 2025: 4
- FWCI
- 6.05
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2112633)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Keywords
- human activity recognition, wearable sensors, natural language supervision, sensor heterogeneity, self-supervised learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
