CoChat

Zero-Shot Learning for IMU-Based Activity Recognition Using Video Embeddings

Tong, Catherine, Ge, Jinchen, Lane, Nicholas D. · 2021 · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Cited by 40

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

The Activity Recognition Chain generally precludes the challenging scenario of recognizing new activities that were unseen during training, despite this scenario being a practical and common one as users perform diverse activities at test time. A few prior works have adopted zero-shot learning methods for IMU-based activity recognition, which work by relating seen and unseen classes through an auxiliary semantic space. However, these methods usually rely heavily on a hand-crafted attribute space which is costly to define, or a learnt semantic space based on word embedding, which lacks motion-related information crucial for distinguishing IMU features. Instead, we propose a strategy to exploit videos of human activities to construct an informative semantic space. With our approach, knowledge from state-of-the-art video action recognition models is encoded into video embeddings to relate seen and unseen activity classes. Experiments on three public datasets find that our approach outperforms other learnt semantic spaces, with an additional desirable feature of scalability, as recognition performance is seen to scale with the amount of data used. More generally, our results indicate that exploiting information from the video domain for IMU-based tasks is a promising direction, with tangible returns in a zero-shot learning scenario.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3494995

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 41

  1. 10.1016/j.patrec.2014.04.011 · DOI · Paper page
  2. 10.1145/3411764.3445138 · DOI
  3. Zeynep Akata , Honglak Lee , and Bernt Schiele . 2014. Zero-shot learning with structured embeddings. arXiv preprint arXiv:1409.8403 ( 2014 ). Zeynep Akata, Honglak Lee, and Bernt Schiele. 2014. Zero-shot learning with structured embeddings. arXiv preprint arXiv:1409.8403 (2014). · arXiv
  4. 10.1109/CVPR.2013.111 · DOI · Paper page
  5. 10.1109/CVPR.2015.7298911 · DOI · Paper page
  6. Jimmy Ba , Kevin Swersky , Sanja Fidler , and Ruslan Salakhutdinov . 2015. Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions. CoRR abs/1506.00511 ( 2015 ). arXiv:1506.00511 http://arxiv.org/abs/1506.00511 Jimmy Ba, Kevin Swersky, Sanja Fidler, and Ruslan Salakhutdinov. 2015. Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions. CoRR abs/1506.00511 (2015). arXiv:1506.00511 http://arxiv.org/abs/1506.00511 · arXiv
  7. Abhijit Bendale and Terrance E. Boult . 2016. Towards Open Set Deep Networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Abhijit Bendale and Terrance E. Boult. 2016. Towards Open Set Deep Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  8. Yannick Le Cacheux , Hervé Le Borgne, and Michel Crucianu . 2018 . From Classical to Generalized Zero-Shot Learning: a Simple Adaptation Process. CoRR abs/1809.10120 (2018). arXiv:1809.10120 http://arxiv.org/abs/1809.10120 Yannick Le Cacheux, Hervé Le Borgne, and Michel Crucianu. 2018. From Classical to Generalized Zero-Shot Learning: a Simple Adaptation Process. CoRR abs/1809.10120 (2018). arXiv:1809.10120 http://arxiv.org/abs/1809.10120 · arXiv
  9. 10.1145/3432224 · DOI
  10. 10.1109/CVPR.2017.502 · DOI · Paper page
  11. 10.1109/CVPR.2016.575 · DOI · Paper page
  12. 10.1109/CVPR.2016.575 · DOI · Paper page
  13. 10.1109/ICIP.2015.7350781 · DOI · Paper page
  14. 10.1145/2493432.2493511 · DOI · Paper page
  15. 10.1145/2462456.2464438 · DOI · Paper page
  16. 10.1016/j.neucom.2021.01.036 · DOI
  17. Rafael Felix , Michele Sasdelli , Ian D. Reid , and Gustavo Carneiro . 2019. Multi-modal Ensemble Classification for Generalized Zero Shot Learning. CoRR abs/1901.04623 ( 2019 ). arXiv:1901.04623 http://arxiv.org/abs/1901.04623 Rafael Felix, Michele Sasdelli, Ian D. Reid, and Gustavo Carneiro. 2019. Multi-modal Ensemble Classification for Generalized Zero Shot Learning. CoRR abs/1901.04623 (2019). arXiv:1901.04623 http://arxiv.org/abs/1901.04623 · arXiv
  18. 10.1016/j.patcog.2020.107263 · DOI
  19. Sergey Ioffe and Christian Szegedy . 2015 . Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. CoRR abs/1502.03167 (2015). arXiv:1502.03167 http://arxiv.org/abs/1502.03167 Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. CoRR abs/1502.03167 (2015). arXiv:1502.03167 http://arxiv.org/abs/1502.03167 · arXiv
  20. David Kahle , Terrance Savitsky , Stephen Schnelle , and Volkan Cevher . 2008. Junction tree algorithm. Stat 631 ( 2008 ). David Kahle, Terrance Savitsky, Stephen Schnelle, and Volkan Cevher. 2008. Junction tree algorithm. Stat 631 (2008).
  21. Will Kay , João Carreira , Karen Simonyan , Brian Zhang , Chloe Hillier , Sudheendra Vijayanarasimhan , Fabio Viola , Tim Green , Trevor Back , Paul Natsev , Mustafa Suleyman , and Andrew Zisserman . 2017. The Kinetics Human Action Video Dataset. CoRR abs/1705.06950 ( 2017 ). arXiv:1705.06950 http://arxiv.org/abs/1705.06950 Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. 2017. The Kinetics Human Action Video Dataset. CoRR abs/1705.06950 (2017). arXiv:1705.06950 http://arxiv.org/abs/1705.06950 · arXiv
  22. Diederik Kingma and Jimmy Ba . 2014 . Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014). Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014).
  23. 10.1109/ICCV.2015.282 · DOI · Paper page
  24. H. Kuehne , H. Jhuang , E. Garrote , T. Poggio , and T. Serre . 2011. HMDB: a large video database for human motion recognition . In Proceedings of the International Conference on Computer Vision (ICCV). H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. 2011. HMDB: a large video database for human motion recognition. In Proceedings of the International Conference on Computer Vision (ICCV).
  25. 10.1371/journal.pone.0075196 · DOI · Paper page
  26. 10.3390/s19225043 · DOI
  27. Tomás Mikolov , Ilya Sutskever , Kai Chen , Greg Corrado , and Jeffrey Dean . 2013. Distributed Representations of Words and Phrases and their Compositionality. CoRR abs/1310.4546 ( 2013 ). arXiv:1310.4546 http://arxiv.org/abs/1310.4546 Tomás Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. CoRR abs/1310.4546 (2013). arXiv:1310.4546 http://arxiv.org/abs/1310.4546 · arXiv
  28. David C Mohr , Mi Zhang , and Stephen M Schueller . 2017. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual review of clinical psychology 13 ( 2017 ), 23--47. David C Mohr, Mi Zhang, and Stephen M Schueller. 2017. Personal sensing: understanding mental health using ubiquitous sensors and machine learning. Annual review of clinical psychology 13 (2017), 23--47.
  29. 10.3390/s18082485 · DOI
  30. Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , Alban Desmaison , Andreas Kopf , Edward Yang , Zachary DeVito , Martin Raison , Alykhan Tejani , Sasank Chilamkurthy , Benoit Steiner , Lu Fang , Junjie Bai , and Soumith Chintala . 2019. PyTorch: An Imperative Style , High-Performance Deep Learning Library . In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  31. 10.3115/v1 · doi:10.3115/v1
  32. Farhad Pourpanah , Moloud Abdar , Yuxuan Luo , Xinlei Zhou , Ran Wang , Chee Peng Lim, and Xi-Zhao Wang . 2020 . A Review of Generalized Zero-Shot Learning Methods. CoRR abs/2011.08641 (2020). arXiv:2011.08641 https://arxiv.org/abs/2011.08641 Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, and Xi-Zhao Wang. 2020. A Review of Generalized Zero-Shot Learning Methods. CoRR abs/2011.08641 (2020). arXiv:2011.08641 https://arxiv.org/abs/2011.08641 · arXiv
  33. Richard Socher , Milind Ganjoo , Christopher D. Manning , and Andrew Y. Ng . 2013. Zero-Shot Learning through Cross-Modal Transfer . In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1 (Lake Tahoe, Nevada) (NIPS'13). Curran Associates Inc., Red Hook, NY, USA, 935--943. Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-Shot Learning through Cross-Modal Transfer. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1 (Lake Tahoe, Nevada) (NIPS'13). Curran Associates Inc., Red Hook, NY, USA, 935--943.
  34. Khurram Soomro , Amir Roshan Zamir, and Mubarak Shah . 2012 . UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. CoRR abs/1212.0402 (2012). arXiv:1212.0402 http://arxiv.org/abs/1212.0402 Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. CoRR abs/1212.0402 (2012). arXiv:1212.0402 http://arxiv.org/abs/1212.0402 · arXiv
  35. Catherine Tong , Shyam A. Tailor , and Nicholas D . Lane . 2020 . Are Accelerometers for Activity Recognition a Dead-end? CoRR abs/2001.08111 (2020). arXiv:2001.08111 https://arxiv.org/abs/2001.08111 Catherine Tong, Shyam A. Tailor, and Nicholas D. Lane. 2020. Are Accelerometers for Activity Recognition a Dead-end? CoRR abs/2001.08111 (2020). arXiv:2001.08111 https://arxiv.org/abs/2001.08111 · arXiv
  36. 10.1007/s11263-017-1027-5 · DOI · Paper page
  37. 10.1145/3106426.3106526 · DOI
  38. Matthew Willetts , Sven Hollowell , Louis Aslett , Chris Holmes , and Aiden Doherty . 2018. Statistical machine learning of sleep and physical activity phenotypes from sensor data in 96,220 UK Biobank participants. Scientific reports 8, 1 ( 2018 ), 1--10. Matthew Willetts, Sven Hollowell, Louis Aslett, Chris Holmes, and Aiden Doherty. 2018. Statistical machine learning of sleep and physical activity phenotypes from sensor data in 96,220 UK Biobank participants. Scientific reports 8, 1 (2018), 1--10.
  39. Tong Wu , Yiqiang Chen , Yang Gu , Jiwei Wang , Siyu Zhang , and Zhanghu Zhechen . 2020. Multi-Layer Cross Loss Model for Zero-Shot Human Activity Recognition . In Advances in Knowledge Discovery and Data Mining, Hady W. Lauw, Raymond Chi-Wing Wong, Alexandros Ntoulas, Ee-Peng Lim, See-Kiong Ng , and Sinno Jialin Pan (Eds.). Springer International Publishing , Cham , 210--221. Tong Wu, Yiqiang Chen, Yang Gu, Jiwei Wang, Siyu Zhang, and Zhanghu Zhechen. 2020. Multi-Layer Cross Loss Model for Zero-Shot Human Activity Recognition. In Advances in Knowledge Discovery and Data Mining, Hady W. Lauw, Raymond Chi-Wing Wong, Alexandros Ntoulas, Ee-Peng Lim, See-Kiong Ng, and Sinno Jialin Pan (Eds.). Springer International Publishing, Cham, 210--221.
  40. 10.1109/CVPR.2016.15 · DOI · Paper page
  41. 10.1109/CVPR.2018.00800 · DOI

Cited by

See all 40

Metadata

Authors
Tong, C., Ge, J., & Lane, N. D.
Venue
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Published
2021-12-27
Volume
5
Issue
4
Pages
1-23
ISSN
2474-9567 (ISSN-L)
OpenAlex
W4200081143
Cited by
40
References
18
FWCI
1.85
Type
journal article
Language
en
Open access
closed
Funders
Engineering and Physical Sciences Research Council (EP/R513295/1)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.