CoChat
Sign Up
Cited by 29Human Pose and Action RecognitionMultimodal Machine Learning ApplicationsHand Gesture Recognition Systems

The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.

Hear Me out: Fusional Approaches for Audio Augmented Temporal Action Localization

Open access
Bagchi, Anurag, Mahmood, Jazib, Fernandes, Dolton, Sarvadevabhatla, Ravi·2022·Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
DOI 10.5220/0010832700003124
Open this paper in CoChat — free account

Abstract

State of the art architectures for untrimmed video Temporal Action Localization (TAL) have only considered RGB and Flow modalities, leaving the information-rich audio modality totally unexploited. Audio fusion has been explored for the related but arguably easier problem of trimmed (clip-level) action recognition. However, TAL poses a unique set of challenges. In this paper, we propose simple but effective fusion-based approaches for TAL. To the best of our knowledge, our work is the first to jointly consider audio and video modalities for supervised TAL. We experimentally show that our schemes consistently improve performance for state of the art video-only TAL approaches. Specifically, they help achieve new state of the art performance on large-scale benchmark datasets - ActivityNet-1.3 (54.34 mAP@0.5) and THUMOS14 (57.18 mAP@0.5). Our experiments include ablations involving multiple fusion schemes, modality combinations and TAL architectures. Our code, models and associated data are available at https://github.com/skelemoa/tal-hmo.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.5220/0010832700003124

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 3 references

Research works printed but not linked: 37

  1. [1][1] Humam Alwassel, Fabian Caba Heilbron, Victor Escorcia, and Bernard Ghanem. Diagnosing error in temporal action detectors. In ECCV , 2018. -
  2. [2][2] Humam Alwassel, Silvio Giancola, and Bernard Ghanem. TSP: temporally-sensitive pretraining of video encoders for localization tasks. CoRR , abs/2011.11479, 2020. - · arXiv
  3. [3][3] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV , 2017. -
  4. [6][6] Jo˜ ao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR , 2017. -
  5. [10][10] Jiyang Gao, Kan Chen, and Ram Nevatia. CTAP: complementary temporal action proposal generation. CoRR , abs/1807.04821, 2018. - · arXiv
  6. [12][12] Ross Girshick. Fast r-cnn. In ICCV , pages 1440-1448, 2015. -
  7. [15][15] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, - Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In ICASSP , 2017. -
  8. [16][16] Hongda Jiang, Yanghao Li, Sijie Song, and Jiaying Liu. Rethinking Fusion Baselines for Multi-modal Human Action Recognition: 19th Pacific-Rim Conference on Multimedia, Hefei, China, September 21-22, 2018, Proceedings, Part III . 2018. -
  9. [18][18] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV , 2019. -
  10. [19][19] Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. Cross-attentional audio-visual fusion for weaklysupervised action localization. In ICLR , 2021. -
  11. [21][21] Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li, Feiyue Huang, and Rongrong Ji. Fast learning of temporal action proposal via dense boundary generator. Proceedings of the AAAI Conference on Artificial Intelligence , 2020. -
  12. [22][22] Chuming Lin, Chengming Xu, Donghao Luo, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yanwei Fu. Learning salient boundary feature for anchor-free temporal action localization. In CVPR , pages 3320-3329, 2021. -
  13. [24][24] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. BMN: boundary-matching network for temporal action proposal generation. CoRR , abs/1907.09702, 2019. - · arXiv
  14. [25][25] Tianwei Lin, Xu Zhao, and Zheng Shou. Single shot temporal action detection. CoRR , abs/1710.06236, 2017. - · arXiv
  15. [26][26] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: boundary sensitive network for temporal action proposal generation. CoRR , abs/1806.02964, 2018. - · arXiv
  16. [27][27] Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, and Philip H. S. Torr. Multi-shot temporal event localization: A benchmark. In CVPR , pages 12596-12606, June 2021. -
  17. [28][28] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multi-granularity generator for temporal action proposal. CoRR , abs/1811.11524, 2018. - · arXiv
  18. [29][29] Xiang Long, Chuang Gan, Gerard De Melo, Xiao Liu, Yandong Li, Fu Li, and Shilei Wen. Multimodal keyless attention fusion for video classification. In AAAI 2018 , 2018. -
  19. [30][30] Xiang Long, Chuang Gan, Gerard de Melo, Jiajun Wu, Xiao Liu, and Shilei Wen. Attention clusters: Purely attention based local feature integration for video classification. In CVPR , 2018. -
  20. [32][32] Alberto Montes, Amaia Salvador, and Xavier Gir´ o-i-Nieto. Temporal activity detection in untrimmed videos with recurrent neural networks. CoRR , abs/1608.08128, 2016. - · arXiv
  21. [33][33] Nobutaka Ono, Noboru Harada, Yohei Kawaguchi, Annamaria Mesaros, Keisuke Imoto, Yuma Koizumi, , and Tatsuya Komatsu. Proceedings of the Fifth Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2020) . 2020. -
  22. [34][34] Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV , 2018. -
  23. [37][37] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang. CDC: convolutional-deconvolutional networks for precise temporal action localization in untrimmed videos. CoRR , abs/1703.01515, 2017. - · arXiv
  24. [38][38] Zheng Shou, Dongang Wang, and Shih-Fu Chang. Action temporal localization in untrimmed videos via multi-stage cnns. CoRR , abs/1601.02129, 2016. - · arXiv
  25. [39][39] Haisheng Su, Weihao Gan, Wei Wu, Junjie Yan, and Yu Qiao. BSN++: complementary boundary regressor with scale-balanced relation modeling for temporal action proposal generation. CoRR , abs/2009.07641, 2020. - · arXiv
  26. [41][41] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. TPAMI , 2019. -
  27. [42][42] Quan Wang, Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopz Moreno. Speaker diarization with lstm. In ICASSP , 2018. -
  28. [43][43] Zuxuan Wu, Yu-Gang Jiang, Xi Wang, Hao Ye, and Xiangyang Xue. Multi-stream multi-class fusion of deep networks for video classification. In ACM Multimedia , 2016. -
  29. [45][45] Huijuan Xu, Abir Das, and Kate Saenko. R-C3D: region convolutional 3d network for temporal activity detection. CoRR , abs/1703.07814, 2017. - · arXiv
  30. [46][46] Haoming Xu, Runhao Zeng, Qingyao Wu, Mingkui Tan, and Chuang Gan. Cross-modal relation-aware networks for audio-visual event localization. In ACM Multimedia , pages 3893-3901, 2020. -
  31. [47][47] Mengmeng Xu, Chen Zhao, David S. Rojas, Ali Thabet, and Bernard Ghanem. G-tad: Sub-graph localization for temporal action detection. In CVPR , 2020. -
  32. [48][48] Serena Yeung, Olga Russakovsky, Greg Mori, and Li FeiFei. End-to-end learning of action detection from frame glimpses in videos. CoRR , 2015. -
  33. [49][49] Runhao Zeng, Chuang Gan, Peihao Chen, Wenbing Huang, Qingyao Wu, and Mingkui Tan. Breaking winner-takes-all: Iterative-winners-out networks for weakly supervised temporal action localization. IEEE Transactions on Image Processing , 2019. -
  34. [50][50] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan. Graph convolutional networks for temporal action localization. In ICCV , 2019. -
  35. [51][51] Aonan Zhang, Quan Wang, Zhenyao Zhu, John Paisley, and Chong Wang. Fully supervised speaker diarization. In ICASSP , 2019. -
  36. [52][52] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Dahua Lin, and Xiaoou Tang. Temporal action detection with structured segment networks. CoRR , abs/1704.06228, 2017. - · arXiv
  37. [53][53] Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audiovisual event line. In CVPR , 2021.

Cited by

Sorted by citations

See all 29

Metadata

Authors
Bagchi, A., Mahmood, J., Fernandes, D., & Sarvadevabhatla, R.
Affiliations
International Institute of Information Technology, Hyderabad (IN)
ORCID
Sarvadevabhatla, Ravi
Venue
Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications
Publisher
SCITEPRESS - Science and Technology Publications
Published
2022
Pages
144-154
DOI
10.5220/0010832700003124
OpenAlex
W3173161217
Cited by
29
References
3
Citations by year
2026: 4 · 2025: 8 · 2024: 5 · 2023: 8 · 2022: 4
FWCI
2.27
Type
conference paper
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
temporal action localization, audiovisual integration, audio modality, untrimmed video, multimodal fusion, THUMOS14, video action recognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review