The DOI matches its DataCite record; 52 of the 54 references it cites were found in a registry or an index.
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Open accessAbstract
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Squeeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,794
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658
MUSAN: A Music, Speech, and Noise CorpusPreprint
Snyder, D., Chen, G., & Povey, D. (2015) · arXiv (Cornell University) · doi:10.1007/978-3-032-27827-2_35 · Cited by 1,454
A study on data augmentation of reverberant speech for robust speech recognition
Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., & Khudanpur, S. P. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953152 · Cited by 965
Research works printed but not linked: 2
- Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. The Conversation: deep audio-visual speech enhancement. Proc. Interspeech (2018), 3244-3248.
- Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. 2017. S3FD: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference on computer vision (ICCV). 192-201.
Cited by
Sorted by citations
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
Self-citationSub-word Level Lip Reading With Visual Attention
R, P. K., Afouras, T., & Zisserman, A. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00510 · Cited by 137
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116
Self-citationSelf-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72
Metadata
- Authors
- Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H.
- Affiliations
- National University of Singapore (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 3927-3935
- arXiv
- 2107.06592
- OpenAlex
- W3206008172
- Cited by
- 156
- References
- 54
- Citations by year
- 2026: 16 · 2025: 35 · 2024: 34 · 2023: 39 · 2022: 30 · 2021: 2
- FWCI
- 15.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- active speaker detection, auditory-visual interaction, self-attention, temporal encoder, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
