The DOI matches its DataCite record; 52 of the 54 references it cites were found in a registry or an index.
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Open accessAbstract
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Squeeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,794
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658
MUSAN: A Music, Speech, and Noise CorpusPreprint
Snyder, D., Chen, G., & Povey, D. (2015) · arXiv (Cornell University) · doi:10.1007/978-3-032-27827-2_35 · Cited by 1,454
A study on data augmentation of reverberant speech for robust speech recognition
Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., & Khudanpur, S. P. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953152 · Cited by 965
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689
S^3FD: Single Shot Scale-Invariant Face Detector
Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., & Li, S. Z. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.30 · Cited by 675
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620
In Defence of Metric Learning for Speaker Recognition
Chung, J. S., Huh, J., Mun, S., Lee, M., Heo, H.-S., Choe, S., Ham, C., Jung, S., Lee, B., & Han, I. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1064 · Cited by 397
Speech Rates in British English
Tauroza, S., & Allison, D. (1990) · Applied Linguistics · doi:10.1093/applin/11.1.90 · Cited by 286
Sequence Level Semantics Aggregation for Video Object Detection
Wu, H., Chen, Y., Wang, N., & Zhang, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00931 · Cited by 282
LRS3-TED: a large-scale dataset for visual speech recognitionPreprint
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00496 · Cited by 263
Self-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_13 · Cited by 221
Spot the Conversation: Speaker Diarisation in the Wild
Chung, J. S., Huh, J., Nagrani, A., Afouras, T., & Zisserman, A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2337 · Cited by 186
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection
Roth, J., Chaudhuri, S., Klejch, O., Marvin, R., Gallagher, A., Kaver, L., Ramaswamy, S., Stopczynski, A., Schmid, C., Xi, Z., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053900 · Cited by 156
Cutts, M. (2004) · Cited by 151
A Convolutional Neural Network Smartphone App for Real-Time Voice Activity Detection
Sehgal, A., & Kehtarnavaz, N. (2018) · IEEE Access · doi:10.1109/access.2018.2800728 · Cited by 120
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682524 · Cited by 108
Cheng, Y., Wang, R., Pan, Z., Feng, R., & Zhang, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413869 · Cited by 106
Self-citationLeveraging Long-Range Temporal Relationships Between Proposals for Video Object Detection
Shvets, M., Liu, W., & Berg, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00985 · Cited by 102
Personal VAD: Speaker-Conditioned Voice Activity Detection
Ding, S., Wang, Q., Chang, S.-Y., Wan, L., & Moreno, I. L. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-62 · Cited by 90
An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks
Ariav, I., & Cohen, I. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2901195 · Cited by 74
Alcázar, J. L., Caba, F., Mai, L., Perazzi, F., Lee, J., Arbeláez, P., & Ghanem, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01248 · Cited by 69
Detecting Aedes aegypti mosquitoes through audio classification with convolutional neural networks
Fernandes, M. S., Cordeiro, W., & Recamonde-Mendoza, M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104152 · Cited by 64
Cross-Modal Supervision for Learning Active Speaker Detection in Video
Chakravarty, P., & Tuytelaars, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_18 · Cited by 59
MAAS: Multi-modal Assignation for Active Speaker Detection
Alcázar, J. L., Heilbron, F. C., Thabet, A., & Ghanem, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00033 · Cited by 57
Multi-Target DoA Estimation with an Audio-Visual Fusion Mechanism
Qian, X., Madhavi, M. C., Pan, Z., Wang, J., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413776 · Cited by 49
Self-citationMuse: Multi-Modal Target Speaker Extraction with Visual Cues
Pan, Z., Tao, R., Xu, C., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414023 · Cited by 46
Self-citationMultimodal Multi-Channel On-Line Speaker Diarization Using Sensor Fusion Through SVM
Minotto, V. P., Jung, C. R., & Lee, B. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2463722 · Cited by 41
Who Said That?: Audio-Visual Speaker Diarisation of Real-World Meetings
Chung, J. S., Lee, B., & Han, I. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3116 · Cited by 36
Visual Voice Activity Detection in the Wild
Patrona, F., Iosifidis, A., Tefas, A., Nikolaidis, N., & Pitas, I. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2535357 · Cited by 35
End-to-end audiovisual speech activity detection with bimodal recurrent neural models
Tao, F., & Busso, C. (2019) · Speech Communication · doi:10.1016/j.specom.2019.07.003 · Cited by 35
Chakravarty, P., Mirzaei, S., Tuytelaars, T., & hamme, H. V. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2820780 · Cited by 34
Audio-Visual Tracking of Concurrent Speakers
Qian, X., Brutti, A., Lanz, O., Omologo, M., & Cavallaro, A. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3061800 · Cited by 34
Self-citationS-VVAD: Visual Voice Activity Detection by Motion Segmentation
Shahid, M., Beyan, C., & Murino, V. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00238 · Cited by 32
Bimodal Recurrent Neural Network for Audiovisual Voice Activity Detection
Tao, F., & Busso, C. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1573 · Cited by 27
Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)Preprint
Chung, J. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10555 · Cited by 26
RealVAD: A Real-World Dataset and A Method for Voice Activity Detection by Body Motion Analysis
Beyan, C., Shahid, M., & Murino, V. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3007350 · Cited by 25
Improved Active Speaker Detection based on Optical Flow
Huang, C. X., & Koishida, K. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00483 · Cited by 18
Randomly Weighted CNNs for (Music) Audio Classification
Pons, J., & Serra, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682912 · Cited by 10
End to End Lip Synchronization with a Temporal AutoEncoder
Shalev, Y., & Wolf, L. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093490 · Cited by 10
Crossmodal learning for audio-visual speech event localizationPreprint
Sharma, R. A., Somandepalli, K., & Narayanan, S. S. (2020) · arXiv (Cornell University) · Cited by 10
Comparisons of Visual Activity Primitives for Voice Activity Detection
Shahid, M., Beyan, C., & Murino, V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30642-7_5 · Cited by 9
Flagged first, then by citations
2 of the 54 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. The Conversation: deep audio-visual speech enhancement. Proc. Interspeech (2018), 3244-3248.
- Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. 2017. S3FD: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference on computer vision (ICCV). 192-201.
Cited by
Sorted by citations
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
Self-citationSub-word Level Lip Reading With Visual Attention
R, P. K., Afouras, T., & Zisserman, A. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00510 · Cited by 137
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116
Self-citationSelf-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72
Metadata
- Authors
- Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H.
- Affiliations
- National University of Singapore (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 3927-3935
- arXiv
- 2107.06592
- OpenAlex
- W3206008172
- Cited by
- 156
- References
- 54
- Citations by year
- 2026: 16 · 2025: 35 · 2024: 34 · 2023: 39 · 2022: 30 · 2021: 2
- FWCI
- 15.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- active speaker detection, auditory-visual interaction, self-attention, temporal encoder, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
