Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Open accessAbstract
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Squeeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · see citing works
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,837 · see citing works
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,698 · see citing works
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634 · see citing works
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256 · see citing works
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · see citing works
A study on data augmentation of reverberant speech for robust speech recognition
Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., & Khudanpur, S. P. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953152 · Cited by 914 · see citing works
Perceptual Signal Analysis and Eigenvalue-Weighted Metrics for Musical Audio Quality Assessment
Majumder, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-27827-2_35 · Cited by 904 · see citing works
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · see citing works
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704 · see citing works
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · see citing works
S^3FD: Single Shot Scale-Invariant Face Detector
Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., & Li, S. Z. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.30 · Cited by 662 · see citing works
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · see citing works
In Defence of Metric Learning for Speaker Recognition
Chung, J. S., Huh, J., Mun, S., Lee, M., Heo, H.-S., Choe, S., Ham, C., Jung, S., Lee, B., & Han, I. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1064 · Cited by 397 · see citing works
Sequence Level Semantics Aggregation for Video Object Detection
Wu, H., Chen, Y., Wang, N., & Zhang, Z. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00931 · Cited by 282 · see citing works
Speech Rates in British English
Tauroza, S., & Allison, D. (1990) · Applied Linguistics · doi:10.1093/applin/11.1.90 · Cited by 278 · see citing works
LRS3-TED: a large-scale dataset for visual speech recognitionPreprint
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00496 · Cited by 263 · see citing works
Self-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_13 · Cited by 221 · see citing works
Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection
Roth, J., Chaudhuri, S., Klejch, O., Marvin, R., Gallagher, A., Kaver, L., Ramaswamy, S., Stopczynski, A., Schmid, C., Xi, Z., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053900 · Cited by 156 · see citing works
Spot the Conversation: Speaker Diarisation in the Wild
Chung, J. S., Huh, J., Nagrani, A., Afouras, T., & Zisserman, A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2337 · Cited by 155 · see citing works
Cutts, M. (2004) · Cited by 151 · see citing works
A Convolutional Neural Network Smartphone App for Real-Time Voice Activity Detection
Sehgal, A., & Kehtarnavaz, N. (2018) · IEEE Access · doi:10.1109/access.2018.2800728 · Cited by 122 · see citing works
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682524 · Cited by 108 · see citing works
Cheng, Y., Wang, R., Pan, Z., Feng, R., & Zhang, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413869 · Cited by 103 · see citing works
Self-citationLeveraging Long-Range Temporal Relationships Between Proposals for Video Object Detection
Shvets, M., Liu, W., & Berg, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00985 · Cited by 102 · see citing works
Personal VAD: Speaker-Conditioned Voice Activity Detection
Ding, S., Wang, Q., Chang, S.-Y., Wan, L., & Moreno, I. L. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-62 · Cited by 90 · see citing works
An End-to-End Multimodal Voice Activity Detection Using WaveNet Encoder and Residual Networks
Ariav, I., & Cohen, I. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2901195 · Cited by 73 · see citing works
Alcázar, J. L., Caba, F., Mai, L., Perazzi, F., Lee, J., Arbeláez, P., & Ghanem, B. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01248 · Cited by 66 · see citing works
Detecting Aedes aegypti mosquitoes through audio classification with convolutional neural networks
Fernandes, M. S., Cordeiro, W., & Recamonde-Mendoza, M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104152 · Cited by 64 · see citing works
Cross-Modal Supervision for Learning Active Speaker Detection in Video
Chakravarty, P., & Tuytelaars, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_18 · Cited by 59 · see citing works
MAAS: Multi-modal Assignation for Active Speaker Detection
Alcázar, J. L., Heilbron, F. C., Thabet, A., & Ghanem, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00033 · Cited by 50 · see citing works
Multi-Target DoA Estimation with an Audio-Visual Fusion Mechanism
Qian, X., Madhavi, M. C., Pan, Z., Wang, J., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413776 · Cited by 49 · see citing works
Self-citationMuse: Multi-Modal Target Speaker Extraction with Visual Cues
Pan, Z., Tao, R., Xu, C., & Li, H. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414023 · Cited by 46 · see citing works
Self-citationMultimodal Multi-Channel On-Line Speaker Diarization Using Sensor Fusion Through SVM
Minotto, V. P., Jung, C. R., & Lee, B. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2463722 · Cited by 41 · see citing works
Who Said That?: Audio-Visual Speaker Diarisation of Real-World Meetings
Chung, J. S., Lee, B., & Han, I. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3116 · Cited by 36 · see citing works
Visual Voice Activity Detection in the Wild
Patrona, F., Iosifidis, A., Tefas, A., Nikolaidis, N., & Pitas, I. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2535357 · Cited by 35 · see citing works
End-to-end audiovisual speech activity detection with bimodal recurrent neural models
Tao, F., & Busso, C. (2019) · Speech Communication · doi:10.1016/j.specom.2019.07.003 · Cited by 35 · see citing works
Chakravarty, P., Mirzaei, S., Tuytelaars, T., & hamme, H. V. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2820780 · Cited by 34 · see citing works
Audio-Visual Tracking of Concurrent Speakers
Qian, X., Brutti, A., Lanz, O., Omologo, M., & Cavallaro, A. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3061800 · Cited by 34 · see citing works
Self-citationS-VVAD: Visual Voice Activity Detection by Motion Segmentation
Shahid, M., Beyan, C., & Murino, V. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00238 · Cited by 32 · see citing works
Bimodal Recurrent Neural Network for Audiovisual Voice Activity Detection
Tao, F., & Busso, C. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1573 · Cited by 27 · see citing works
Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)Preprint
Chung, J. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10555 · Cited by 26 · see citing works
RealVAD: A Real-World Dataset and A Method for Voice Activity Detection by Body Motion Analysis
Beyan, C., Shahid, M., & Murino, V. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3007350 · Cited by 25 · see citing works
Improved Active Speaker Detection based on Optical Flow
Huang, C. X., & Koishida, K. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00483 · Cited by 17 · see citing works
Randomly Weighted CNNs for (Music) Audio Classification
Pons, J., & Serra, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682912 · Cited by 10 · see citing works
End to End Lip Synchronization with a Temporal AutoEncoder
Shalev, Y., & Wolf, L. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093490 · Cited by 10 · see citing works
Crossmodal learning for audio-visual speech event localizationPreprint
Sharma, R. A., Somandepalli, K., & Narayanan, S. S. (2020) · arXiv (Cornell University) · Cited by 10 · see citing works
Comparisons of Visual Activity Primitives for Voice Activity Detection
Shahid, M., Beyan, C., & Murino, V. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30642-7_5 · Cited by 9 · see citing works
Flagged first, then by citations
2 of the 54 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. The Conversation: deep audio-visual speech enhancement. Proc. Interspeech (2018), 3244-3248.
- D. Snyder, G. Chen, and D. Povey. 2015. MUSAN: A Music, Speech, and Noise Corpus. CoRR abs/1510.08484 (2015). http://arxiv.org/abs/1510.08484 · arXiv
- Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. 2017. S3FD: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference on computer vision (ICCV). 192-201.
Cited by
Sorted by citations
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630 · see citing works
Self-citationSub-word Level Lip Reading With Visual Attention
Prajwal, K. R., Afouras, T., & Zisserman, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00510 · Cited by 115 · see citing works
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116 · see citing works
Self-citationSelf-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101 · see citing works
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72 · see citing works
MIntRec: A New Dataset for Multimodal Intent Recognition
Zhang, H., Xu, H., Wang, X. Y., Zhou, Q., Zhao, S., & Teng, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547906 · Cited by 70 · see citing works
M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database
Zhao, J., Zhang, T., Hu, J., Liu, Y., Jin, Q., Wang, X., & Li, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.391 · Cited by 72 · see citing works
Self-citationMake Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64 · see citing works
A Light Weight Model for Active Speaker Detection
Liao, J., Duan, H., Feng, K., Zhao, W., Yang, Y., & Chen, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02196 · Cited by 60 · see citing works
USEV: Universal Speaker Extraction With Visual Cue
Pan, Z., Ge, M., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3205759 · Cited by 54 · see citing works
Self-citationZheng, W., Yu, J., Xia, R., & Wang, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.861 · Cited by 51 · see citing works
“In the first stage, we design a pipeline solution to obtain the face sequence of the real speaker, which contains three steps: 1) Extract the face sequence containing all possible speakers based on the combination of multimodal rules and an active speaker detection model (Tao et al., 2021); 2) Identify the number of f…”
BackgroundMethodInfluentialLearning Long-Term Spatial-Temporal Graphs for Active Speaker Detection
Min, K., Roy, S., Tripathi, S., Guha, T., & Majumdar, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_22 · Cited by 46 · see citing works
Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization
Jiang, H., Murdock, C., & Ithapu, V. K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01029 · Cited by 46 · see citing works
Audio-Visual Cross-Attention Network for Robotic Speaker Tracking
Qian, X., Wang, Z., Wang, J., Guan, G., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3226330 · Cited by 42 · see citing works
Self-citationSelective Listening by Synchronizing Speech With Lips
Pan, Z., Tao, R., Xu, C., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153258 · Cited by 43 · see citing works
Self-citationKhoo, W. Y. H., Hsu, L.-J., Amon, K. J., Chakilam, P. V., Chen, W.-C., Kaufman, Z., Lungu, A., Sato, H., Seliger, E., Swaminathan, M., et al. (2023) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3568294.3580067 · Cited by 39 · see citing works
AVA-AVD: Audio-visual Speaker Diarization in the Wild
Xu, Z., Song, Z., Tsutsui, S., Feng, C., Ye, M., & Shou, M. Z. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548027 · Cited by 35 · see citing works
Self-citationEnd-to-End Active Speaker Detection
Alcázar, J. L., Cordes, M., Zhao, C., & Ghanem, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_8 · Cited by 35 · see citing works
LoCoNet: Long-Short Context Network for Active Speaker Detection
Wang, X., Cheng, F., & Bertasius, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01747 · Cited by 33 · see citing works
Speaker Extraction With Co-Speech Gestures Cue
Pan, Z., Qian, X., & Li, H. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3175130 · Cited by 28 · see citing works
Self-citationLook&listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement
Xiong, J., Yu, Z., Zhang, P., Xie, L., Huang, W., & Zha, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3199109 · Cited by 28 · see citing works
Asd-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers
Datta, G., Etchart, T., Yadav, V., Hedau, V., Natarajan, P., & Chang, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746991 · Cited by 24 · see citing works
Chen, C., Richard, A., Shapovalov, R., Ithapu, V. K., Neverova, N., Grauman, K. L., & Vedaldi, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00620 · Cited by 23 · see citing works
Liu, M., Wang, J., Qian, X., & Li, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3326694 · Cited by 24 · see citing works
Self-citationA Deep Learning-Based GPR Forward Solver for Predicting B-Scans of Subsurface Objects
Dai, Q., Lee, Y. H., Sun, H.-H., Qian, J., Ow, G., Yusof, M. L. M., & Yücel, A. C. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3192003 · Cited by 22 · see citing works
Av-Sepformer: Cross-Attention Sepformer for Audio-Visual Target Speaker Extraction
Lin, J., Cai, X., Dinkel, H., Chen, J., Yan, Z., Wang, Y., Zhang, J., Wu, Z., Wang, Y., & Meng, H. M. L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094306 · Cited by 22 · see citing works
Chae, H., Lee, J., 김종관, Lee, S., Lee, J., Chung, Y., & Park, D. (2024) · Sensors · doi:10.3390/s24227232 · Cited by 23 · see citing works
AV-FDTI: Audio-visual fusion for drone threat identification
Yang, Y., Yuan, S., Yang, J., Nguyen, T. H., Cao, M., Nguyen, T., Wang, H., & Xie, L. (2024) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2024.06.002 · Cited by 20 · see citing works
Rethinking Audio-Visual Synchronization for Active Speaker Detection
Wuerkaixi, A., Zhang, Y., Duan, Z., & Zhang, C. (2022) · Machine learning for signal processing ... · doi:10.1109/mlsp55214.2022.9943352 · Cited by 19 · see citing works
Time-Domain Speech Separation Networks With Graph Encoding Auxiliary
Wang, T., Pan, Z., Ge, M., Yang, Z., & Li, H. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3243764 · Cited by 19 · see citing works
Self-citation3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
Chen, Y., Zheng, S., Wang, H., Cheng, L., Zhu, T., Huang, R., Deng, C., Chen, Q., Zhang, S., Wang, W., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888389 · Cited by 20 · see citing works
Egocentric Auditory Attention Localization in Conversations
Ryan, F., Jiang, H., Shukla, A., Rehg, J. M., & Ithapu, V. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01409 · Cited by 18 · see citing works
Deep Audio-Visual Beamforming for Speaker Localization
Qian, X., Zhang, Q., Guan, G., & Xue, W. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3165466 · Cited by 16 · see citing works
Self-citationPaul, S. K., Nicolescu, M., & Nicolescu, M. N. (2023) · Sensors · doi:10.3390/s23135798 · Cited by 15 · see citing works
Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs
Tao, R., Lee, K. A., Das, R. K., Hautamäki, V., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268568 · Cited by 13 · see citing works
Self-citationEgocentric Video Task Translation
Xue, Z., Song, Y., Grauman, K. L., & Torresani, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00229 · Cited by 13 · see citing works
CMOSE: Comprehensive Multi-Modality Online Student Engagement Dataset with High-Quality Labels
Wu, C.-H., Liu, S.-Y., Huang, X., Wang, X., Zhang, R., Minciullo, L., Yiu, W. K., Kwan, K., & Cheng, K.-T. T. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00466 · Cited by 14 · see citing works
Gu, X., Ou, L., Ong, D., & Wang, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548411 · Cited by 12 · see citing works
ImagineNet: Target Speaker Extraction with Intermittent Visual Cue Through Embedding Inpainting
Pan, Z., Wang, W., Borsdorf, M., & Li, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097232 · Cited by 10 · see citing works
Self-citationA multi-purpose audio-visual corpus for multi-modal Persian speech recognition: The Arman-AV dataset
Peymanfard, J., Heydarian, S., Lashini, A., Zeinali, H., Mohammadi, M. R., & Mozayani, N. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121648 · Cited by 11 · see citing works
Audio-Visual Fusion With Temporal Convolutional Attention Network for Speech Separation
Liu, D., Zhang, T., Christensen, M. G., Yi, C., & An, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3463411 · Cited by 9 · see citing works
SelM: Selective Mechanism based Audio-Visual Segmentation
Li, J., Yu, S., Wang, Y., Wang, L., & Lu, H. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680926 · Cited by 9 · see citing works
AdVerb: Visually Guided Audio Dereverberation
Chowdhury, S., Ghosh, S., Dasgupta, S., Ratnarajah, A. J., Tyagi, U., & Manocha, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00725 · Cited by 9 · see citing works
DyViSE: Dynamic Vision-Guided Speaker Embedding for Audio-Visual Speaker Diarization
Wuerkaixi, A., Yan, K., Zhang, Y., Duan, Z., & Zhang, C. (2022) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp55362.2022.9948860 · Cited by 9 · see citing works
Gao, J., Zhou, S., Yu, H., Li, C., & Hu, X. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104551 · Cited by 8 · see citing works
VCSE: Time-Domain Visual-Contextual Speaker Extraction Network
Li, J., Ge, M., Pan, Z., Wang, L., & Dang, J. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11183 · Cited by 8 · see citing works
Self-citationSound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation
Chen, Z., Qian, S., & Owens, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00726 · Cited by 9 · see citing works
WASD: A Wilder Active Speaker Detection Dataset
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3412821 · Cited by 8 · see citing works
MMAL: Multi-Modal Analytic Learning for Exemplar-Free Audio-Visual Class Incremental Tasks
Yue, X., Zhang, X., Chen, Y., Zhang, C., Lao, M., Zhuang, H., Qian, X., & Li, H. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681607 · Cited by 7 · see citing works
Self-citationLR-ASD: Lightweight and Robust Network for Active Speaker Detection
Liao, J., Duan, H., Feng, K., Zhao, W., Yang, Y., Chen, L., & Chen, Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02399-2 · Cited by 8 · see citing works
Gurvich, I., Leichter, I., Palle, D. R., Asher, Y., Vinnikov, A., Abramovski, I., Gopal, V., Cutler, R., & Krupka, E. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446169 · Cited by 7 · see citing works
Whose emotion matters? Speaking activity localisation without prior knowledge
Carneiro, H., Weber, C., & Wermter, S. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126271 · Cited by 7 · see citing works
Leveraging Visual Supervision for Array-Based Active Speaker Detection and Localization
Berghi, D., & Jackson, P. J. B. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3346643 · Cited by 7 · see citing works
Chae, H., Park, M., & Noh, B. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110909 · Cited by 6 · see citing works
Self-Motion As Supervision For Egocentric Audiovisual Localization
Murdock, C., Ananthabhotla, I., Lü, H., & Ithapu, V. K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447683 · Cited by 6 · see citing works
Efficient time-domain speech separation using short encoded sequence network
Liu, D., Zhang, T., Christensen, M. G., Ma, B., & Deng, P. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103150 · Cited by 6 · see citing works
Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection
Sharma, R. A., & Narayanan, S. S. (2023) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2023.3267269 · Cited by 6 · see citing works
Multimodal Transformer Distillation for Audio-Visual Synchronization
Chen, X., Wu, H., Wang, C.-C., Lee, H.-Y., & Jang, J. R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446372 · Cited by 6 · see citing works
Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos
Majumder, S., Al-Halah, Z., & Grauman, K. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02555 · Cited by 5 · see citing works
Word-level Persian Lipreading Dataset
Peymanfard, J., Lashini, A., Heydarian, S., Zeinali, H., & Mozayani, N. (2022) · International Conference on Computer and Knowledge Engineering · doi:10.1109/iccke57176.2022.9960105 · Cited by 5 · see citing works
On Exploring Audio Anomaly in Speech
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2023) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs58808.2023.10374734 · Cited by 5 · see citing works
Detecting Autism from Head Movements using Kinesics
Gökmen, M., Sarıyanidi, E., Yankowitz, L. D., Zampella, C., Schultz, R. T., & Tunç, B. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685711 · Cited by 4 · see citing works
BIAS: A Body-Based Interpretable Active Speaker Approach
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3520030 · Cited by 4 · see citing works
Girmaji, R., Beri, B., Subramanian, R., & Gandhi, V. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712113 · Cited by 4 · see citing works
Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings
Clarke, J., Gotoh, Y., & Goetze, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890414 · Cited by 4 · see citing works
Zhou, X., Song, X., Wu, H., Zhang, J., & Xu, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548383 · Cited by 4 · see citing works
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection
Chen, X., Wu, H., Meng, H. M. L., Lee, H.-Y., & Jang, J. R. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022646 · Cited by 4 · see citing works
Tesema, F. B., Gu, J. J., Song, W., Wu, H. Q., Zhu, S., Lin, Z., Huang, M., Wang, W., & Kumar, R. (2023) · IEEE Systems Man and Cybernetics Magazine · doi:10.1109/msmc.2022.3224843 · Cited by 4 · see citing works
KMSAV: Korean multi‐speaker spontaneous audiovisual dataset
Park, K., Oh, C., & Dong, S. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0352 · Cited by 4 · see citing works
Clarke, J., Gotoh, Y., & Goetze, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389764 · Cited by 3 · see citing works
Robust Audio–Visual Speaker Localization in Noisy Aircraft Cabins for Inflight Medical Assistance
Qin, Q., & Zhu, Y. (2025) · Sensors · doi:10.3390/s25185827 · Cited by 3 · see citing works
Efficient Audiovisual Fusion for Active Speaker Detection
Tesema, F. B., Gu, J. J., Song, W., Wu, H. B., Zhu, S., & Lin, Z. (2023) · IEEE Access · doi:10.1109/access.2023.3267668 · Cited by 3 · see citing works
Feature Alignment and Reconstruction Constraints for Multimodal Sentiment Analysis
Zhu, Q., Lan, T., Liang, J. P., Xiang, D., & He, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650976 · Cited by 3 · see citing works
Spherical World-Locking for Audio-Visual Localization in Egocentric Videos
Yun, H., Gao, R., Ananthabhotla, I., Kumar, A. K., Donley, J., Li, C., Kim, G., Ithapu, V. K., & Murdock, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72691-0_15 · Cited by 3 · see citing works
Multi-modal, multi-step human–robot interaction method for natural interaction of service robots
Yura, J., Byambasuren, B.-E., & Kim, D. (2025) · Intelligent Service Robotics · doi:10.1007/s11370-025-00604-2 · Cited by 3 · see citing works
Fanaras, K., Tragoudaras, A., Antoniadis, C., & Massoud, Y. (2022) · IEEE International Midwest Symposium on Circuits and Systems (MWSCAS) · doi:10.1109/mwscas54063.2022.9859533 · Cited by 3 · see citing works
Tran, M., Kim, Y., Su, C.-C., Kuo, C.-H., Sun, M., & Soleymani, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_1 · Cited by 2 · see citing works
Zhu, C., Chen, M., Li, H., Zhang, S., Liang, H., Sun, C., Liu, Y., & Chen, J. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10678-1 · Cited by 2 · see citing works
Self-citationASDnB: Merging Face with Body Cues For Robust Active Speaker Detection
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2025) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb65343.2025.11410746 · Cited by 2 · see citing works
Geeroms, W., Allebosch, G., Kindt, S., Kadri, L., Veelaert, P., & Madhu, N. (2022) · Sensor Data Fusion: Trends, Solutions, Applications (SDF) · doi:10.1109/sdf55338.2022.9931697 · Cited by 2 · see citing works
E-Talk: Accelerating Active Speaker Detection with Audio-Visual Fusion and Edge-Cloud Computing
Yu, X., Zhang, L., & Li, X. (2023) · IEEE International Conference on Sensing, Communication, and Networking (SECON) · doi:10.1109/secon58729.2023.10287518 · Cited by 2 · see citing works
AS-Net: active speaker detection using deep audio-visual attention
Radman, A., & Laaksonen, J. T. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18457-9 · Cited by 2 · see citing works
Long-Term Social Interaction Context: The Key to Egocentric Addressee Detection
Kong, D., Khan, F. M., Zhang, X., Singhal, P., & Wu, Y. N. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447323 · Cited by 2 · see citing works
Evaluation of end-to-end continuous spanish lipreading in different data conditions
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09809-4 · Cited by 2 · see citing works
Referring Expression Comprehension in semi-structured human–robot interaction
Jin, T., Meng, Q., Zhang, G., Huang, Q., Guo, F., Kong, S., Song, W., Zhu, J., & Gu, J. J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126965 · Cited by 2 · see citing works
Recognizing Indonesian words based on visual cues of lip movement using deep learning
Rahmatullah, G. M., Ruan, S.-J., & Li, L. P. (2025) · Measurement · doi:10.1016/j.measurement.2025.116968 · Cited by 2 · see citing works
Advancing Active Speaker Detection for Egocentric Videos
Huh, J., Ortiz, J. A., Kumar, A., Pandey, A., Aroudi, A., Wong, D. D. E., Nesta, F., Xu, B., & Donley, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888166 · Cited by 2 · see citing works
SLoClas: A Database for Joint Sound Localization and Classification
Qian, X., Sharma, B., Abridi, A. E., & Li, H. (2021) · International Conference on Speech Database and Assessments · doi:10.1109/o-cocosda202152914.2021.9660517 · Cited by 2 · see citing works
Self-citationDoes Audio help in deep Audio-Visual Saliency prediction models?
Agrawal, R., Jyoti, S., Girmaji, R., Sivaprasad, S., & Gandhi, V. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556625 · Cited by 2 · see citing works
Multi-modality Multi-attention Network for Ventricular Arrhythmia Classification
Xu, Z., Zang, M., Liu, T., Zhou, S., Liu, C., & Wang, Q. (2023) · International Conference on Bioinformatics and Intelligent Computing (BIC) · doi:10.1145/3592686.3592746 · Cited by 2 · see citing works
Audio Inputs for Active Speaker Detection and Localization Via Microphone Array
Berghi, D., & Jackson, P. J. B. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248185 · Cited by 2 · see citing works
Uncertainty-Guided End-to-End Audio-Visual Speaker Diarization for Far-Field Recordings
Yang, C., Chen, M., Wang, Y., & Wang, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612424 · Cited by 2 · see citing works
Predicting Individual Well-Being in Teamwork Contexts Based on Multi-Modal Speech FeaturesPreprint
Zeulner, T., Hagerer, G., Mueller, M., Vazquez, I., & Gloor, P. A. (2024) · Preprints.org · doi:10.20944/preprints202401.2030.v1 · Cited by 2 · see citing works
CASA-Net: Cross-attention and Self-attention for End-to-End Audio-visual Speaker Diarization
Zhou, H., Li, T., Wang, J., Li, L., & Hong, Q. (2023) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc58517.2023.10317320 · Cited by 1 · see citing works
A Two-Stage Audio-Visual Fusion Piano Transcription Model Based on the Attention Mechanism
Li, Y., Wang, X., Wu, R., Xu, W., & Cheng, W. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426303 · Cited by 1 · see citing works
Jawade, B., Gadde, R. T., Bejjani, C., & Lan, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888378 · Cited by 1 · see citing works
Li, Y., Zhou, X., He, Z., Yu, W., & Luo, Y. H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888235 · Cited by 1 · see citing works
Audio-Faces Intra-Frame Alignment with Graph Attention Networks for Active Speaker Detection
Yin, Y., Yang, X., Liang, L., Xu, L., & Zou, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889368 · Cited by 1 · see citing works
Multimodal Audio-Visual Emotion Recognition for Social Robotics
Simone, G. D., Greco, L., Saggese, A., & Vento, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05060-1_21 · Cited by 1 · see citing works
Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings
Clarke, J., Gotoh, Y., & Goetze, S. (2025) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco63237.2025.11226795 · Cited by 1 · see citing works
- Authors
- Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H.
- Affiliations
- National University of Singapore (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 3927-3935
- arXiv
- 2107.06592
- OpenAlex
- W3206008172
- Cited by
- 156
- References
- 54
- Citations by year
- 2026: 16 · 2025: 35 · 2024: 34 · 2023: 39 · 2022: 30 · 2021: 2
- FWCI
- 15.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- active speaker detection, auditory-visual interaction, self-attention, temporal encoder, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
