The DOI matches its DataCite record; 52 of the 54 references it cites were found in a registry or an index.
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Open accessAbstract
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Squeeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,794
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658
MUSAN: A Music, Speech, and Noise CorpusPreprint
Snyder, D., Chen, G., & Povey, D. (2015) · arXiv (Cornell University) · doi:10.1007/978-3-032-27827-2_35 · Cited by 1,454
A study on data augmentation of reverberant speech for robust speech recognition
Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., & Khudanpur, S. P. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953152 · Cited by 965
Research works printed but not linked: 2
- Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. The Conversation: deep audio-visual speech enhancement. Proc. Interspeech (2018), 3244-3248.
- Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. 2017. S3FD: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference on computer vision (ICCV). 192-201.
Cited by
Sorted by citations
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
Self-citationSub-word Level Lip Reading With Visual Attention
R, P. K., Afouras, T., & Zisserman, A. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00510 · Cited by 137
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Wang, J., Qian, X., Zhang, M., Tan, R. T., & Li, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01408 · Cited by 116
Self-citationSelf-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72
MIntRec: A New Dataset for Multimodal Intent Recognition
Zhang, H., Xu, H., Wang, X. Y., Zhou, Q., Zhao, S., & Teng, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547906 · Cited by 77
M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database
Zhao, J., Zhang, T., Hu, J., Liu, Y., Jin, Q., Wang, X., & Li, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.391 · Cited by 72
Self-citationMake Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., & Gao, K. (2022) · Proceedings of the 2022 International Conference on Multimodal Interaction · doi:10.1145/3536221.3556630 · Cited by 64
A Light Weight Model for Active Speaker Detection
Liao, J., Duan, H., Feng, K., Zhao, W., Yang, Y., & Chen, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02196 · Cited by 60
USEV: Universal Speaker Extraction With Visual Cue
Pan, Z., Ge, M., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3205759 · Cited by 54
Self-citationZheng, W., Yu, J., Xia, R., & Wang, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.861 · Cited by 51
“In the first stage, we design a pipeline solution to obtain the face sequence of the real speaker, which contains three steps: 1) Extract the face sequence containing all possible speakers based on the combination of multimodal rules and an active speaker detection model (Tao et al., 2021); 2) Identify the number of f…”
BackgroundMethodInfluentialLearning Long-Term Spatial-Temporal Graphs for Active Speaker Detection
Min, K., Roy, S., Tripathi, S., Guha, T., & Majumdar, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_22 · Cited by 46
Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localization
Jiang, H., Murdock, C., & Ithapu, V. K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01029 · Cited by 46
Audio-Visual Cross-Attention Network for Robotic Speaker Tracking
Qian, X., Wang, Z., Wang, J., Guan, G., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3226330 · Cited by 42
Self-citationSelective Listening by Synchronizing Speech With Lips
Pan, Z., Tao, R., Xu, C., & Li, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153258 · Cited by 43
Self-citationKhoo, W. Y. H., Hsu, L.-J., Amon, K. J., Chakilam, P. V., Chen, W.-C., Kaufman, Z., Lungu, A., Sato, H., Seliger, E., Swaminathan, M., et al. (2023) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3568294.3580067 · Cited by 39
AVA-AVD: Audio-visual Speaker Diarization in the Wild
Xu, Z., Song, Z., Tsutsui, S., Feng, C., Ye, M., & Shou, M. Z. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548027 · Cited by 35
Self-citationEnd-to-End Active Speaker Detection
Alcázar, J. L., Cordes, M., Zhao, C., & Ghanem, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_8 · Cited by 38
LoCoNet: Long-Short Context Network for Active Speaker Detection
Wang, X., Cheng, F., & Bertasius, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01747 · Cited by 33
Speaker Extraction With Co-Speech Gestures Cue
Pan, Z., Qian, X., & Li, H. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3175130 · Cited by 28
Self-citationLook&listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement
Xiong, J., Yu, Z., Zhang, P., Xie, L., Huang, W., & Zha, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3199109 · Cited by 28
Asd-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers
Datta, G., Etchart, T., Yadav, V., Hedau, V., Natarajan, P., & Chang, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746991 · Cited by 24
Chen, C., Richard, A., Shapovalov, R., Ithapu, V. K., Neverova, N., Grauman, K. L., & Vedaldi, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00620 · Cited by 23
Liu, M., Wang, J., Qian, X., & Li, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3326694 · Cited by 30
Self-citationA Deep Learning-Based GPR Forward Solver for Predicting B-Scans of Subsurface Objects
Dai, Q., Lee, Y. H., Sun, H.-H., Qian, J., Ow, G., Yusof, M. L. M., & Yücel, A. C. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3192003 · Cited by 22
Av-Sepformer: Cross-Attention Sepformer for Audio-Visual Target Speaker Extraction
Lin, J., Cai, X., Dinkel, H., Chen, J., Yan, Z., Wang, Y., Zhang, J., Wu, Z., Wang, Y., & Meng, H. M. L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094306 · Cited by 22
Chae, H., Lee, J., 김종관, Lee, S., Lee, J., Chung, Y., & Park, D. (2024) · Sensors · doi:10.3390/s24227232 · Cited by 25
AV-FDTI: Audio-visual fusion for drone threat identification
Yang, Y., Yuan, S., Yang, J., Nguyen, T. H., Cao, M., Nguyen, T., Wang, H., & Xie, L. (2024) · Journal of Automation and Intelligence · doi:10.1016/j.jai.2024.06.002 · Cited by 20
Rethinking Audio-Visual Synchronization for Active Speaker Detection
Wuerkaixi, A., Zhang, Y., Duan, Z., & Zhang, C. (2022) · Machine learning for signal processing ... · doi:10.1109/mlsp55214.2022.9943352 · Cited by 19
Time-Domain Speech Separation Networks With Graph Encoding Auxiliary
Wang, T., Pan, Z., Ge, M., Yang, Z., & Li, H. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3243764 · Cited by 20
Self-citation3D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarization
Chen, Y., Zheng, S., Wang, H., Cheng, L., Zhu, T., Huang, R., Deng, C., Chen, Q., Zhang, S., Wang, W., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888389 · Cited by 32
Egocentric Auditory Attention Localization in Conversations
Ryan, F., Jiang, H., Shukla, A., Rehg, J. M., & Ithapu, V. K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01409 · Cited by 18
Deep Audio-Visual Beamforming for Speaker Localization
Qian, X., Zhang, Q., Guan, G., & Xue, W. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3165466 · Cited by 16
Self-citationPaul, S. K., Nicolescu, M., & Nicolescu, M. N. (2023) · Sensors · doi:10.3390/s23135798 · Cited by 16
Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs
Tao, R., Lee, K. A., Das, R. K., Hautamäki, V., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268568 · Cited by 15
Self-citationEgocentric Video Task Translation
Xue, Z., Song, Y., Grauman, K. L., & Torresani, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00229 · Cited by 15
CMOSE: Comprehensive Multi-Modality Online Student Engagement Dataset with High-Quality Labels
Wu, C.-H., Liu, S.-Y., Huang, X., Wang, X., Zhang, R., Minciullo, L., Yiu, W. K., Kwan, K., & Cheng, K.-T. T. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00466 · Cited by 14
Gu, X., Ou, L., Ong, D., & Wang, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548411 · Cited by 15
ImagineNet: Target Speaker Extraction with Intermittent Visual Cue Through Embedding Inpainting
Pan, Z., Wang, W., Borsdorf, M., & Li, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10097232 · Cited by 10
Self-citationA multi-purpose audio-visual corpus for multi-modal Persian speech recognition: The Arman-AV dataset
Peymanfard, J., Heydarian, S., Lashini, A., Zeinali, H., Mohammadi, M. R., & Mozayani, N. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121648 · Cited by 11
Audio-Visual Fusion With Temporal Convolutional Attention Network for Speech Separation
Liu, D., Zhang, T., Christensen, M. G., Yi, C., & An, Z. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3463411 · Cited by 9
SelM: Selective Mechanism based Audio-Visual Segmentation
Li, J., Yu, S., Wang, Y., Wang, L., & Lu, H. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680926 · Cited by 9
AdVerb: Visually Guided Audio Dereverberation
Chowdhury, S., Ghosh, S., Dasgupta, S., Ratnarajah, A. J., Tyagi, U., & Manocha, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00725 · Cited by 9
DyViSE: Dynamic Vision-Guided Speaker Embedding for Audio-Visual Speaker Diarization
Wuerkaixi, A., Yan, K., Zhang, Y., Duan, Z., & Zhang, C. (2022) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp55362.2022.9948860 · Cited by 9
Gao, J., Zhou, S., Yu, H., Li, C., & Hu, X. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104551 · Cited by 8
VCSE: Time-Domain Visual-Contextual Speaker Extraction Network
Li, J., Ge, M., Pan, Z., Wang, L., & Dang, J. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11183 · Cited by 8
Self-citationSound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation
Chen, Z., Qian, S., & Owens, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00726 · Cited by 9
WASD: A Wilder Active Speaker Detection Dataset
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3412821 · Cited by 8
MMAL: Multi-Modal Analytic Learning for Exemplar-Free Audio-Visual Class Incremental Tasks
Yue, X., Zhang, X., Chen, Y., Zhang, C., Lao, M., Zhuang, H., Qian, X., & Li, H. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681607 · Cited by 7
Self-citationLR-ASD: Lightweight and Robust Network for Active Speaker Detection
Liao, J., Duan, H., Feng, K., Zhao, W., Yang, Y., Chen, L., & Chen, Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02399-2 · Cited by 8
Gurvich, I., Leichter, I., Palle, D. R., Asher, Y., Vinnikov, A., Abramovski, I., Gopal, V., Cutler, R., & Krupka, E. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446169 · Cited by 7
Whose emotion matters? Speaking activity localisation without prior knowledge
Carneiro, H., Weber, C., & Wermter, S. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126271 · Cited by 7
Leveraging Visual Supervision for Array-Based Active Speaker Detection and Localization
Berghi, D., & Jackson, P. J. B. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3346643 · Cited by 7
Chae, H., Park, M., & Noh, B. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.110909 · Cited by 6
Self-Motion As Supervision For Egocentric Audiovisual Localization
Murdock, C., Ananthabhotla, I., Lü, H., & Ithapu, V. K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447683 · Cited by 6
Efficient time-domain speech separation using short encoded sequence network
Liu, D., Zhang, T., Christensen, M. G., Ma, B., & Deng, P. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103150 · Cited by 6
Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection
Sharma, R. A., & Narayanan, S. S. (2023) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2023.3267269 · Cited by 6
Multimodal Transformer Distillation for Audio-Visual Synchronization
Chen, X., Wu, H., Wang, C.-C., Lee, H.-Y., & Jang, J. R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446372 · Cited by 6
Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos
Majumder, S., Al-Halah, Z., & Grauman, K. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02555 · Cited by 9
Word-level Persian Lipreading Dataset
Peymanfard, J., Lashini, A., Heydarian, S., Zeinali, H., & Mozayani, N. (2022) · International Conference on Computer and Knowledge Engineering · doi:10.1109/iccke57176.2022.9960105 · Cited by 5
On Exploring Audio Anomaly in Speech
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2023) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs58808.2023.10374734 · Cited by 5
Detecting Autism from Head Movements using Kinesics
Gökmen, M., Sarıyanidi, E., Yankowitz, L. D., Zampella, C., Schultz, R. T., & Tunç, B. (2024) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3678957.3685711 · Cited by 4
BIAS: A Body-Based Interpretable Active Speaker Approach
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3520030 · Cited by 5
Girmaji, R., Beri, B., Subramanian, R., & Gandhi, V. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712113 · Cited by 6
Speaker Embedding Informed Audiovisual Active Speaker Detection for Egocentric Recordings
Clarke, J., Gotoh, Y., & Goetze, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890414 · Cited by 4
Zhou, X., Song, X., Wu, H., Zhang, J., & Xu, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548383 · Cited by 4
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection
Chen, X., Wu, H., Meng, H. M. L., Lee, H.-Y., & Jang, J. R. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022646 · Cited by 4
Tesema, F. B., Gu, J. J., Song, W., Wu, H. Q., Zhu, S., Lin, Z., Huang, M., Wang, W., & Kumar, R. (2023) · IEEE Systems Man and Cybernetics Magazine · doi:10.1109/msmc.2022.3224843 · Cited by 4
KMSAV: Korean multi‐speaker spontaneous audiovisual dataset
Park, K., Oh, C., & Dong, S. (2024) · ETRI Journal · doi:10.4218/etrij.2023-0352 · Cited by 4
Clarke, J., Gotoh, Y., & Goetze, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389764 · Cited by 5
Robust Audio–Visual Speaker Localization in Noisy Aircraft Cabins for Inflight Medical Assistance
Qin, Q., & Zhu, Y. (2025) · Sensors · doi:10.3390/s25185827 · Cited by 3
Efficient Audiovisual Fusion for Active Speaker Detection
Tesema, F. B., Gu, J. J., Song, W., Wu, H. B., Zhu, S., & Lin, Z. (2023) · IEEE Access · doi:10.1109/access.2023.3267668 · Cited by 3
Feature Alignment and Reconstruction Constraints for Multimodal Sentiment Analysis
Zhu, Q., Lan, T., Liang, J. P., Xiang, D., & He, H. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650976 · Cited by 3
Spherical World-Locking for Audio-Visual Localization in Egocentric Videos
Yun, H., Gao, R., Ananthabhotla, I., Kumar, A. K., Donley, J., Li, C., Kim, G., Ithapu, V. K., & Murdock, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72691-0_15 · Cited by 3
Multi-modal, multi-step human–robot interaction method for natural interaction of service robots
Yura, J., Byambasuren, B.-E., & Kim, D. (2025) · Intelligent Service Robotics · doi:10.1007/s11370-025-00604-2 · Cited by 3
Fanaras, K., Tragoudaras, A., Antoniadis, C., & Massoud, Y. (2022) · IEEE International Midwest Symposium on Circuits and Systems (MWSCAS) · doi:10.1109/mwscas54063.2022.9859533 · Cited by 3
Tran, M., Kim, Y., Su, C.-C., Kuo, C.-H., Sun, M., & Soleymani, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72989-8_1 · Cited by 2
Zhu, C., Chen, M., Li, H., Zhang, S., Liang, H., Sun, C., Liu, Y., & Chen, J. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10678-1 · Cited by 2
Self-citationASDnB: Merging Face with Body Cues For Robust Active Speaker Detection
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2025) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb65343.2025.11410746 · Cited by 2
Geeroms, W., Allebosch, G., Kindt, S., Kadri, L., Veelaert, P., & Madhu, N. (2022) · Sensor Data Fusion: Trends, Solutions, Applications (SDF) · doi:10.1109/sdf55338.2022.9931697 · Cited by 2
E-Talk: Accelerating Active Speaker Detection with Audio-Visual Fusion and Edge-Cloud Computing
Yu, X., Zhang, L., & Li, X. (2023) · IEEE International Conference on Sensing, Communication, and Networking (SECON) · doi:10.1109/secon58729.2023.10287518 · Cited by 2
AS-Net: active speaker detection using deep audio-visual attention
Radman, A., & Laaksonen, J. T. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18457-9 · Cited by 2
Long-Term Social Interaction Context: The Key to Egocentric Addressee Detection
Kong, D., Khan, F. M., Zhang, X., Singhal, P., & Wu, Y. N. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447323 · Cited by 2
Evaluation of end-to-end continuous spanish lipreading in different data conditions
Gimeno-Gómez, D., & Martínez-Hinarejos, C.-D. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09809-4 · Cited by 2
Referring Expression Comprehension in semi-structured human–robot interaction
Jin, T., Meng, Q., Zhang, G., Huang, Q., Guo, F., Kong, S., Song, W., Zhu, J., & Gu, J. J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126965 · Cited by 2
Recognizing Indonesian words based on visual cues of lip movement using deep learning
Rahmatullah, G. M., Ruan, S.-J., & Li, L. P. (2025) · Measurement · doi:10.1016/j.measurement.2025.116968 · Cited by 2
Advancing Active Speaker Detection for Egocentric Videos
Huh, J., Ortiz, J. A., Kumar, A., Pandey, A., Aroudi, A., Wong, D. D. E., Nesta, F., Xu, B., & Donley, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888166 · Cited by 2
SLoClas: A Database for Joint Sound Localization and Classification
Qian, X., Sharma, B., Abridi, A. E., & Li, H. (2021) · International Conference on Speech Database and Assessments · doi:10.1109/o-cocosda202152914.2021.9660517 · Cited by 2
Self-citationDoes Audio help in deep Audio-Visual Saliency prediction models?
Agrawal, R., Jyoti, S., Girmaji, R., Sivaprasad, S., & Gandhi, V. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536221.3556625 · Cited by 2
Multi-modality Multi-attention Network for Ventricular Arrhythmia Classification
Xu, Z., Zang, M., Liu, T., Zhou, S., Liu, C., & Wang, Q. (2023) · International Conference on Bioinformatics and Intelligent Computing (BIC) · doi:10.1145/3592686.3592746 · Cited by 2
Audio Inputs for Active Speaker Detection and Localization Via Microphone Array
Berghi, D., & Jackson, P. J. B. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248185 · Cited by 2
Uncertainty-Guided End-to-End Audio-Visual Speaker Diarization for Far-Field Recordings
Yang, C., Chen, M., Wang, Y., & Wang, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612424 · Cited by 2
Predicting Individual Well-Being in Teamwork Contexts Based on Multi-Modal Speech FeaturesPreprint
Zeulner, T., Hagerer, G., Mueller, M., Vazquez, I., & Gloor, P. A. (2024) · Preprints.org · doi:10.20944/preprints202401.2030.v1 · Cited by 2
CASA-Net: Cross-attention and Self-attention for End-to-End Audio-visual Speaker Diarization
Zhou, H., Li, T., Wang, J., Li, L., & Hong, Q. (2023) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc58517.2023.10317320 · Cited by 1
A Two-Stage Audio-Visual Fusion Piano Transcription Model Based on the Attention Mechanism
Li, Y., Wang, X., Wu, R., Xu, W., & Cheng, W. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3426303 · Cited by 2
Jawade, B., Gadde, R. T., Bejjani, C., & Lan, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888378 · Cited by 1
Li, Y., Zhou, X., He, Z., Yu, W., & Luo, Y. H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888235 · Cited by 1
Audio-Faces Intra-Frame Alignment with Graph Attention Networks for Active Speaker Detection
Yin, Y., Yang, X., Liang, L., Xu, L., & Zou, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889368 · Cited by 1
Multimodal Audio-Visual Emotion Recognition for Social Robotics
Simone, G. D., Greco, L., Saggese, A., & Vento, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05060-1_21 · Cited by 1
Face-Voice Association for Audiovisual Active Speaker Detection in Egocentric Recordings
Clarke, J., Gotoh, Y., & Goetze, S. (2025) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco63237.2025.11226795 · Cited by 1
Metadata
- Authors
- Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H.
- Affiliations
- National University of Singapore (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 3927-3935
- arXiv
- 2107.06592
- OpenAlex
- W3206008172
- Cited by
- 156
- References
- 54
- Citations by year
- 2026: 16 · 2025: 35 · 2024: 34 · 2023: 39 · 2022: 30 · 2021: 2
- FWCI
- 15.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- active speaker detection, auditory-visual interaction, self-attention, temporal encoder, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
