The DOI matches its DataCite record; 52 of the 54 references it cites were found in a registry or an index.
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Open accessAbstract
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Squeeze-and-Excitation NetworksPreprint
Hu, J., Shen, L., & Sun, G. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1709.01507 · Cited by 37,845
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 5,270
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,794
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658
MUSAN: A Music, Speech, and Noise CorpusPreprint
Snyder, D., Chen, G., & Povey, D. (2015) · arXiv (Cornell University) · doi:10.1007/978-3-032-27827-2_35 · Cited by 1,454
A study on data augmentation of reverberant speech for robust speech recognition
Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., & Khudanpur, S. P. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953152 · Cited by 965
Flagged first, then by citations
2 of the 54 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. The Conversation: deep audio-visual speech enhancement. Proc. Interspeech (2018), 3244-3248.
- Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. 2017. S3FD: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference on computer vision (ICCV). 192-201.
Cited by
Sorted by citations
Items 101–156 of 156
Watanabe, K., Großmann, N., Maerz, C., Ishimaru, S., & Dengel, A. R. (2026) · Springer eBooks · doi:10.1007/978-981-95-1622-3_4 · Cited by 1
Beyond Words and Pictures: A Survey of Multimodal Fake News Detection
Hu, T., Lei, Y., Wang, Y., Ma, R., Yin, X., Liu, M., & Zhao, R. (2026) · Expert Systems · doi:10.1111/exsy.70260 · Cited by 4
GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection
Wang, Y., Ha, J., Ramirez, F. M., Wang, Y., & Crandall, D. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00111 · Cited by 1
A First Exploration of Neuromorphic OT-CFM for Multi-speaker VSR
Chen, L., Fang, J., Liu, H., Xu, C., Chen, J., Li, X., Cai, W., & Chen, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37281-9_8 · Cited by 3
Visual-Enhanced End-to-End Neural Diarization
Qiu, Q., Xu, T., & Chen, E. (2022) · International Conference on Image, Video and Signal Processing (IVSP) · doi:10.1145/3531232.3531250 · Cited by 1
Salient Face Prediction without Bells and Whistles
Jyoti, S., Girmaji, R., Agrawal, R., Sivaprasad, S., & Gandhi, V. (2022) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta56598.2022.10034571 · Cited by 1
Kyoung, M., & Song, H. J. (2023) · IEEE Access · doi:10.1109/access.2023.3325474 · Cited by 1
Predicting Individual Well-Being in Teamwork Contexts Based on Speech Features
Zeulner, T., Hagerer, G., Müller, M., Vazquez, I., & Gloor, P. A. (2024) · Information · doi:10.3390/info15040217 · Cited by 1
Perceptual Synchronization Scoring of Dubbed Content using Phoneme-Viseme Agreement
Gupta, H. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00048 · Cited by 1
Zhao, G., Liu, Y., Wang, X., & Yan, F. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658055 · Cited by 1
FabuLight-ASD: unveiling speech activity via body language
Carneiro, H., & Wermter, S. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10792-0 · Cited by 1
VAD-CLVA: Integrating CLIP with LLaVA for Voice Activity Detection
Appiani, A., & Beyan, C. (2025) · Information · doi:10.3390/info16030233 · Cited by 1
Target Speech Detection With Multimodal Prompts
Jiang, Y., Tao, R., Chen, Z., Qian, Y., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3579304 · Cited by 1
Self-citationA Multi-Purpose Audio-Visual Corpus for Multi-ModalPreprint
Peymanfard, J., Heydarian, S., Lashinila, A., Mozayani, N., Zeinali, H., & Mohammadi, M. R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4494603 · Cited by 1
SLoClas: A Database for Joint Sound Localization and ClassificationPreprint
Qian, X., Sharma, B., Abridi, A. E., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02539
Self-citationReal-time Architecture for Audio-Visual Active Speaker Detection
Huang, M., Wang, W., Lin, Z., Tesema, F. B., Ji, S., Gu, J. J., Wan, M., Song, W., Li, T., & Zhu, S. (2022) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio55434.2022.10011692
Assessing active speaker detection algorithms through the lens of automated editing
Girmaji, R., Achary, S., Deshmukh, A. A., & Gandhi, V. (2023) · ACM International Conference on Interactive Media Experiences · doi:10.1145/3604321.3604373
Cross-Modal Sentiment Analysis
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_5
Multimodal Sentiment Analysis Data Sets and Preprocessing
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_2
Journal of Engineering and Applied Sciences Technology
Kadam, A. V. (2022) · Journal of Engineering and Applied Sciences Technology · doi:10.47363/jeast/2023(4)184
Hai, J., Liu, Y.-J., & Elhilali, M. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389694
Zhao, H., Zhang, L., Li, Y., Wang, Y., Wang, H., Rao, W., Wang, Q., & Xie, L. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0601-3_23
Multimodal active speaker detection using cross-attention and contextual information
Mocanu, B., & Țapu, R. (2024) · International Conference on Consumer Electronics · doi:10.1109/icce59016.2024.10444380
Zhang, L., Zhou, Y., Zhao, Y., & Liu, H. (2023) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc59590.2023.10507617
Cross-Modal Active Speaker Detection Algorithm in Video and End-To-End Landing Solution
Yang, Y., & Yan, S. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-2200-6_29
DiVAS: Video and Audio Synchronization with Dynamic Frame Rates
Fernandez-Labrador, C., Akçay, M., Abecassis, E., Massich, J., & Schroers, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02535
AI-Powered Video and Image Processing for Video Meetings
Mohite, A., Arora, G., Khanderao, N., Rathod, A., & Kulkarni, P. D. (2024) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-22355
Li, Y., Luo, Y. A., & Zhou, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2064-7_24
Egocentric Speaker Diarization with Vision-Guided Clustering and Adaptive Speech Re-detection
Huang, H., Yu, H., Liu, D., Chen, H., & Cai, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889699
An Efficient and Streaming Audio Visual Active Speaker Detection System
Kundu, A., Jin, Y., Sekhavat, M. H., Horton, M., Tormoen, D., & Naik, D. K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887588
Jung, S., Kum, J., & Lee, M. (2025) · Electronics · doi:10.3390/electronics14081592
Graph-Based Active Speaker Detection in Fisheye Meeting Scenes with Spatial SpectraPreprint
Jiang, W., Rao, L., Dai, G., Huang, X., & Xu, W. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5260082
How to Squeeze An Explanation Out of Your Model
Roxo, T., Costa, J. C., Inácio, P. R. M., & Proença, H. P. M. C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92089-9_16
Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement
Li, X., Chen, Z., Yu, X., Chu, I.-H., Chang, P., & Xiao, J. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01063
Vietnamese Automatic Speech Recognition Utilizing Audio and Visual Data
Duong, T.-T., Nguyen, V.-M., Phạm, T. T. H., & Le, H.-M. (2025) · International Conference on Multimedia Analysis and Pattern Recognition (MAPR) · doi:10.1109/mapr67746.2025.11133884
MSSG: Multi-Scale Speaker Graph Network for Active Speaker Detection
Li, G., Yi, J., Wen, Z., Fu, R., Wang, Y., & Tao, J. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3608949 · Cited by 1
MC-Mamba: Cross-modal target speaker extraction model based on multiple consistency
Lv, K., Deng, Y., & Wei, Y. (2025) · Speech Communication · doi:10.1016/j.specom.2025.103304
Clarke, J., Gotoh, Y., & Goetze, S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07959-6_21
Sarcasm Subtype-Specific Reasoning in Dialogue with Multimodal Cues Using Large Language Models
Kang, C. G., Lee, W., Hwang, S., Tae, S., Sun, S., & Kim, J. H. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760953
Xu, L., Luo, Z., & Kang, C. (2025) · Physical Communication · doi:10.1016/j.phycom.2025.102897
Gaze-Driven Active Speaker Detection in Meetings
Jiang, W., Rao, L., Dai, G., Wu, Y., & Xu, W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-9849-3_20
Audio-Visual Deepfake Detection via Multi-Teacher Distillation of Content and Semantic Consistency
Sun, B., Du, H., Fu, Y., Dong, T., & Lu, W. (2025) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip67698.2025.11396833
Robust Training of Low-Parameter Models for Active Speaker Detection from Digital Videos
Janampa, S., Ugarte, B. M., Pattichis, C. S., & Pattichis, M. S. (2025) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf67917.2025.11443698
MSCT: Differential Cross-Modal Attention for Deepfake Detection
Wei, F., Liu, M., Wang, Y., Wang, J., Zhao, S., & Li, N. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460824
β-AVSDNET: A Novel End-To-End Neural Network Architecture For Audio-Visual Speaker Diarization
You, C. H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460727
Yin, Y., Zhuang, Y., Xie, Z., Li, C., Xu, L., & Zou, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464605
Vision language model-based multimodal voice activity detection
Li, J., Li, X., & Wei, H. (2026) · Real-Time Image Processing and Deep Learning · doi:10.1117/12.3091979
A Gated Multi-Modal Fusion Framework Via Mixture of Experts for Environment-Aware UAV Detection
Wang, Y., Zou, J., An, R., & Jia, L. (2026) · International Symposium on Intelligent Robotics and Systems (ISoIRS) · doi:10.1109/isoirs70157.2026.11545292
CHM: Context Hiding and Misguidance for Robust Adversarial Attacks on Active Speaker Detection
Ye, X. H., Xiao, Y., Guan, Q., & Liu, Z. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810812
Clustering-Based Learning for UAV Tracking and Position Estimation
Xiao, J., Pisutsin, P., Tsao, C. W., & Feroskhan, M. (2026) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms69701.2026.11602209
Learning Adaptive Cross-Modal Interactions for Multimodal Sentiment Analysis
Cheng, C., Wu, H., Wang, J., & Ouyang, Y. (2026) · Data · doi:10.3390/data11080209
Ou, S., & Chen, X. (2026) · IEEE Conference on Cybernetics and Intelligent Systems · doi:10.1109/cis-ram71614.2026.11668514
RealDyadic: Synthesizing Realistic Dyadic 3D Dialogue with Neural Appearance Priors
Zhu, L., Lin, L., Zhu, Y. P., Wu, J., Hou, X., Li, Y., Liu, Y., & Chen, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37602-2_30
Hong, J., Park, J., & Kwon, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37324-3_10
Speaker Diarization in Static and Egocentric Videos via Speech-Face Alignment
Zhang, Z., Yin, X. O., Zheng, Z., Akash, K., Misu, T., & Busso, C. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831222
Metadata
- Authors
- Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H.
- Affiliations
- National University of Singapore (SG)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 3927-3935
- arXiv
- 2107.06592
- OpenAlex
- W3206008172
- Cited by
- 156
- References
- 54
- Citations by year
- 2026: 16 · 2025: 35 · 2024: 34 · 2023: 39 · 2022: 30 · 2021: 2
- FWCI
- 15.98
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- active speaker detection, auditory-visual interaction, self-attention, temporal encoder, multimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
