Audio-Visual Deep Neural Network for Robust Person Verification
Abstract
Voice and face are two most popular biometrics for person verification, usually used in speaker verification and face verification tasks. It has already been observed that simply combining the information from these two modalities can lead to a more powerful and robust person verification system. In this article, to fully explore the multi-modal learning strategies for person verification, we proposed three types of audio-visual deep neural network (AVN), including feature level AVN (AVN-F), embedding level AVN (AVN-E), and embedding level combination with joint learning AVN (AVN-J). To further enhance the system robustness in real noisy conditions where not both modalities can be accessed with high-quality, we proposed several data augmentation strategies for each proposed AVN: A feature-level multi-modal data augmentation is proposed for AVN-F and an embedding-level data augmentation with novel noise distribution matching is designed for AVN-E. For AVN-J, both the feature and embedding level multi-modal data augmentation methods can be applied. All the proposed models are trained on the VoxCeleb2 dev dataset and evaluated on the standard VoxCeleb1 dataset, and the best system achieves 0.558, 0.441% and 0.793% EER on the three official trial lists of VoxCeleb1, which is to our knowledge the best published single system results on this corpus for person verification. To validate the robustness of the proposed approaches, a noisy evaluation set based on the VoxCeleb1 is constr…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing unimodal deepfake detectors are primarily designed to detect a single type of manipulation, such as visual, acoustic, and text.
“Existing unimodal deepfake detectors are primarily designed to detect a single type of manipulation, such as visual, acoustic, and text.”
From Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization (2024) · cites this paper as [46]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267 · see citing works
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203 · see citing works
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157 · see citing works
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781 · see citing works
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · see citing works
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214 · see citing works
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479 · see citing works
Dimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · see citing works
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,837 · see citing works
Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments
Huang, G. B., Mattar, M., Berg, T. L., & Miller, E. G. (2008) · HAL · Cited by 4,445 · see citing works
Speaker Verification Using Adapted Gaussian Mixture Models
Reynolds, D. A., Quatieri, T. F., & Dunn, R. B. (2000) · Digital Signal Processing · doi:10.1006/dspr.1999.0361 · Cited by 4,282 · see citing works
Front-End Factor Analysis for Speaker Verification
Dehak, N., Kenny, P. J., Dehak, R., Dumouchel, P., & Ouellet, P. (2011) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2010.2064307 · Cited by 3,549 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461375 · Cited by 2,698 · see citing works
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256 · see citing works
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · see citing works
Deep Learning Face Representation from Predicting 10,000 Classes
Sun, Y. J., Wang, X., & Tang, X. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.244 · Cited by 1,995 · see citing works
Deep Learning Face Representation by Joint Identification-Verification
Sun, Y., Wang, X., & Tang, X. (2014) · arXiv · doi:10.48550/arxiv.1406.4773 · Cited by 1,906 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Deep neural networks for small footprint text-dependent speaker verification
Variani, E., Lei, X., McDermott, E., Moreno, I. L., & Gónzalez-Domínguez, J. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854363 · Cited by 1,073 · see citing works
Deeply learned face representations are sparse, selective, and robust
Sun, Y. J., Wang, X., & Tang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298907 · Cited by 952 · see citing works
Perceptual Signal Analysis and Eigenvalue-Weighted Metrics for Musical Audio Quality Assessment
Majumder, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-27827-2_35 · Cited by 904 · see citing works
DeepID3: Face Recognition with Very Deep Neural NetworksPreprint
Sun, Y. J., Liang, D., Wang, X., & Tang, X. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.00873 · Cited by 842 · see citing works
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · see citing works
Deep Neural Network Embeddings for Text-Independent Speaker Verification
Snyder, D., Garcia-Romero, D., Povey, D., & Khudanpur, S. P. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-620 · Cited by 759 · see citing works
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · see citing works
Face recognition with learning-based descriptor
Cao, Z., Yin, Q., Tang, X., & Sun, J. Z. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539992 · Cited by 438 · see citing works
Combining Residual Networks with LSTMs for Lipreading
Stafylakis, T., & Tzimiropoulos, G. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-85 · Cited by 348 · see citing works
Learning Discriminant Face Descriptor
Lei, Z., Pietikäinen, M., & Li, S. Z. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.112 · Cited by 347 · see citing works
Gated Multimodal Units for Information FusionPreprint
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 302 · see citing works
End-to-End Audiovisual Speech Recognition
Petridis, S., Stafylakis, T., Ma, P., Cai, F., Tzimiropoulos, G., & Pantić, M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461326 · Cited by 301 · see citing works
Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching
Nagrani, A., Albanie, S., & Zisserman, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00879 · Cited by 224 · see citing works
End-to-End Text-Independent Speaker Verification with Triplet Loss on Short Utterances
Zhang, C., & Koishida, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1608 · Cited by 216 · see citing works
BUT System Description to VoxCeleb Speaker Recognition Challenge 2019Preprint
Zeinali, H., Wang, S., Silnova, A., Matějka, P., & Plchot, O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12592 · Cited by 193 · see citing works
Self-citationDeep feature for text-dependent speaker verification
Liu, Y., Qian, Y., Chen, N., Fu, T., Zhang, Y., & Yu, K. (2015) · Speech Communication · doi:10.1016/j.specom.2015.07.003 · Cited by 154 · see citing works
Self-citationMargin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition
Xiang, X., Wang, S., Huang, H., Qian, Y., & Yu, K. (2019) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc47483.2019.9023039 · Cited by 146 · see citing works
Self-citationOn Learning Associations of Faces and Voices
Kim, C., Shin, H. V., Oh, T.-H., Kaspar, A., Elgharib, M. A., & Matusik, W. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20873-8_18 · Cited by 88 · see citing works
Face-Voice Matching using Cross-modal Embeddings
Horiguchi, S., Kanda, N., & Nagamatsu, K. (2018) · ACM Multimedia · doi:10.1145/3240508.3240601 · Cited by 59 · see citing works
Data Augmentation Using Deep Generative Models for Embedding Based Speaker Recognition
Wang, S., Yang, Y., Wu, Z., Qian, Y., & Yu, K. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3016498 · Cited by 54 · see citing works
Self-citationShon, S., Oh, T.-H., & Glass, J. R. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683477 · Cited by 46 · see citing works
The 2019 NIST Audio-Visual Speaker Recognition Evaluation
Sadjadi, O., Greenberg, C. S., Singer, E., Reynolds, D. A., Mason, L. P., & Hernández-Cordero, J. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-37 · Cited by 42 · see citing works
Discriminative Neural Embedding Learning for Short-Duration Text-Independent Speaker Verification
Wang, S., Huang, Z., Qian, Y., & Yu, K. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2928128 · Cited by 35 · see citing works
Self-citationAudio-Visual Person Recognition in Multimedia Data From the Iarpa Janus Program
Sell, G., Duh, K., Snyder, D., Etter, D., & Garcia-Romero, D. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462122 · Cited by 34 · see citing works
Cross-modal Speaker Verification and Recognition: A Multilingual Perspective
Nawaz, S., Saeed, M. S., Morerio, P., Mahmood, A., Gallo, I., Yousaf, M. H., & Bue, A. D. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00184 · Cited by 34 · see citing works
Deep Audio-Visual Speech Separation with Attention Mechanism
Li, C., & Qian, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054180 · Cited by 33 · see citing works
Self-citationListen, Watch and Understand at the Cocktail Party: Audio-Visual-Contextual Speech Separation
Li, C., & Qian, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2028 · Cited by 30 · see citing works
Self-citationMulti-Modality Matters: A Performance Leap on VoxCeleb
Chen, Z., Wang, S., & Qian, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2229 · Cited by 23 · see citing works
Self-citationYang, Y., Wang, S., Sun, M., Qian, Y., & Yu, K. (2018) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2018.8706575 · Cited by 15 · see citing works
Self-citationAnalysis of ABC Submission to NIST SRE 2019 CMN and VAST Challenge
Alam, J., Boulianne, G., Burget, L., Dahmane, M., Sánchez, M. D., Lozano-Díez, A., Glembek, O., St-Charles, P.-L., Lalonde, M., Matějka, P., et al. (2020) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2020-41 · Cited by 14 · see citing works
Self-citationUsing Lip Features for Multimodal Speaker Verification
Zhang, X., & Broun, C. C. (2001) · DigitalCommons - CalPoly (California State Polytechnic University) · Cited by 2 · see citing works
Cross-modal Speaker Verification and Recognition: A Multilingual PerspectivePreprint
Saeed, M. S., Nawaz, S., Morerio, P., Mahmood, A., Gallo, I., Yousaf, M. H., & Bue, A. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13780 · Cited by 1 · see citing works
Flagged first, then by citations
7 of the 62 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake
Yang, W., Zhou, X., Chen, Z., Guo, B., Ba, Z., Xia, Z., Cao, X., & Ren, K. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3262148 · Cited by 198 · see citing works
Self-citationMultimodal emotion recognition based on audio and text by using hybrid attention networks
Zhang, S., Yang, Y., Chen, C., Liu, R., Tao, X., Guo, W., Xu, Y., & Zhao, X. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.105052 · Cited by 76 · see citing works
Multimodal Emotion Recognition With Temporal and Semantic Consistency
Chen, B., Cao, Q., Hou, M., Zhang, Z., Lu, G., & Zhang, D. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3129331 · Cited by 71 · see citing works
Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Ma, F., Li, Y., Ni, S., Huang, S., & Zhang, L. (2022) · Applied Sciences · doi:10.3390/app12010527 · Cited by 69 · see citing works
AVA-AVD: Audio-visual Speaker Diarization in the Wild
Xu, Z., Song, Z., Tsutsui, S., Feng, C., Ye, M., & Shou, M. Z. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548027 · Cited by 35 · see citing works
Wang, S., Chen, Z., Lee, K. A., Qian, Y., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3492793 · Cited by 32 · see citing works
Self-citationLi, W., Wang, Z., Mai, R., Ren, P., Zhang, Q., Zhou, Y., Xu, N., Zhuang, J., Xin, B., Gao, L., et al. (2023) · Visual Intelligence · doi:10.1007/s44267-023-00006-x · Cited by 25 · see citing works
MIS-AVoiDD: Modality Invariant and Specific Representation for Audio-Visual Deepfake Detection
Katamneni, V. S., & Rattani, A. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00207 · Cited by 28 · see citing works
Deep Learning-Based Holistic Speaker Independent Visual Speech Recognition
Nemani, P., Krishna, G. S., Nikhil, R., Sai, B. D. S., & Kumar, S. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3220190 · Cited by 22 · see citing works
Cross-Modal Knowledge Distillation With Multi-Stage Adaptive Feature Fusion for Speech Separation
Fan, C., Xiang, W., Tao, J., Yi, J., & Lv, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3533359 · Cited by 21 · see citing works
Few-shot class-incremental audio classification via discriminative prototype learning
Xie, W. C., Li, Y., He, Q., & Cao, W. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120044 · Cited by 21 · see citing works
Cross-Modal Distillation for Speaker Recognition
Jin, Y., Hu, G., Chen, H., Miao, D., Hu, L., & Zhao, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26525 · Cited by 21 · see citing works
Incorporating Visual Information in Audio Based Self-Supervised Speaker Recognition
Cai, D., Wang, W., & Li, M. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3162078 · Cited by 19 · see citing works
Lin, J., Cui, Y., Fang, C., Pan, B., Pan, C., Jiang, G., Zhang, S., Ma, S., & Tian, Q. C. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3652641 · Cited by 18 · see citing works
Cross-Modal Learning Based Flexible Bimodal Biometric Authentication With Template Protection
Jiang, Q. G., Zhao, G., Ma, X., Li, M., Tian, Y., & Li, X. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3364092 · Cited by 19 · see citing works
Maskeliūnas, R., Kulikajevas, A., Damaševičius, R., Pribuišis, K., Stanienė, N. U. , & Uloza, V. (2022) · Cancers · doi:10.3390/cancers14102366 · Cited by 19 · see citing works
Cross-Modal Audio-Visual Co-Learning for Text-Independent Speaker Verification
Liu, M., Lee, K. A., Wang, L., Zhang, H., Zeng, C., & Dang, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095883 · Cited by 17 · see citing works
STNet: Deep Audio–Visual Fusion Network for Robust Speaker Tracking
Li, Y., Liu, H., & Yang, B. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521737 · Cited by 16 · see citing works
Liu, J., Wang, C., & Zha, L. (2021) · Electronics · doi:10.3390/electronics10182206 · Cited by 16 · see citing works
Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
Katamneni, V. S., & Rattani, A. (2024) · 2024 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744480 · Cited by 20 · see citing works
CAD - Contextual Multi-modal Alignment for Dynamic AVQA
Nadeem, A., Hilton, A., Dawes, R., Thomas, G., & Mustafa, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00709 · Cited by 15 · see citing works
Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs
Tao, R., Lee, K. A., Das, R. K., Hautamäki, V., & Li, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3268568 · Cited by 13 · see citing works
Cuzzocrea, A., & Mumolo, E. (2021) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms53169.2021.9582651 · Cited by 9 · see citing works
DyViSE: Dynamic Vision-Guided Speaker Embedding for Audio-Visual Speaker Diarization
Wuerkaixi, A., Yan, K., Zhang, Y., Duan, Z., & Zhang, C. (2022) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp55362.2022.9948860 · Cited by 9 · see citing works
One Model to Rule Them all: A Universal Transformer for Biometric Matching
Abdrakhmanova, M., Yermekova, A., Barko, Y., Ryspayev, V., Jumadildayev, M., & Varol, H. A. (2024) · IEEE Access · doi:10.1109/access.2024.3426602 · Cited by 8 · see citing works
Audio-Visual Person Verification Based on Recursive Fusion of Joint Cross-Attention
Praveen, R. G., & Alam, J. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10582018 · Cited by 8 · see citing works
Lu, H., Gao, Y., & Xu, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3483265 · Cited by 7 · see citing works
A Study of Multimodal Person Verification Using Audio-Visual-Thermal Data
Abdrakhmanova, M., Abushakimova, S., Khassanov, Y., & Varol, H. A. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-33 · Cited by 6 · see citing works
Verifiable speech retrieval algorithm based on KNN secure hashing
Li, A., Huang, Y., & Zhang, Q. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13387-w · Cited by 6 · see citing works
Kashani, H. B., & Jazmi, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119833 · Cited by 6 · see citing works
TMS: Temporal multi-scale in time-delay neural network for speaker verification
Zhang, R., Wei, J., Lu, X., Lu, W., Jin, D., Zhang, L., Xu, J., & Dang, J. (2023) · Applied Intelligence · doi:10.1007/s10489-023-04953-2 · Cited by 5 · see citing works
Multimodal Speaker Recognition using voice and lip movement with decision and feature level fusionPreprint
Nainan, S., & Kulkarni, V. H. (2024) · Research Square · doi:10.21203/rs.3.rs-2960709/v1 · Cited by 5 · see citing works
Dynamic Cross Attention for Audio-Visual Person Verification
Praveen, R. G., & Alam, J. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10581998 · Cited by 4 · see citing works
DeepLip: A Benchmark for Deep Learning-Based Audio-Visual Lip Biometrics
Liu, M., Wang, L., Lee, K. A., Zhang, H., Zeng, C., & Dang, J. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688240 · Cited by 4 · see citing works
Multimodal Person Verification With Generative Thermal Data Augmentation
Abdrakhmanova, M., Unaspekov, T., & Varol, H. A. (2023) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2023.3346938 · Cited by 4 · see citing works
Exploring Deep Learning for Joint Audio-Visual Lip BiometricsPreprint
Liu, M., Wang, L., Lee, K. A., Zhang, H., Zeng, C., & Dang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08510 · Cited by 4 · see citing works
AVCLNet: Multimodal Multispeaker Tracking Network Using Audio‐Visual Contrastive Learning
Li, Y., Li, Y., Xu, Z., Guo, H., Liu, M., & Wan, W. (2025) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.70092 · Cited by 3 · see citing works
Audio–Visual Fusion Based on Interactive Attention for Person Verification
Jing, X., He, L. J., Song, Z., & Wang, S. (2023) · Sensors · doi:10.3390/s23249845 · Cited by 3 · see citing works
Self-citationAn AI based Solution for Predicting the Text Pattern from Sign Language
Bhargav, B., Abhishek, D. N., Deekshitha, D., Talanki, S., Aradhya, S., & Thejaswini (2021) · International Conference on Contemporary Computing · doi:10.1145/3474124.3474210 · Cited by 3 · see citing works
Liu, L. (2022) · Journal of Mathematics · doi:10.1155/2022/7987592 · Cited by 3 · see citing works
RetractedERDBF: Embedding-Regularized Double Branches Fusion for Multi-Modal Age Estimation
Wu, B., Lu, H., Chen, Z., Zhu, C., & Xu, S. (2023) · IEEE Access · doi:10.1109/access.2023.3275765 · Cited by 2 · see citing works
Self-citationAudio-Visual Wake-up Word Spotting Under Noisy and Multi-person Scenarios
Li, C., Su, F., & Liu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-80136-5_12 · Cited by 2 · see citing works
LAVViT: Latent Audio-Visual Vision Transformers for Speaker Verification
Praveen, R. G., & Alam, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888977 · Cited by 2 · see citing works
2021 Index IEEE/ACM Transactions on Audio, Speech, and Language Processing Vol. 29
(2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3147096 · Cited by 2 · see citing works
Enhanced Speaker Verification Incorporated with Face Recognition
Ramya., R., Nivash, P. D., Khansa, I. A., & Iyappa, B. H. (2022) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec54921.2022.9951892 · Cited by 1 · see citing works
MHAN: Bottleneck Fusion Model Based on Hybrid Attention Network for Multimodal Emotion Recognition
Li, K., Huang, Y., Zhong, G., Yolwas, N., & Wushouer, S. (2025) · Journal of Shanghai Jiaotong University (Science) · doi:10.1007/s12204-025-2820-x · Cited by 1 · see citing works
Speech-Based Robust Speaker Authentication Against Voice Conversion-Based Spoofing Attacks
Revathi, A. A., Vallinayagam, R., Sivaranjani, S., & Deepthi, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-06700-5_23 · Cited by 1 · see citing works
Research on the Improvement of the Target Speaker Recognition System Based on Dual-Modal Fusion
Wang, C., Zhu, H., & Xu, L. (2024) · International Conference on Computer Vision, Image and Deep Learning (CVIDL) · doi:10.1109/cvidl62147.2024.10603613 · Cited by 1 · see citing works
Navigating Audio-Visual Event Detection Across Mismatched Modalities
Li, G., Xu, X., Wu, M., & Yu, K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747579
Medical Difficult Airway Detection using Speech Technology
Zhou, Z., Cao, S., Chen, Z., Liu, B., Xia, M., Jiang, H., & Qian, Y. (2022) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp57327.2022.10037911
Self-citationA Target Identification Method Based on Uncertainty Estimation and Evidence Theory
Lin, L., Chen, J., Cai, X., & Jin, L. (2022) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic57696.2022.10075117
Speaker Verification Uasing Spatial Attention Mechanism And Semantic Enhancement
Li, P., Liu, X., Suqin, W., & Xie, X. M. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10661780
Exploiting Visual Lip Movement Information for Automatic Speaker Recognition
Nainan, S., Parmar, S. N., & Bakade, K. (2025) · Circuits Systems and Signal Processing · doi:10.1007/s00034-025-03113-w
On the Use of Cross-Attentive Fusion Techniques for Audio-Visual Speaker Verification
Alam, J., & Praveen, R. G. (2025) · Machine translation · doi:10.1007/978-3-031-85747-8_9
Boosting Video Deepfake Detection via Quality-Aware and Multi-scale Audio-Visual Alignment
Chen, J., Che, X., Li, Q., & Zhao, Q. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4109-6_18
Constructing Audio-Visual Heterogeneous Graphs for Deepfake Detection
Xie, N., Li, J., Deng, Y., Guo, M., & Yang, L. (2025) · Signals and communication technology · doi:10.1007/978-3-031-94973-9_4
Martínez-Díaz, Y., Méndez-Vázquez, H., Hernández-Sierra, G., & Larcher, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-11358-0_34
Robust Speaker Authentication System Against Replay Attacks: Perceptual Features and Models
Revathi, A., Vallinayagam, R., Alapati, G., & Ramisetti, P. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-19690-3_18
Leveraging Large Multimodal Models for Audio-Video Deepfake Detection: A Pilot Study
Cao, S., Li, Y., Luo, Y., Yin, J., & Ma, L. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463461
Multimodal Recognition of Out-of-Distribution Individuals Using Contrastive Learning
García, S., Gomez‐Donoso, F., & Cazorla, M. (2026) · AI · doi:10.3390/ai7050162
Olarewaju, S., Sadiq, A. S., Kaiwartya, O., & Konios, A. (2026) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp6030103
Noise- and Occlusion-Robust Mask-Guided Adaptive Fusion for Audio-Visual Person Verification
Wang, J., Zhao, Q., Wu, X., 徐明星, Hamdulla, A., & Zheng, T. F. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728508
MDD-CR: End-to-end multimodal deepfake detection fusing spatio-temporal and spectrographic features
Das, A. K., & Naskar, R. (2026) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2026.105021
- Authors
- Qian, Y., Chen, Z., & Wang, S.
- Affiliations
- Shanghai Jiao Tong University (CN)
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2021
- Volume
- 29
- Pages
- 1079-1092
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3126757411
- Cited by
- 63
- References
- 62
- Citations by year
- 2026: 9 · 2025: 8 · 2024: 13 · 2023: 13 · 2022: 13 · 2021: 7
- FWCI
- 5.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- Funders
- National Natural Science Foundation of China (62071288, U1736202)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- human verification, multimodal fusion, speaker verification, face verification, noise robustness, data augmentation, VoxCeleb1, VoxCeleb2
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review