Multimodal Forgery Detection Using Ensemble Learning
Open accessAbstract
The recent rapid revolution in Artificial Intelligence (AI) technology has enabled the creation of hyper-realistic deepfakes, and detecting deepfake videos (also known as AI-synthesized videos) has become a critical task. The existing systems generally do not fully consider the unified processing of audio and video data, so there is still room for further improvement. In this paper, we focus on the multimodal forgery detection task and propose a deep forgery detection method based on audiovisual ensemble learning. The proposed method consists of four parts, namely a Video Network, an Audio Network, an Audiovisual Network, and a Voting Module. Given a video, the proposed multimodal and ensemble learning system can identify whether it is fake or real. Experimental results on a recently released multimodal FakeAVCeleb dataset show that the proposed method achieves 89% accuracy, significantly outperforming existing models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The multimodal approach [75] achieved the second-best results for both audio-visual deepfake detection task.
“In comparison, the multimodal approach [75] achieved the second-best results for both audio-visual deepfake detection task.”
From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [75]
The approach in [75] relies on a late fusion of audio-video features for prediction and fails to consider the intricacies of the inherent correlation pattern between modalities.
“However, this approach relies on a late fusion of audio-video features for prediction and fails to consider the intricacies of the inherent correlation pattern between modalities.”
From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [75]
Chung et al. [18] proposed an audio-visual dissonance-based model for deepfake video detection, and the approach showed an accuracy of 69% on the FakeAVCeleb dataset.
“Chung et al. [18] proposed an audio-visual dissonance-based model for deepfake video detection. However, the approach showed an accuracy of 69% on the FakeAVCeleb dataset [75], demonstrating that using only spatio-temporal audio and visual information is insufficient for the task of multimodal forgery detection.”
From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [75]
Using only spatio-temporal audio and visual information is insufficient for the task of multimodal forgery detection.
“Chung et al. [18] proposed an audio-visual dissonance-based model for deepfake video detection. However, the approach showed an accuracy of 69% on the FakeAVCeleb dataset [75], demonstrating that using only spatio-temporal audio and visual information is insufficient for the task of multimodal forgery detection.”
From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [75]
Existing audio-visual detection methods do not always surpass single-modality analysis.
“Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities.”
From Integrating Audio-Visual Features For Multimodal Deepfake Detection (2023) · cites this paper as [15]
Incorporating a single modality detection loss can alleviate the confusion.
“Incorporating a single modality detection loss [15] can alleviate the confusion.”
From Integrating Audio-Visual Features For Multimodal Deepfake Detection (2023) · cites this paper as [15]
Existing feature fusion methods treat different types of deepfakes as the same fake type.
“However, existing feature fusion methods treat different types of deepfakes as the same fake type.”
From Integrating Audio-Visual Features For Multimodal Deepfake Detection (2023) · cites this paper as [15]
Treating different types of deepfakes as the same fake type may confuse feature learning of a single modality, especially in ensemble-based fusion.
“This may confuse the feature learning of a single modality, especially in ensemble-based fusion [9, 11, 15].”
From Integrating Audio-Visual Features For Multimodal Deepfake Detection (2023) · cites this paper as [15]
Cites
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256 · see citing works
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · see citing works
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 1× · see citing works
“Some well-known examples include FFW [48], UADFV [49], DeepfakeTIMIT dataset [50], FaceForensics++ [51], Celeb-DF [52] , Google DFD [51], DeeperForensics [53], DFDC [54], KoDF [55] and FakeAVCeleb [17].”
MethodVaccari, C., & Chadwick, A. (2020) · Social Media + Society · doi:10.1177/2056305120903408 · Cited by 978 · see citing works
Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security
Chesney, R., & Citron, D. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3213954 · Cited by 880 · see citing works
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
Jiang, L., Li, R., Wu, W., Qian, C., & Loy, C. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00296 · Cited by 599 · see citing works
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Haliassos, A., Vougioukas, K., Petridis, S., & Pantic, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00500 · Cited by 537 · see citing works
DeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · see citing works
The deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427 · see citing works
ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K. A., Kinnunen, T., Evans, N., et al. (2021) · 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge · doi:10.21437/asvspoof.2021-8 · Cited by 380 · see citing works
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319 · see citing works
Deep fake image detection based on pairwise learning
Hsu, C.-C., Zhuang, Y.-X., & Lee, C.-Y. (2020) · Applied Sciences · doi:10.3390/app10010370 · Cited by 304 · see citing works
Deepfake Detection: Current Challenges and Next Steps
Lyu, S. (2020) · 2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) · doi:10.1109/icmew46912.2020.9105991 · Cited by 234 · see citing works
Joint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217 · see citing works
Fooled twice – people cannot detect deepfakes but think they can
Köbis, N., Doležalová, B., & Soraperra, I. (2021) · doi:10.2139/ssrn.3832978 · Cited by 194 · see citing works
Fooled twice: People cannot detect deepfakes but think they can
Köbis, N. C., Doležalová, B., & Soraperra, I. (2021) · iScience · doi:10.1016/j.isci.2021.103364 · Cited by 193 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodaltrace: Deepfake Detection using Audiovisual Representation Learning
Anas Raza, M., & Mahmood Malik, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00106 · Cited by 63 · see citing works
Integrating Audio-Visual Features For Multimodal Deepfake Detection
Muppalla, S., Jia, S., & Lyu, S. (2023) · 2023 IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc60662.2023.10534969 · Cited by 46 · see citing works
“The analysis of two modalities, such as by fusing audio-visual features [9, 11] or learning audio-visual inconsistencies [14, 12, 15], can provide rich information to expose the deepfakes.”
BackgroundMethodInfluential"Better Be Computer or I'm Dumb": A Large-Scale Evaluation of Humans as Audio Deepfake Detectors
Warren, K., Tucker, T., Crowder, A., Olszewski, D., Lu, A., Fedele, C., Pasternak, M., Layton, S., Butler, K., Gates, C., et al. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670325 · Cited by 15 · see citing works
Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection
Masood, M., Javed, A., & Irtaza, A. (2023) · Preprint (Research Square) · doi:10.21203/rs.3.rs-3415144/v1 · Cited by 6 · see citing works
Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges
Liu, P., Tao, Q., & Zhou, J. T. (2024) · arXiv · doi:10.48550/arxiv.2406.06965 · Cited by 6 · see citing works
- Authors
- Hashmi, A., Shahzad, S. A., Ahmad, W., Lin, C. W., Tsao, Y., & Wang, H.-M.
- Affiliations
- National Tsing Hua University (TW)
- Institute of Information Science, Academia Sinica (TW)
- Research Center for Information Technology Innovation, Academia Sinica (TW)
- Venue
- 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)
- Publisher
- IEEE
- Published
- 2022-11-07
- Pages
- 1524-1532
- OpenAlex
- W4312095999
- Cited by
- 32
- References
- 16
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 10 · 2023: 3
- FWCI
- 2.27
- Type
- conference paper
- Language
- en
- Open access
- green
- dx.doi.org
- Locations
- Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) (publishedVersion)
- Funders
- National Science and Technology Council
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- deepfake detection, ensemble learning, audiovisual fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
