The DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index; 9 of the 11 sources its claims cite were checked for support.
Integrating Audio-Visual Features For Multimodal Deepfake Detection
Open accessAbstract
Deepfakes are AI-generated media in which an image or video has been digitally modified. The advancements made in deepfake technology have led to privacy and security issues. Most deepfake detection techniques rely on the detection of a single modality. Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities. Therefore, this paper proposes an audio visual based method for deepfake detection, which integrates fine-grained deepfake identification with binary classification. We categorize the samples into four types by combining labels specific to each single modality. This method enhances the detection under intra-domain and cross-domain testing.
Overview
CoChat Summary
TL;DR
This paper proposes an audio-visual deepfake detector that combines binary classification for each modality with a fine-grained four-class identification module for real/fake audio-video combinations. It reports improved intra-domain and cross-domain detection on FakeAVCeleb and TMC over single-modality and traditional fusion baselines.
Key findings
- On FakeAVCeleb intra-domain testing, the proposed score-fusion method with Capsule Forensics achieved overall AUC 99.30 and ACC 99.20, outperforming reported baselines such as AVoiD-DF (89.20 AUC, 83.70 ACC) and AV-Lip-Sync (94.00 ACC).
- On TMC intra-domain testing, the proposed score-fusion method with Capsule Forensics achieved overall AUC 93.67 and ACC 93.15; with Swin Transformer it achieved 80.11 AUC and 85.57 ACC.
- Under cross-domain testing, the proposed method achieved the best performance in both scenarios; for example, TMC→FakeAVCeleb Ours-S reached 82.17 AUC and 89.73 ACC, while FakeAVCeleb→TMC Ours-S reached 70.41 AUC and 79.68 ACC in one reported setting.
- In cross-type testing on TMC mismatched real audio/video, the score-fusion Swin Transformer model achieved 63.83 AUC and 69.63 ACC, while the naive fusion baseline achieved 44.00 ACC; most mismatched samples were classified as “Real Video Fake Audio.”
Methods
- Video preprocessing extracts frames (1 frame per second for videos longer than a minute, all frames for shorter videos), detects and crops faces with MTCNN; audio is converted to WAV and then to mel-spectrogram images with frequency scale 0–8000 Hz and length 4 seconds.
- Features are extracted using Capsule network and Swin Transformer backbones; fusion is either feature fusion (concatenating averaged visual features with audio features) or score fusion (averaging video and audio network scores), followed by a four-class classification head.
- The multi-task loss combines binary cross-entropy losses for audio and video with a four-class cross-entropy loss for the four audio-video types; models are trained for 50 epochs with batch size 10, Adam optimizer, and learning rate 5×10⁻⁴, evaluated on FakeAVCeleb and TMC with an 80-20 split using Accuracy and AUC.
Limitations
- The paper does not include a dedicated limitations section; it states future work will focus on more robust multimodal networks and improving generalizability to unseen deepfakes.
- Evident from the reported results: cross-domain performance drops substantially compared with intra-domain performance (e.g., FakeAVCeleb→TMC Ours-S around 65–70 AUC versus 99.30 AUC on FakeAVCeleb intra-domain), indicating a generalization gap.
- The authors attribute lower TMC results to the dataset’s high diversity and additional perturbations in both video and audio tracks.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 1 Not valid · 2 Not checked · 19 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This may confuse the feature learning of a single modality, especially in ensemble-based fusion [9, 11, 15].”
What the sources say
- Partly supportedcites [9] Khalid, 2021, Fakeavceleb: A novel audio-video multimodal deepfake dataset, arXiv preprint sourceExperimental — accuracy not yet measured
C.3 Ensemble Results (p. 20) reports that ensemble-based methods struggled to detect multimodal deepfakes, and C.1 Unimodal Results (p. 19) describes testing on multiple fake-video categories. The paper does not say that treating different deepfake types as one fake class confuses single-modality feature learning, or identify that as the reason for the ensemble results. (LLM, conf=0.82)
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
Section 4.3 (p. 6) reports that unimodal detectors struggled on a test set containing three types of fakes, and Section 4.4 (p. 7) reports that ensemble detectors also fell short of high detection accuracy. The paper does not say that treating those types as one fake class confused single-modality feature learning, or that this was especially a problem in ensemble-based fusion. (LLM, conf=0.78)
- Partly supportedcites [15] sourceExperimental — accuracy not yet measured
Our copy shows the phenomenon only at the level of class composition and test-set performance: Section IV-B ('Training Sets for Different Models', Table I) groups heterogeneous FakeAVCeleb categories into one 'fake' class, and Sections IV-C1/IV-C3 report degraded results when that class mixes clips with and without manipulation in the target modality (video-only network 0.50 on RTVC; Audiovisual Network 0.49 on faceswap/fsgan, blamed on 'the dominance of the audio modality in the final prediction'). The paper never states that this confuses the feature learning of a single modality. The 'especially in ensemble-based fusion' emphasis runs against our copy: the ensemble/voting model is reported as the best and 'more balanced' performer (0.89 on Testset-I/Testset-II, vs 0.78/0.70/0.69/0.515 for the prior ensemble and multimodal models of [16] and [45] in Table VII). (LLM, conf=0.65)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, existing feature fusion methods treat different types of deepfakes as the same fake type.”
What the sources say
- Partly supportedcites [9] Khalid, 2021, Fakeavceleb: A novel audio-video multimodal deepfake dataset, arXiv preprint sourceExperimental — accuracy not yet measured
The Background and Motivation section (p. 3) says the DFDC dataset labels an entire video as fake without specifying whether its audio or video is fake. That supports the concern that different kinds of manipulation can share a fake label, but the paper does not establish that existing feature-fusion methods generally treat all deepfake types as the same type. (LLM, conf=0.77)
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
The paper identifies three forgery types (Figure 1, p. 1) but evaluates its detectors using a test set grouped into real and fake videos (Section 4.1, p. 5). That supports the narrower observation that this evaluation groups different forgeries under a fake label; it does not establish that existing feature-fusion methods generally do so. (LLM, conf=0.78)
- Partly supportedcites [15] sourceExperimental — accuracy not yet measured
Our copy does discuss the limits of existing fusion/multimodal approaches — Related Work II-B (p. 2) describes [45], [46] and [47] as audiovisual feature-fusion systems whose output is only 'whether a given video is real or not', and Sec. IV/Table VII (pp. 6-7) reports that 'Despite exploiting both modalities, their multimodal and ensemble models yield poor results' across the per-technique test sets (faceswap, fsgan, RTVC, wav2lip). However, the specific criticism the claim attributes — that existing feature fusion methods treat different types of deepfakes as the same fake type — is nowhere stated; the gap this paper actually asserts is a modality gap: 'The existing systems generally do not fully consider the unified processing of audio and video data' (Abstract) and 'most of these existing methods are unimodal... existing techniques may not be perfect designs for detecting multimodal manipulations' (Introduction, p. 1). Collapsing manipulation categories into one Fake class is in fact this paper's own training design (Sec. IV-A/B, Tables I-III), not a stated fault of prior fusion work. (LLM, conf=0.62)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities.”
- Also stated in: § I. INTRODUCTION · p. 1 · plain
“However, existing studies reveal that employing multimodalities does not guarantee a higher accuracy compared to the application of a single modality [9, 11, 15].”
What the sources say
- Supportedcites [9] Khalid, 2021, Fakeavceleb: A novel audio-video multimodal deepfake dataset, arXiv preprint source
On FakeAVCeleb, the Unimodal Results (p. 19) report best AUC scores of approximately 97% for audio-only and 93% for video-only detection. The Multimodal Results (p. 21) report 53.3%, 68.8%, and 66.2% for three audio-video methods. These results support the qualified claim that audio-visual methods do not always outperform single-modality analysis. (LLM, conf=0.90)
- Supportedcites [11] source
The Conclusion (p. 7) reports that, among the methods evaluated on FakeAVCeleb, multimodal methods performed worst, while single-modality methods performed better than those multimodal methods. This supports the qualified claim that audio-visual methods do not always surpass single-modality analysis. (LLM, conf=0.92)
- Supportedcites [15] source
The comparison subsection (Sec. 3 'Audiovisual Detection Results' / Table VII discussion) reports that existing audiovisual methods underperform single-modality ones: unimodal VGG16 reached 0.81 accuracy while the multimodal and ensemble models of [16] yielded 0.78, 0.50, 0.674 and 0.515, and the paper states explicitly 'Despite exploiting both modalities, their multimodal and ensemble models yield poor results.' Their dissonance-based Multimodal-4 [45] reached only 0.69. This directly supports the hedged claim that existing audio-visual detection does not always surpass single-modality analysis (the related-work note that multimodal can beat unimodal [43][44] is consistent with 'not always'). (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § I. INTRODUCTION · p. 1 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This method enhances the detection under intra-domain and cross-domain testing.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the advantage of the finegrained video identification module, our method, especially with score fusion, clearly surpasses other approaches, achieving the highest AUC and ACC scores in the majority of cases.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
14 more claims
Create a free account to see all 25 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 3
- Probably not a substantial claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“RNNs excel in the realm of sequence modeling, making them particularly effective for tasks involving language and temporal data, as their architecture is specifically designed to process sequences of data, such as text, enabling them to capture the context and order of the input effectively [150], [151], [152], [153], [154].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [150]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Each ethnic group consists of 100 real videos of 100 celebrities, 50 for each gender. This outputs 600 different videos with an average duration of about 7 seconds.”
Reported 600 · recomputed 500
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ B.1. Video Branch · p. 2
“we extract one frame per second for videos that are longer than a minute, such as the videos in TMC dataset [10]”
§ B. Dataset · p. 3
“The duration of these videos varies from 10 seconds to a minute.”
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We employed the Adam optimizer [26] with β 1 = 0 .”
MethodSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 6×
“In our experiment, we will utilize the Capsule network [16], widely used in deepfake detection methods, and the Swin Transformer [17], which has recently demonstrated powerful feature learning capabilities in various image classification tasks, as two examples for the video and audio networks to show the flexibility o…”
MethodInfluentialLabaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Learning internal representations by error propagation
RUMELHART, D. E., HINTON, G. E., & WILLIAMS, R. J. (1988) · Readings in Cognitive Science · doi:10.1016/b978-1-4832-1446-7.50035-2 · Cited by 3,002
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256 · cited 1×
“FakeAVCeleb [9] dataset was generated using videos from VoxCeleb2 [27] which consists of real Youtube videos of celebrities with five ethnic backgrounds: Caucasian (Americans), Caucasian (Europeans), African American, South Asian, and East Asian.”
MethodMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · International Workshop on Information Forensics and Security · doi:10.48550/arxiv.1809.00888 · Cited by 1,940
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872 · cited 4×
“Overall AUC ACC MesoInception-4 [28] 72.22 75.82 FTCN [30] 84.00 64.90 EfficientNet [29] 81.03 - AVoiD-DF [14] 89.20 83.70 AV-Lip-Sync [31] - 94.00 Ours-S-Capsule Forensics 99.30 99.20 Ours-S-Swin Transformer 91.65 90.51 Table II .”
MethodInfluentialCeleb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 1×
“Early datasets focused on visual modality generation, either utilizing authentic audio or excluding audio entirely, such as FaceForensics++ (FF++) [18], CelebDF [19], DFDC (Deepfake Detection Challenge) [13], and KoDF (Korean Deepfake Detection Dataset) [20].”
MethodSelf-citation
Research works printed but not linked: 1
- Roberts, 2022, Understanding the mel spectrogram
Cited by
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“…impact public opinion, political landscapes, and personal reputations, thereby compromising the authenticity of digital media and raising issues in social engineering and digital forensics where distinguishing genuine from AI-generated content becomes increasingly challenging [128], [129].”
BackgroundAudio–visual deepfake detection using articulatory representation learning
Wang, Y., & Huang, H. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104133 · Cited by 28
Hashmi, A., Shahzad, S. A., Lin, C., Tsao, Y., & Wang, H. (2025) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2025.3554477 · Cited by 47
“Similarly, for audio-visual forgery detection, the authors of [50] incorporate audio loss, video loss, and audio-visual loss to capture the inconsistency of multimodality and artifacts from individual modalities.”
Exposing Lip-syncing Deepfakes from Mouth Inconsistencies
Datta, S. K., Jia, S., & Lyu, S. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687902 · Cited by 43
“In recent developments, lip-syncing detection methodologies have focused on learning features related to speech-audio inconsistency and dis-synchronization [13, 14].”
BackgroundSelf-citationDeepFake video detection: Insights into model generalisation — A Systematic review
Ramanaharan, R., Guruge, D. B., & Agbinya, J. I. (2025) · Data and Information Management · doi:10.1016/j.dim.2025.100099 · Cited by 48
Metadata
- Authors
- Muppalla, S., Jia, S., & Lyu, S.
- Affiliations
- Cupertino High School,Cupertino,CA
- University at Buffalo, State University of New York (US)
- ORCID
- Lyu, Siwei
- Venue
- 2023 IEEE MIT Undergraduate Research Technology Conference (URTC)
- Publisher
- IEEE
- Published
- 2023-10-06
- Pages
- 1-5
- OpenAlex
- W4398789263
- Cited by
- 46
- References
- 37
- Citations by year
- 2026: 12 · 2025: 22 · 2024: 12
- FWCI
- 4.90
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE MIT Undergraduate Research Technology Conference (URTC) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Keywords
- deepfake detection, multimodal fusion, audio-visual features, binary classification, domain generalization, AI-generated content
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Integrating Audio-Visual Features For Multimodal Deepfake Detection. https://app.cochat.ai/papers/10.1109/urtc60662.2023.10534969
- The paper
- Muppalla, S., Jia, S., & Lyu, S. (2023). Integrating Audio-Visual Features For Multimodal Deepfake Detection. 2023 IEEE MIT Undergraduate Research Technology Conference (URTC). https://doi.org/10.1109/urtc60662.2023.10534969
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.