The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Abstract
We present a learning-based method for detecting real and fake deepfake multimedia content. To maximize information for learning, we extract and analyze the similarity between the two audio and visual modalities from within the same video. Additionally, we extract and compare affective cues corresponding to perceived emotion from the two modalities within a video to infer whether the input video is "real" or "fake". We propose a deep learning network, inspired by the Siamese network architecture and the triplet loss. To validate our model, we report the AUC metric on two large-scale deepfake detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach with several SOTA deepfake detection methods and report per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of our knowledge, ours is the first approach that simultaneously exploits audio and video modalities and also perceived emotions from the two modalities for deepfake detection.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To overcome these limitations, multimodal approaches have been recently proposed, able to combine information from various domains to enhance the accuracy of the detection process [7,8].”
From A Robust Approach to Multimodal Deepfake Detection (2023) · cites this paper as [8]
In the citing paper
“They evaluate their technique to multiple SOTA deepfake detection approaches, reporting pervideo AUCs of 84.4% on the DFDC dataset and 96.6% on the DF-TIMIT dataset.”
From Deepfake detection using deep learning methods: A systematic and comprehensive review (2024) · cites this paper as Mittal et al. (2020)
In the citing paper
“Their method is the first to use both audio and video modalities and perceived emotions from two modalities to detect deepfakes.”
From Deepfake detection using deep learning methods: A systematic and comprehensive review (2024) · cites this paper as Mittal et al. (2020)
In the citing paper
“The performance of any deepfake detection method is highly dependent on the quantitative and qualitative aspects of the datasets used for development [18, 21, 25, 40, 44, 47, 49, 51, 52, 61, 63, 69, 72, 75].”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [40]
In the citing paper
“Then, compared with several SOTA DFD methods, such as Two-Stream, MesoNet, HeadPose, FWA, VA, Xception, Multi-task, Capsule, and DSP-FWA, they report a per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets.”
From A Comprehensive Review of DeepFake Detection Using Advanced Machine Learning and Fusion Methods (2023) · cites this paper as [21]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 5×
“[49] use the XceptionNet, and Zhou et al.”
MethodInfluentialMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 2×
“There are many other applications of video processing that use and combine multiple modalities for audio-visual speech recognition [28], emotion recognition [40, 63], and language and vision tasks [17, 29].”
BackgroundMethodDeepfake Video Detection Using Recurrent Neural Networks
Guera, D., & Delp, E. J. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639163 · Cited by 1,285 · cited 2×
“3413570 23, 33, 49] and algorithms [9, 26, 36, 38, 42, 43, 49, 50, 57, 61, 64] for deepfake detection.”
MethodExposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682602 · Cited by 951
Facial signs of emotional experience.
Ekman, P., Freisen, W. V., & Ancoli, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077722 · Cited by 886 · cited 1×
“The CMU-MOSEI dataset describes the perceived emotion space with 6 discrete emotions following the Ekman model [24]: happy, sad, angry, fearful, surprise, and disgust, and a “neutral” emotion to denote the absence of any of these emotions.”
MethodDeep Fakes: A Looming Challenge for Privacy, Democracy, and National Security
Chesney, R., & Citron, D. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3213954 · Cited by 880 · cited 1×
“deepfakes [22], that use artificial intelligence to present false media as realistic in highly sophisticated and convincing ways.”
BackgroundExploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
Matern, F., Riess, C., & Stamminger, M. (2019) · 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw.2019.00020 · Cited by 824 · cited 4×
“(5) VA [38] focuses on capturing visual artifacts in the eyes, teeth and facial contours of synthesized faces.”
BackgroundMethodInfluential
Research works printed but not linked: 10
- Kingsley Oryina Akputu , Kah Phooi Seng, and Yun Li Lee . 2013 . Facial emotion recognition for intelligent tutoring environment. In IMLCS. 9--13. Kingsley Oryina Akputu, Kah Phooi Seng, and Yun Li Lee. 2013. Facial emotion recognition for intelligent tutoring environment. In IMLCS. 9--13.
- Themis Balomenos , Amaryllis Raouzaiou , Spiros Ioannou , Athanasios Drosopoulos , Kostas Karpouzis , and Stefanos Kollias . 2004 . Emotion analysis in man-machine interaction systems . In International Workshop on Machine Learning for Multimodal Interaction. Springer, 318--328 . Themis Balomenos, Amaryllis Raouzaiou, Spiros Ioannou, Athanasios Drosopoulos, Kostas Karpouzis, and Stefanos Kollias. 2004. Emotion analysis in man-machine interaction systems. In International Workshop on Machine Learning for Multimodal Interaction. Springer, 318--328.
- Tadas Baltruvs aitis, Peter Robinson , and Louis-Philippe Morency . 2016 . Openface: an open source facial behavior analysis toolkit . In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10 . Tadas Baltruvs aitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10.
- Uttaran Bhattacharya , Trisha Mittal , Rohan Chandra , Tanmay Randhavane , Aniket Bera , and Dinesh Manocha . 2019 . Step: Spatial temporal graph convolutional networks for emotion perception from gaits. arXiv preprint arXiv:1910.12906 (2019). Uttaran Bhattacharya, Trisha Mittal, Rohan Chandra, Tanmay Randhavane, Aniket Bera, and Dinesh Manocha. 2019. Step: Spatial temporal graph convolutional networks for emotion perception from gaits. arXiv preprint arXiv:1910.12906 (2019). · arXiv
- Vladimir Chernykh and Pavel Prikhodko . 2017. Emotion recognition from speech with recurrent neural networks. arXiv preprint arXiv:1701.08071 ( 2017 ). Vladimir Chernykh and Pavel Prikhodko. 2017. Emotion recognition from speech with recurrent neural networks. arXiv preprint arXiv:1701.08071 (2017). · arXiv
- R Benjamin Knapp , Jonghwa Kim , and Elisabeth André . 2011. Physiological signals and their use in augmenting emotion recognition for human--machine interaction . In Emotion-oriented systems . Springer , 133--159. R Benjamin Knapp, Jonghwa Kim, and Elisabeth André. 2011. Physiological signals and their use in augmenting emotion recognition for human--machine interaction. In Emotion-oriented systems . Springer, 133--159.
- Gangeshwar Krishnamurthy , Navonil Majumder , Soujanya Poria , and Erik Cambria . 2018. A deep learning approach for multimodal deception detection. arXiv preprint arXiv:1803.00344 ( 2018 ). Gangeshwar Krishnamurthy, Navonil Majumder, Soujanya Poria, and Erik Cambria. 2018. A deep learning approach for multimodal deception detection. arXiv preprint arXiv:1803.00344 (2018). · arXiv
- Yuezun Li , Xin Yang , Pu Sun , Honggang Qi , and Siwei Lyu . 2019 . Celeb-df: A new dataset for deepfake forensics. arXiv preprint arXiv:1909.12962 (2019). Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2019. Celeb-df: A new dataset for deepfake forensics. arXiv preprint arXiv:1909.12962 (2019). · arXiv
- Luisa Verdoliva and Paolo Bestagini . 2019 b . Multimedia Forensics. In Proceedings of the 27th ACM International Conference on Multimedia ( Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 2701--2702. https://doi.org/10.1145/3343031.3350542 10.1145/3343031.3350542 Luisa Verdoliva and Paolo Bestagini. 2019 b. Multimedia Forensics. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 2701--2702. https://doi.org/10.1145/3343031.3350542 · DOI
- Amir Zadeh , Paul Pu Liang , Navonil Mazumder , Soujanya Poria , Erik Cambria , and Louis-Philippe Morency . 2018 a. Memory fusion network for multi-view sequential learning . In Thirty-Second AAAI Conference on Artificial Intelligence . Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence .
Cited by
Deepfake Detection: A Systematic Literature Review
Rana, M. S., Nobi, M. N., Murali, B., & Sung, A. H. (2022) · IEEE Access · doi:10.1109/access.2022.3154404 · Cited by 573
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Haliassos, A., Vougioukas, K., Petridis, S., & Pantic, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00500 · Cited by 537
“datasets [46, 38, 30, 16], it is possible to train deep convolutional neural networks (CNNs) to detect fake videos [3, 25, 64, 4, 14, 45, 42, 36].”
BackgroundDeepfake detection using deep learning methods: A systematic and comprehensive review
Heidari, A., Jafari Navimipour, N., Dag, H., & Unal, M. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1520 · Cited by 349
“In addition, Mittal et al. (2020) demonstrated a learning-based technique for distinguishing between genuine and phony deepfake multimedia content.”
BackgroundExploring Temporal Coherence for More General Video Face Forgery Detection
Zheng, Y., Bao, J., Chen, D., Zeng, M., & Wen, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01477 · Cited by 320
“Unlike previous works, they either directly apply 3D convolution network [15, 21] on video or detect a specific kind of incoherence, such as irregular eye blinking [33], lip motions [22] or emotion [37].”
Background
Metadata
- Authors
- Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D.
- Affiliations
- University of Maryland, College Park (US)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 2823-2832
- OpenAlex
- W3093010840
- Cited by
- 319
- References
- 49
- Citations by year
- 2026: 48 · 2025: 81 · 2024: 48 · 2023: 61 · 2022: 45 · 2021: 34 · 2020: 2
- FWCI
- 10.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Keywords
- deepfake detection, multimodal fusion, emotional cues, perceived emotion, Siamese network, triplet loss, DFDC
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues. https://app.cochat.ai/papers/10.1145/3394171.3413570
- The paper
- Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020). Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3413570
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.