Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.

Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues

Mittal, Trisha, Bhattacharya, Uttaran, Chandra, Rohan, Bera, Aniket, Manocha, Dinesh · 2020 · Proceedings of the 28th ACM International Conference on Multimedia

Abstract

We present a learning-based method for detecting real and fake deepfake multimedia content. To maximize information for learning, we extract and analyze the similarity between the two audio and visual modalities from within the same video. Additionally, we extract and compare affective cues corresponding to perceived emotion from the two modalities within a video to infer whether the input video is "real" or "fake". We propose a deep learning network, inspired by the Siamese network architecture and the triplet loss. To validate our model, we report the AUC metric on two large-scale deepfake detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach with several SOTA deepfake detection methods and report per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of our knowledge, ours is the first approach that simultaneously exploits audio and video modalities and also perceived emotions from the two modalities for deepfake detection.

Overview

Integrity signals

Registry

Crossref · 10.1145/3394171.3413570

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 49 references

Research works printed but not linked: 10

  1. Kingsley Oryina Akputu , Kah Phooi Seng, and Yun Li Lee . 2013 . Facial emotion recognition for intelligent tutoring environment. In IMLCS. 9--13. Kingsley Oryina Akputu, Kah Phooi Seng, and Yun Li Lee. 2013. Facial emotion recognition for intelligent tutoring environment. In IMLCS. 9--13.
  2. Themis Balomenos , Amaryllis Raouzaiou , Spiros Ioannou , Athanasios Drosopoulos , Kostas Karpouzis , and Stefanos Kollias . 2004 . Emotion analysis in man-machine interaction systems . In International Workshop on Machine Learning for Multimodal Interaction. Springer, 318--328 . Themis Balomenos, Amaryllis Raouzaiou, Spiros Ioannou, Athanasios Drosopoulos, Kostas Karpouzis, and Stefanos Kollias. 2004. Emotion analysis in man-machine interaction systems. In International Workshop on Machine Learning for Multimodal Interaction. Springer, 318--328.
  3. Tadas Baltruvs aitis, Peter Robinson , and Louis-Philippe Morency . 2016 . Openface: an open source facial behavior analysis toolkit . In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10 . Tadas Baltruvs aitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10.
  4. Uttaran Bhattacharya , Trisha Mittal , Rohan Chandra , Tanmay Randhavane , Aniket Bera , and Dinesh Manocha . 2019 . Step: Spatial temporal graph convolutional networks for emotion perception from gaits. arXiv preprint arXiv:1910.12906 (2019). Uttaran Bhattacharya, Trisha Mittal, Rohan Chandra, Tanmay Randhavane, Aniket Bera, and Dinesh Manocha. 2019. Step: Spatial temporal graph convolutional networks for emotion perception from gaits. arXiv preprint arXiv:1910.12906 (2019). · arXiv
  5. Vladimir Chernykh and Pavel Prikhodko . 2017. Emotion recognition from speech with recurrent neural networks. arXiv preprint arXiv:1701.08071 ( 2017 ). Vladimir Chernykh and Pavel Prikhodko. 2017. Emotion recognition from speech with recurrent neural networks. arXiv preprint arXiv:1701.08071 (2017). · arXiv
  6. R Benjamin Knapp , Jonghwa Kim , and Elisabeth André . 2011. Physiological signals and their use in augmenting emotion recognition for human--machine interaction . In Emotion-oriented systems . Springer , 133--159. R Benjamin Knapp, Jonghwa Kim, and Elisabeth André. 2011. Physiological signals and their use in augmenting emotion recognition for human--machine interaction. In Emotion-oriented systems . Springer, 133--159.
  7. Gangeshwar Krishnamurthy , Navonil Majumder , Soujanya Poria , and Erik Cambria . 2018. A deep learning approach for multimodal deception detection. arXiv preprint arXiv:1803.00344 ( 2018 ). Gangeshwar Krishnamurthy, Navonil Majumder, Soujanya Poria, and Erik Cambria. 2018. A deep learning approach for multimodal deception detection. arXiv preprint arXiv:1803.00344 (2018). · arXiv
  8. Yuezun Li , Xin Yang , Pu Sun , Honggang Qi , and Siwei Lyu . 2019 . Celeb-df: A new dataset for deepfake forensics. arXiv preprint arXiv:1909.12962 (2019). Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2019. Celeb-df: A new dataset for deepfake forensics. arXiv preprint arXiv:1909.12962 (2019). · arXiv
  9. Luisa Verdoliva and Paolo Bestagini . 2019 b . Multimedia Forensics. In Proceedings of the 27th ACM International Conference on Multimedia ( Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 2701--2702. https://doi.org/10.1145/3343031.3350542 10.1145/3343031.3350542 Luisa Verdoliva and Paolo Bestagini. 2019 b. Multimedia Forensics. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 2701--2702. https://doi.org/10.1145/3343031.3350542 · DOI
  10. Amir Zadeh , Paul Pu Liang , Navonil Mazumder , Soujanya Poria , Erik Cambria , and Louis-Philippe Morency . 2018 a. Memory fusion network for multi-view sequential learning . In Thirty-Second AAAI Conference on Artificial Intelligence . Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence .

Cited by

See all 322

Metadata

Authors
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D.
Affiliations
University of Maryland, College Park (US)
Venue
Proceedings of the 28th ACM International Conference on Multimedia
Publisher
ACM
Published
2020-10-12
Pages
2823-2832
OpenAlex
W3093010840
Cited by
319
References
49
Citations by year
2026: 48 · 2025: 81 · 2024: 48 · 2023: 61 · 2022: 45 · 2021: 34 · 2020: 2
FWCI
10.95
Type
conference paper
Language
en
Open access
closed
Locations
ACM International Conference on Multimedia (ACM MM) (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
Keywords
deepfake detection, multimodal fusion, emotional cues, perceived emotion, Siamese network, triplet loss, DFDC

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues. https://app.cochat.ai/papers/10.1145/3394171.3413570
The paper
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020). Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3413570

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.