Do Deepfakes Feel Emotions? A Semantic Approach to Detecting Deepfakes Via Emotional Inconsistencies
Open accessAbstract
Recent advances in deep learning and computer vision have spawned a new class of media forgeries known as deepfakes, which typically consist of artificially generated human faces or voices. The creation and distribution of deepfakes raise many legal and ethical concerns. As a result, the ability to distinguish between deepfakes and authentic media is vital. While deepfakes can create plausible video and audio, it may be challenging for them to to generate content that is consistent in terms of high-level semantic features, such as emotions. Unnatural displays of emotion, measured by features such as valence and arousal, can provide significant evidence that a video has been synthesized. In this paper, we propose a novel method for detecting deepfakes of a human speaker using the emotion predicted from the speaker’s face and voice. The proposed technique leverages Long Short-Term Memory (LSTM) networks that predict emotion from audio and video Low-Level Descriptors (LLDs). Predicted emotion in time is used to classify videos as authentic or deepfakes through an additional supervised classifier.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 1× · see citing works
“In the second case, emotions are described by means of the values of quantitative features, such as valence and arousal and organized in a two-dimensional circumplex space [48].”
Background3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,222 · see citing works
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · see citing works
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791 · see citing works
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · see citing works
Ekman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,975 · see citing works
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872 · see citing works
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 1× · see citing works
“Several approaches have been proposed to identify both video [25, 36, 32, 3, 12] and audio [34, 5] deepfakes, and multiple deepfake databases have been created to support this research [46, 33, 19, 56].”
BackgroundThe Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
OpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · see citing works
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644 · see citing works
Affective Computing and Sentiment Analysis
Cambria, E. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.31 · Cited by 1,445 · see citing works
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,410 · see citing works
Deepfake Video Detection Using Recurrent Neural Networks
Guera, D., & Delp, E. J. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639163 · Cited by 1,285 · see citing works
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196 · see citing works
In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
Li, Y., Chang, M.-C., & Lyu, S. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630787 · Cited by 1,089 · see citing works
Media Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979 · see citing works
Efficient Neural Audio SynthesisPreprint
Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., van den Oord, A., Dieleman, S., & Kavukcuoglu, K. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.08435 · Cited by 948 · see citing works
Context-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939 · see citing works
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
Matern, F., Riess, C., & Stamminger, M. (2019) · 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw.2019.00020 · Cited by 824 · see citing works
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785 · see citing works
Akçay, M. B., & Oğuz, K. (2019) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 743 · see citing works
ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection
Todisco, M., Wang, X., Vestman, V., Sahidullah, M., Delgado, H., Nautsch, A., Yamagishi, J., Evans, N., Kinnunen, T., & Lee, K. A. (2019) · arXiv · doi:10.48550/arxiv.1904.05441 · Cited by 688 · see citing works
ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection
Todisco, M., Wang, X., Vestman, V., Sahidullah, M., Delgado, H., Nautsch, A., Yamagishi, J., Evans, N., Kinnunen, T. H., & Lee, K. A. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2249 · Cited by 687 · see citing works
McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schroder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.20 · Cited by 665 · see citing works
Emotion Recognition from Physiological Signal Analysis: A Review
Egger, M., Ley, M., & Hanke, S. (2019) · Electronic Notes in Theoretical Computer Science · doi:10.1016/j.entcs.2019.04.009 · Cited by 568 · see citing works
Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · arXiv · doi:10.48550/arxiv.1906.06876 · Cited by 551 · see citing works
Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS) · doi:10.1109/btas46853.2019.9185974 · Cited by 549 · see citing works
DeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · see citing works
The deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427 · see citing works
(2019) · Cited by 415 · see citing works
Reexamining the circumplex model of affect.
Remington, N. A., Fabrigar, L. R., & Visser, P. S. (2000) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.79.2.286 · Cited by 403 · see citing works
A comparison of features for synthetic speech detection
Sahidullah, M., Kinnunen, T., & Hanilçi, C. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-472 · Cited by 382 · see citing works
3D Constrained Local Model for rigid and non-rigid facial tracking
Baltrušaitis, T., Robinson, P., & Morency, L. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247980 · Cited by 320 · see citing works
State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications
Zollhöfer, M., Thies, J., Garrido, P., Bradley, D., Beeler, T., Pérez, P., Stamminger, M., Nießner, M., & Theobalt, C. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13382 · Cited by 307 · see citing works
Detecting Deep-Fake Videos from Phoneme-Viseme Mismatches
Agarwal, S., Farid, H., Fried, O., & Agrawala, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00338 · Cited by 234 · see citing works
Reexamining the circumplex model of affect.
Remington, N. A., Fabrigar, L. R., & Visser, P. S. (2000) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.79.2.286 · Cited by 228 · see citing works
Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis
Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., & Morency, L. (2017) · Proceedings · doi:10.1109/icdm.2017.134 · Cited by 215 · see citing works
(2020) · arXiv (Cornell University) · Cited by 204 · see citing works
Rubin, D. C., & Talarico, J. M. (2009) · Memory · doi:10.1080/09658210903130764 · Cited by 163 · see citing works
Advances in anti-spoofing: from the perspective of ASVspoof challenges
Kamble, M. R., Sailor, H. B., Patil, H. A., & Li, H. (2020) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2019.21 · Cited by 102 · see citing works
Computationally modeling human emotion
Marsella, S., & Gratch, J. S. (2014) · Communications of the ACM · doi:10.1145/2631912 · Cited by 92 · see citing works
Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
Chen, S., & Jin, Q. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967286 · Cited by 87 · see citing works
Sun, L., Lian, Z., Tao, J., Liu, B., & Niu, M. (2020) · Proceedings of the 1st International on Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop · doi:10.1145/3423327.3423672 · Cited by 81 · see citing works
End-to-end spoofing detection with raw waveform CLDNNS
Dinkel, H., Chen, N., Qian, Y., & Yu, K. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953080 · Cited by 69 · see citing works
Using i-vector space model for emotion recognition
Xia, R., & Liu, Y. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-128 · Cited by 63 · see citing works
Adversarial Multimedia Forensics: Overview and Challenges Ahead
Barni, M., Stamm, M. C., & Tondi, B. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553305 · Cited by 62 · see citing works
Self-citationMultimodal Speech Emotion Recognition and Ambiguity ResolutionPreprint
Sahu, G. (2019) · arXiv.org · doi:10.48550/arxiv.1904.06022 · Cited by 53 · see citing works
Atmaja, B. T., & Akagi, M. (2021) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1896/1/012004 · Cited by 48 · see citing works
Detecting AI-Synthesized Speech Using Bispectral Analysis
AlBadawy, E. A., Lyu, S., & Farid, H. (2019) · Cited by 36 · see citing works
Multimodal Utterance-level Affect Analysis using Visual, Audio and Text FeaturesPreprint
Deng, D., Zhou, Y., Pi, J., & Shi, B. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.00625 · Cited by 29 · see citing works
Video Face Manipulation Detection Through Ensemble of CNNs
Bonettini, N., Cannas, E. D., Mandelli, S., Bondi, L., Bestagini, P., & Tubaro, S. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412711 · Cited by 27 · see citing works
Self-citationRingeval, F., Schuller, B. W., Valstar, M., et al. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350550 · Cited by 24 · see citing works
THE CONCEPTUALISATION OF EMOTION QUALIA: SEMANTIC CLUSTERING OF EMOTIONAL TWEETS
Bann, E. Y., & Bryson, J. J. (2013) · doi:10.1142/9789814458849_0019 · Cited by 22 · see citing works
Emotion recognition with boosted tree classifiers
Day, M. (2013) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2522848.2531740 · Cited by 20 · see citing works
AFFEC TIVE COMPUTING AND SENTIMENT ANALYSIS Affective Computing and Sentiment Analysis
Cambria, E., Das, D., Bandyopadhyay, S., & Feraco, A. (2016) · Socio-affective computing · doi:10.1007/978-3-319-55394-8_1 · Cited by 14 · see citing works
A Literature Review on Emotion Recognition Using Various Methods
Khan, R., & Sharif, O. (2017) · Global Journal of Computer Science and Technology · Cited by 12 · see citing works
Bio-signal based emotion detection device
Rathod, P. J., George, K., & Shinde, N. (2016) · Wearable and Implantable Body Sensor Networks · doi:10.1109/bsn.2016.7516241 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [34]34. A. Lieto, D. Moro, F. Devoti, C. Parera, V. Lipari, P. Bestagini, and S. Tubaro. "Hello? Who Am I Talking to?" A Shallow CNN Approach for Human vs. Bot Speech Clas- sification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019. 1, 3
- [45]45. F. Ringeval, B. Schuller, M. Valstar, N. Cummins, R. Cowie, and M. Pantic. AVEC'19: Audio/visual emotion challenge and workshop. In ACM International Conference on Multi- media, 2019. 2, 3, 4
- [55]55. J. Thies, M. Zollhöfer, and M. Nießner. Deferred neural ren- dering: Image synthesis using neural textures. ACM Trans- actions on Graphics (TOG), 38:1-12, 2019. 2
Cited by
Sorted by citations
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543 · see citing works
Deepfake Attacks: Generation, Detection, Datasets, Challenges, and Research Directions
Naitali, A., Ridouani, M., Salahdine, F., & Kaabouch, N. (2023) · Computers · doi:10.3390/computers12100216 · Cited by 117 · see citing works
Deepfake Media Forensics: Status and Future Challenges
Amerini, I., Barni, M., Battiato, S., Bestagini, P., Boato, G., Bruni, V., Caldelli, R., De Natale, F., De Nicola, R., Guarnera, L., et al. (2025) · Journal of Imaging · doi:10.3390/jimaging11030073 · Cited by 113 · see citing works
Self-citationA Robust Approach to Multimodal Deepfake Detection
Salvi, D., Liu, H., Mandelli, S., Bestagini, P., Zhou, W., Zhang, W., & Tubaro, S. (2023) · Journal of Imaging · doi:10.3390/jimaging9060122 · Cited by 96 · see citing works
Self-citationDeepfake Speech Detection Through Emotion Recognition: A Semantic Approach
Conti, E., Salvi, D., Borrelli, C., Hosler, B., Bestagini, P., Antonacci, F., Sarti, A., Stamm, M. C., & Tubaro, S. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747186 · Cited by 69 · see citing works
Self-citationTIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media Detection
Salvi, D., Hosler, B., Bestagini, P., Stamm, M. C., & Tubaro, S. (2023) · IEEE Access · doi:10.1109/access.2023.3276480 · Cited by 65 · see citing works
Self-citationVisual Deepfake Detection: Review of Techniques, Tools, Limitations, and Future Prospects
Ur Rehman Ahmed, N., Badshah, A., Adeel, H., Tajammul, A., Daud, A., & Alsahfi, T. (2025) · IEEE Access · doi:10.1109/access.2024.3523288 · Cited by 45 · see citing works
Combining Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detection
Attorresi, L., Salvi, D., Borrelli, C., Bestagini, P., & Tubaro, S. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-37742-6_21 · Cited by 30 · see citing works
Self-citation
- Authors
- Hosler, B., Salvi, D., Murray, A., Antonacci, F., Bestagini, P., Tubaro, S., & Stamm, M. C.
- Affiliations
- Drexel University (US)
- Politecnico di Milano (IT)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 1013-1022
- OpenAlex
- W3176193246
- Cited by
- 78
- References
- 59
- Citations by year
- 2026: 9 · 2025: 15 · 2024: 15 · 2023: 25 · 2022: 11 · 2021: 3
- FWCI
- 4.42
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion, other-oa)
- Funders
- Arm
- Air Force Research Laboratory
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- deepfake detection, emotion prediction, valence and arousal, low-level descriptors, LSTM networks, video authenticity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
