CoChat
Sign Up
PreprintCited by 280Digital Media Forensic DetectionFace recognition and analysisGenerative Adversarial Networks and Image Synthesis

DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms

Open access
Qi, Hua, Guo, Qing, Juefei-Xu, Felix, Xie, Xiaofei, Ma, Lei, Feng, Wei, Liu, Yang, Zhao, Jianjun·2020·arXiv
DOI 10.48550/arxiv.2006.07634arXiv 2006.07634
Open this paper in CoChat — free accountOpen PDF

Abstract

As the GAN-based face image and video generation techniques, widely known as DeepFakes, have become more and more matured and realistic, there comes a pressing and urgent demand for effective DeepFakes detectors. Motivated by the fact that remote visual photoplethysmography (PPG) is made possible by monitoring the minuscule periodic changes of skin color due to blood pumping through the face, we conjecture that normal heartbeat rhythms found in the real face videos will be disrupted or even entirely broken in a DeepFake video, making it a potentially powerful indicator for DeepFake detection. In this work, we propose DeepRhythm, a DeepFake detection technique that exposes DeepFakes by monitoring the heartbeat rhythms. DeepRhythm utilizes dual-spatial-temporal attention to adapt to dynamically changing face and fake types. Extensive experiments on FaceForensics++ and DFDC-preview datasets have confirmed our conjecture and demonstrated not only the effectiveness, but also the generalization capability of \emph{DeepRhythm} over different datasets by various DeepFakes generation techniques and multifarious challenging degradations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.07634

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Early temporal consistency methods mainly identified basic physiological anomalies, such as unnatural blinking patterns and irregular heartbeat rhythms.

    “The early temporal consistency methods mainly identified basic physiological anomalies, such as unnatural blinking patterns [60] and irregular heartbeat rhythms [61].”

    From Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges (2024) · cites this paper as [61]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [2][2] G. Balakrishnan, F. Durand, and J. Guttag. 2013. Detecting Pulse from Head Motions in Video. In 2013 IEEE Conference on Computer Vision and Pattern Recognition . 3430-3437. -
  2. [1][1] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen. 2018. MesoNet: a Compact Facial Video Forgery Detection Network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS) . -
  3. [6][6] Giovanni Cennini, Jeremie Arguel, Kaan Akşit, and Arno Leest. 2010. Heart rate monitoring via remote photoplethysmography with motion artifacts reduction. Optics express 18 (03 2010), 4867-75. https://doi.org/10.1364/OE.18.004867 - · DOI
  4. [7][7] Yupeng Cheng, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Shang-Wei Lin, Weisi Lin, Wei Feng, and Yang Liu. 2020. Pasadena: Perceptually Aware and Stealthy Adversarial Denoise Attack. arXiv preprint (2020). ± ± 0.265 0.033 0.90 0.83 0.43 20 ± ± 0.031 0.9 0.006 0.853 ± ± 0.146 ± ± 0.868 ± 0.133 ± ± 0.871 ± ± ± 0.145 ± ± ± 0.006 0.836 0.8 0.7 0.6 0.5 0.4 Blur 813 JPEG Compression 0.95 0.9 0.85 0.8 0.75 0 0.95 0.9 0.85 0.8 0.75 0 Accuracy 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 5 100 Accuracy 5 Accuracy 0.8 1 0.9 0.8 0.7 Sampling Accuracy 0.6 0.5 0.4 Accuracy Blur 868 0.8 Sampling 1 0.96 Anon. 0.94 Accuracy 0.92 0.9 Accuracy 0.9 0.8 0.7 0.6 0.5 0.4 0.88 0.98 Blur Sampling JPEG Compression 1 0.9 0.8 0.7 0.6 0.5 0.4 Accuracy 0.98 0.96 0.94 Accuracy 0.92 Sampling 0.9 Noise 0.95 0.9 0.85 0.8 Accuracy ± Accuracy ± ± ± ± Accuracy 1 0.9 0.8 0.7 0.6 0.5 0.4 ± ± 0.98 0.96 0.94 Accuracy 0.92 Noise 0.9 Anon. Sampling 25 15 15 20 Accuracy 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0 -
  5. [11][11] Kevin Dale, Kalyan Sunkavalli, Micah K. Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. 2011. Video Face Replacement. ACM Trans. Graph. 30, 6 (Dec. 2011), 1-10. https://doi.org/10.1145/2070781.2024164 - · DOI
  6. [12][12] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil Jain. 2019. On the Detection of Digital Face Manipulation. arXiv:cs.CV/1910.01717 -
  7. [16][16] Wei Feng, Ruize Han, Qing Guo, Jianke Zhu, and Song Wang. 2019. Dynamic Saliency-Aware Regularization for Correlation Filter-Based Object Tracking. IEEE TIP 28, 7 (2019), 3232-3245. -
  8. [19][19] P. Garrido, Levi Valgaerts, Hamid Sarmadi, I. Steiner, Kiran Varanasi, P. Pérez, and C. Theobalt. 2015. VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track. Computer Graphics Forum 34 (05 2015). https://doi.org/10.1111/cgf.12552 - · DOI
  9. [18][18] Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormählen, Patrick Pérez, and Christian Theobalt. 2014. Automatic Face Reenactment. 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014), 4217-4224. -
  10. [20][20] Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. Proceedings of SPIE - The International Society for Optical Engineering 9409 (03 2015). -
  11. [22][22] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Networks. arXiv:stat.ML/1406.2661 -
  12. [21][21] E. Gonzalez-Sosa, J. Fierrez, R. Vera-Rodriguez, and F. Alonso-Fernandez. 2018. Facial Soft Biometrics for Recognition in the Wild: Recent Works, Annotation, and COTS Evaluation. IEEE Transactions on Information Forensics and Security 13, 8 (2018), 2001-2014. -
  13. [23][23] Qing Guo, Wei Feng, Ce Zhou, Rui Huang, Liang Wan, and Song Wang. 2017. Learning Dynamic Siamese Network for Visual Object Tracking. In ICCV . 17811789. -
  14. [25][25] Qing Guo, Ruize Han, Wei Feng, Zhihao Chen, and Liang Wan. 2020. Selective Spatial Regularization by Reinforcement Learned Decision Making for Object Tracking. IEEE TIP 29 (2020), 2999-3013. -
  15. [24][24] Qing Guo, Wei Feng, Ce Zhou, Chi-Man Pun, and Bin Wu. 2017. StructureRegularized Compressive Tracking With Online Data-Driven Sampling. IEEE TIP 26, 12 (2017), 5692-5705. -
  16. [26][26] Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Jian Wang, Bing Yu, Wei Feng, and Yang Liu. 2020. Watch out! Motion is Blurring the Vision of Your Deep Neural Networks. arXiv preprint arXiv:2002.03500 (2020). - · arXiv
  17. [28][28] D. Güera and E. J. Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) . 1-6. -
  18. [27][27] Qing Guo, Xiaofei Xie, Felix Juefei-Xu, Lei Ma, Zhongguo Li, Wanli Xue, Wei Feng, and Yang Liu. 2020. SPARK: Spatial-aware Online Incremental Attack Against Visual Tracking. European Conference on Computer Vision (ECCV) (2020). -
  19. [30][30] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen. 2019. AttGAN: Facial Attribute Editing by Only Changing What You Want. IEEE Transactions on Image Processing 28, 11 (2019), 5464-5478. -
  20. [34][34] Yihao Huang, Felix Juefei-Xu, Run Wang, Qing Guo, Xiaofei Xie, Lei Ma, Jianwen Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. FakeLocator: Robust Localization of GAN-Based Face Manipulations. arXiv preprint arXiv:2001.09598 - · arXiv
  21. [33][33] Yihao Huang, Felix Juefei-Xu, Run Wang, Qing Guo, Lei Ma, Xiaofei Xie, Jianwen Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstruction. ACM International Conference on Multimedia (ACM MM) (2020). (2020). -
  22. [36][36] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive Growing of GANs for Improved Quality, Stability, and Variation. arXiv:cs.NE/1710.10196 -
  23. [35][35] Kenneth Humphreys, Tomas Ward, and Charles Markham. 2007. Noncontact simultaneous dual wavelength photoplethysmography: A further step toward noncontact pulse oximetry. The Review of scientific instruments 78 (05 2007), 044304. https://doi.org/10.1063/1.2724789 - · DOI
  24. [37][37] Tero Karras, Samuli Laine, and Timo Aila. 2018. A Style-Based Generator Architecture for Generative Adversarial Networks. arXiv:cs.NE/1812.04948 -
  25. [39][39] X. Li, J. Chen, G. Zhao, and M. Pietikäinen. 2014. Remote Heart Rate Measurement from Face Videos under Realistic Situations. In 2014 IEEE Conference on Computer Vision and Pattern Recognition . 4264-4271. -
  26. [38][38] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2019. Analyzing and Improving the Image Quality of StyleGAN. arXiv:cs.CV/1912.04958 -
  27. [41][41] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen. 2019. STGAN: A Unified Selective Transfer Network for Arbitrary Image Attribute Editing. arXiv:cs.CV/1904.09709 -
  28. [43][43] Lei Ma, Felix Juefei-Xu, Jiyuan Sun, Chunyang Chen, Ting Su, Fuyuan Zhang, Minhui Xue, Bo Li, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems. In The 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) . -
  29. [45][45] Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepMutation: Mutation Testing of Deep Learning Systems. In The 29th IEEE International Symposium on Software Reliability Engineering (ISSRE) . -
  30. [44][44] Lei Ma, Felix Juefei-Xu, Minhui Xue, Bo Li, Li Li, Yang Liu, and Jianjun Zhao. 2019. DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems. Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) (2019). -
  31. [46][46] Scott McCloskey and Michael Albright. 2018. Detecting GAN-generated Imagery using Color Cues. arXiv:cs.CV/1812.08247 -
  32. [48][48] X. Niu, S. Shan, H. Han, and X. Chen. 2020. RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation. IEEE Transactions on Image Processing 29 (2020), 2409-2423. -
  33. [47][47] Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. 2019. Use of a Capsule Network to Detect Fake Images and Videos. arXiv:cs.CV/1910.12467 -
  34. [50][50] Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed Elgharib, Frédo Durand, William T. Freeman, and Wojciech Matusik. 2018. Learning-based Video Motion Magnification. arXiv:cs.CV/1804.02684 -
  35. [54][54] M. Poh, D. J. McDuff, and R. W. Picard. 2011. Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam. IEEE Transactions on Biomedical Engineering 58, 1 (2011), 7-11. -
  36. [53][53] Xunyu Pan, Xing Zhang, and Siwei Lyu. 2012. Exposing image splicing with inconsistent local noise variances. 2012 IEEE International Conference on Computational Photography, ICCP 2012 (04 2012). -
  37. [55][55] Ming-Zher Poh, Daniel McDuff, and Rosalind Picard. 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics express 18 (05 2010), 10762-74. https://doi.org/10.1364/OE.18.010762 - · DOI
  38. [57][57] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2019. FaceForensics++: Learning to Detect Manipulated Facial Images. In International Conference on Computer Vision (ICCV) . -
  39. [58][58] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. 2019. Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. arXiv:cs.CV/1905.00582 -
  40. [60][60] Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019. Deferred Neural Rendering: Image Synthesis using Neural Textures. arXiv:cs.CV/1904.12356 -
  41. [59][59] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. 2016. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. AAAI Conference on Artificial Intelligence (02 2016). -
  42. [61][61] Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time Expression Transfer for Facial Reenactment. ACM Transactions on Graphics 34 (10 2015), 1-14. https: [//doi.org/10.1145/2816795.2818056](https://doi.org/10.1145/2816795.2818056) - · DOI
  43. [62][62] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. 2016. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2387-2395. -
  44. [64][64] Run Wang, Felix Juefei-Xu, Qing Guo, Yihao Huang, Xiaofei Xie, Lei Ma, and Yang Liu. 2020. Amora: Black-box Adversarial Morphing Attack. ACM International Conference on Multimedia (ACM MM) (2020). -
  45. [66][66] Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, and Yang Liu. 2020. FakeSpotter: A Simple yet Robust Baseline for Spotting AISynthesized Fake Faces. International Joint Conference on Artificial Intelligence -
  46. [65][65] Run Wang, Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, and Yang Liu. 2020. DeepSonar: Towards Effective and Robust Detection of AI-Synthesized Fake Voices. ACM International Conference on Multimedia (ACM MM) (2020). (IJCAI) (2020). - [68] Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. 2019. DeepHunter: A CoverageGuided Fuzz Testing Framework for Deep Neural Networks. In ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) . -
  47. [67][67] Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William T. Freeman. 2012. Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Transactions on Graphics (Proc. SIGGRAPH 2012) 31, 4 (2012). - [69] Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. 2019. Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-toend Deep Learning Solution with Video Enhancement. arXiv:eess.IV/1907.11921 - [71] Peng Zhou, Xintong Han, Vlad I. Morariu, and Larry S. Davis. 2018. Two-Stream Neural Networks for Tampered Face Detection. arXiv:cs.CV/1803.11276 - [70] Ce Zhou, Qing Guo, Liang Wan, and Wei Feng. 2017. Selective object and context tracking. In ICASSP . 1947-1951.

Cited by

Sorted by citations

Authors
Qi, H., Guo, Q., Juefei-Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., & Zhao, J.
Affiliations
  • Kyushu University (JP)
  • Nanyang Technological University (SG)
  • Alibaba Group (United States) (US)
  • Tianjin University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
4318-4327
DOI
10.48550/arxiv.2006.07634
arXiv
2006.07634
OpenAlex
W3092997524
Cited by
280
References
6
Citations by year
2026: 26 · 2025: 52 · 2024: 61 · 2023: 52 · 2022: 51 · 2021: 28 · 2020: 10
FWCI
10.71
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • National Natural Science Foundation of China (U1703261, 61871258)
  • Japan Society for the Promotion of Science (19H04086, 20H04168, 19K24348)
  • JST-Mirai Program
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
deepfake detection, remote photoplethysmography, deepfakes, FaceForensics++, DFDC

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review