Media Forensics Considerations on DeepFake Detection with Hand-Crafted Features
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
DeepFake detection is a novel task for media forensics and is currently receiving a lot of research attention due to the threat these targeted video manipulations propose to the trust placed in video footage. The current trend in DeepFake detection is the application of neural networks to learn feature spaces that allow them to be distinguished from unmanipulated videos. In this paper, we discuss, with features hand-crafted by domain experts, an alternative to this trend. The main advantage that hand-crafted features have over learned features is their interpretability and the consequences this might have for plausibility validation for decisions made. Here, we discuss three sets of hand-crafted features and three different fusion strategies to implement DeepFake detection. Our tests on three pre-existing reference databases show detection performances that are under comparable test conditions (peak AUC > 0.95) to those of state-of-the-art methods using learned features. Furthermore, our approach shows a similar, if not better, generalization behavior than neural network-based methods in tests performed with different training and test sets. In addition to these pattern recognition considerations, first steps of a projection onto a data-centric examination approach for forensics process modeling are taken to increase the maturity of the present investigation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 1×
“Layer-wise Relevance Propagation (LRP; [27]), it can be shown that these methods assign meaning to regions in the input (see e.”
BackgroundFaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846
Deepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Evaluating the Visualization of What a Deep Neural Network Has Learned
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Muller, K.-R. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2599820 · Cited by 1,164
In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
Li, Y., Chang, M.-C., & Lyu, S. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630787 · Cited by 1,089
Media Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979 · cited 1×
“Unfortunately, the authors of some survey papers, such as [25,26], refer to learned features using specially designed networks (such as those proposed in [15,18]) and also as being "hand-crafted".”
BackgroundVaccari, C., & Chadwick, A. (2020) · Social Media + Society · doi:10.1177/2056305120903408 · Cited by 978 · cited 1×
“Papers such as [1,2] elaborate on the effect of DeepFakes on current politics, disinformation and trust.”
BackgroundExploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
Matern, F., Riess, C., & Stamminger, M. (2019) · 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw.2019.00020 · Cited by 824
FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
Ciftci, U. A., Demir, I., & Yin, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3009287 · Cited by 499
DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern
Jung, T., Kim, S., & Kim, K. (2020) · IEEE Access · doi:10.1109/access.2020.2988660 · Cited by 326 · cited 7×
“Section 2.1 also recaps the main functionality of DeepVision by Jung et al. [24] where they describe a hand-crafted detection method of the eye blinking behavior of persons in potential DeepFake videos.”
BackgroundMethodInfluentialA Survey on Deepfake Video Detection
Yu, P., Xia, Z., Fei, J., & Lu, Y. (2021) · IET Biometrics · doi:10.1049/bme2.12031 · Cited by 253
Deep learning for deepfakes creation and detection: a survey
Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., & Nguyen, C. M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4030341 · Cited by 112
The Deepfake Challenges and Deepfake Video Detection
Lecturer at Wolkite University, W. E., & Wubet*, W. M. (2020) · International Journal of Innovative Technology and Exploring Engineering · doi:10.35940/ijitee.e2779.049620 · Cited by 27
Research works printed but not linked: 33
- Chesney, 2019, Deepfakes and the new disinformation war: The coming age of post-truth geopolitics, Foreign Aff., 98, 147
- Palmer, G.L. (2001). A Road Map for Digital Forensics Research—Report from the First Digital Forensics Research Workshop (DFRWS) (Technical Report DTR-T001-01 Final), Air Force Research Laboratory, Rome Research Site. Technical Report.
- Champod, C., and Vuille, J. (2011). Scientific Evidence in Europe—Admissibility, Evaluation and Equality of Arms. Int. Comment. Evid., 9. · DOI
- Krätzer, C. (2013). Statistical Pattern Recognition for Audio-forensics—Empirical Investigations on the Application Scenarios Audio Steganalysis and Microphone Forensics. [Ph.D. Thesis, Otto-von-Guericke-University].
- Kiltz, S. (2020). Data-Centric Examination Approach (DCEA) for a Qualitative Determination of Error, Loss and Uncertainty in Digital and Digitised Forensics. [Ph.D. Thesis, Otto-von-Guericke-University].
- Geradts, Z.J.M.H., Franke, K.Y., and Veenman, C.J. (2009). Multimedia forensics is not computer forensics. Computational Forensics, Springer. · doi:10.1007/978-3-642-03521-0
- Bondi, L., Cannas, E.D., Bestagini, P., and Tubaro, S. (2020). Training Strategies and Data Augmentations in CNN-based DeepFake Video Detection. arXiv. · DOI · Paper page
- Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., and Natarajan, P. (2019). Recurrent Convolutional Strategies for Face Manipulation Detection in Videos. arXiv.
- Korshunov, P., and Marcel, S. (2018). DeepFakes: A New Threat to Face Recognition? Assessment and Detection. arXiv.
- Li, Y., and Lyu, S. (2018). Exposing DeepFake Videos By Detecting Face Warping Artifacts. arXiv.
- McCloskey, S., and Albright, M. (2018). Detecting GAN-generated Imagery using Color Cues. arXiv. · DOI · Paper page
- Agarwal, S., Farid, H., Gu, Y., He, M., Nagano, K., and Li, H. (2019, January 16–20). Protecting World Leaders Against Deep Fakes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.
- Lin, 2020, Comparison of handcrafted features and convolutional neural networks for liver MR image adequacy assessment, Sci. Rep., 10, 20336, 10.1038/s41598-020-77264-y · DOI
- Yin, H., Tino, P., Corchado, E., Byrne, W., and Yao, X. (2007). Filter Methods for Feature Selection—A Comparative Study. Intelligent Data Engineering and Automated Learning—IDEAL 2007, Springer. · DOI
- Law, 2004, Simultaneous feature selection and clustering using mixture models, IEEE Trans. Pattern Anal. Mach. Intell., 26, 1154, 10.1109/TPAMI.2004.71 · DOI · Paper page
- Kiltz, S., Dittmann, J., and Vielhauer, C. (2015, January 18–20). Supporting Forensic Design—A Course Profile to Teach Forensics. Proceedings of the 2015 Ninth International Conference on IT Security Incident Management and IT Forensics, Magdeburg, Germany. · doi:10.1109/imf.2015.16
- Altschaffel, R. (2020). Computer Forensics in Cyber-Physical Systems: Applying Existing Forensic Knowledge and Procedures from Classical IT to Automation and Automotive. [Ph.D. Thesis, Otto-von-Guericke-University].
- Kiltz, S., Hoppe, T., and Dittmann, J. (2009, January 5–7). A New Forensic Model and Its Application to the Collection, Extraction and Long Term Storage of Screen Content off a Memory Dump. Proceedings of the 16th International Conference on Digital Signal Processing, DSP’09, Santorini, Greece. · doi:10.1109/icdsp.2009.5201189
- Sagonas, 2016, 300 Faces In-The-Wild Challenge, Image Vis. Comput., 47, 3, 10.1016/j.imavis.2016.01.002 · DOI · Paper page
- King, 2009, Dlib-ml: A Machine Learning Toolkit, J. Mach. Learn. Res., 10, 1755
- Makrushin, A., Neubert, T., and Dittmann, J. (March, January 27). Automatic generation and detection of visually faultless facial morphs. Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications—Volume 6: VISAPP, (VISIGRAPP 2017), INSTICC, Porto, Portugal. · DOI · Paper page
- Kraetzer, C., Makrushin, A., Neubert, T., Hildebrandt, M., and Dittmann, J. (2017, January 20–22). Modeling Attacks on Photo-ID Documents and Applying Media Forensics for the Detection of Facial Morphing. Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security, IH&MMSec ’17, Philadelphia, PA, USA. · DOI
- Hall, 2009, The WEKA data mining software: An update, SIGKDD Explor., 11, 10, 10.1145/1656274.1656278 · DOI · Paper page
- Quinlan, J.R. (1993). C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers Inc.
- Sanderson, 2009, Multi-Region Probabilistic Histograms for Robust and Scalable Identity Inference, LNCS, 5558, 199
- Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Nießner, M. (2018). FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. arXiv.
- Agarwal, S., Farid, H., Fried, O., and Agrawala, M. (2020, January 14–19). Detecting Deep-Fake Videos From Phoneme-Viseme Mismatches. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Seattle, WA, USA. · DOI · Paper page
- Bradski, 2000, The OpenCV Library, Dobb J. Softw. Tools, 120, 122
- Ross, A.A., Nandakumar, K., and Jain, A.K. (2006). Levels of Fusion in Biometrics. Handbook of Multibiometrics, Springer.
- Kuncheva, L.I. (2004). Combining Pattern Classifiers: Methods and Algorithms, Wiley. · DOI · Paper page
- Zhang, W., Zhao, C., and Li, Y. (2020). A Novel Counterfeit Feature Extraction Technique for Exposing Face-Swap Images Based on Deep Learning and Error Level Analysis. Entropy, 22. · DOI · Paper page
- Zhang, W., and Zhao, C. (2020). Exposing Face-Swap Images Based on Deep Learning and ELA Detection. Proceedings, 46. · DOI
- Krawetz, N. (August, January 28). A Picture ’ s Worth…Digital Image Analysis and Forensics. Proceedings of the Black Hat Briefings 2007, Las Vegas, NV, USA.
Cited by
Deepfake detection using deep learning methods: A systematic and comprehensive review
Heidari, A., Jafari Navimipour, N., Dag, H., & Unal, M. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1520 · Cited by 349
“It could lead to a troop being misled when crossing a bridge during combat (Baygin et al., 2022; Siegel et al., 2021).”
BackgroundEvaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors
Khalid, H., Kim, M., Tariq, S., & Woo, S. S. (2021) · Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection · doi:10.1145/3476099.3484315 · Cited by 110
Qureshi, S. M., Saeed, A., Almotiri, S. H., Ahmad, F., & Al Ghamdi, M. A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2037 · Cited by 71
Metadata
- Authors
- Siegel, D., Kraetzer, C., Seidlitz, S., & Dittmann, J.
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2021-07-01
- Volume
- 7
- Issue
- 7
- Pages
- 108
- ISSN
- 2313-433X (ISSN-L)
- OpenAlex
- W3175824036
- Cited by
- 50
- References
- 15
- FWCI
- 3.09
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Bundesministerium für Bildung und Forschung (13N15736)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.