CoChat
Sign Up
PreprintCited by 1Digital Media Forensic DetectionGenerative Adversarial Networks and Image SynthesisMisinformation and Its Impacts

The Tug-of-War Between Deepfake Generation and Detection

Open access
Lee, Hannah, Lee, Changyeon, Farhat, Kevin, Qiu, Lin, Geluso, Steve, Kim, Aerin, Etzioni, Oren·2024·arXiv
DOI 10.48550/arxiv.2407.06174arXiv 2407.06174
Open this paper in CoChat — free accountOpen PDF

Abstract

Multimodal generative models are rapidly evolving, leading to a surge in the generation of realistic video and audio that offers exciting possibilities but also serious risks. Deepfake videos, which can convincingly impersonate individuals, have particularly garnered attention due to their potential misuse in spreading misinformation and creating fraudulent content. This survey paper examines the dual landscape of deepfake video generation and detection, emphasizing the need for effective countermeasures against potential abuses. We provide a comprehensive overview of current deepfake generation techniques, including face swapping, reenactment, and audio-driven animation, which leverage cutting-edge technologies like GANs and diffusion models to produce highly realistic fake videos. Additionally, we analyze various detection approaches designed to differentiate authentic from altered videos, from detecting visual artifacts to deploying advanced algorithms that pinpoint inconsistencies across video and audio signals. The effectiveness of these detection methods heavily relies on the diversity and quality of datasets used for training and evaluation. We discuss the evolution of deepfake datasets, highlighting the importance of robust, diverse, and frequently updated collections to enhance the detection accuracy and generalizability. As deepfakes become increasingly indistinguishable from authentic content, developing advanced detection techniques that can keep pace with generation technologies is crucial. We advocate for a proactive approach in the "tug-of-war" between deepfake creators and detectors, emphasizing the need for continuous research collaboration, standardization of evaluation metrics, and the creation of comprehensive benchmarks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.06174

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Methodologies developed for visual media cannot be directly applied to detect DF audio.

    “Methodologies developed for visual media cannot be directly applied to detect DF audio [129].”

    From Deepfake detection across image, video, and audio: a comprehensive survey with empirical evaluation of generalization and robustness (2026) · cites this paper as [129]

    Section 3.2, “Fake Audio Detection” (page unavailable), describes extracting audio-specific spectrogram features before using a CNN, supporting the need to adapt an image-based approach for audio. It does not state the broader claim that methodologies developed for visual media cannot be directly applied to deepfake audio. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 126

  1. Abdal, R., Qin, Y., and Wonka, P. Image2stylegan: How to embed images into the stylegan latent space? In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432-4441, 2019.
  2. Abdal, R., Qin, Y., and Wonka, P. Image2stylegan++: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296-8305, 2020.
  3. Agarwal, S., Farid, H., Fried, O., and Agrawala, M. Detect- ing deep-fake videos from phoneme-viseme mismatches. pp. 660-661, 2020.
  4. Almutairi, Z. and Elgibreen, H. A review of modern au- dio deepfake detection methods: Challenges and future directions. Algorithms, 15(5):155, 2022.
  5. Amerini, I., Galteri, L., Caldelli, R., and Del Bimbo, A. Deepfake video detection through optical flow based cnn. pp. 1205-1207, 2019. doi: 10.1109/ICCVW.2019.00152. · DOI · Paper page
  6. Ballesteros, D. M., Rodriguez, Y., and Renza, D. A dataset of histograms of original and fake voice recordings (h-voice). Data in Brief, 29:105331, 2020. ISSN 2352- 3409. doi: https://doi.org/10.1016/j.dib.2020.105331. URL https://www.sciencedirect.com/ science/article/pii/S2352340920302250. · DOI
  7. Bird, J. J. and Lotfi, A. Cifake: Image classification and ex- plainable identification of ai-generated synthetic images. IEEE Access, 2024.
  8. Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018. · arXiv
  9. Brooks, T., Holynski, A., and Efros, A. A. Instruct- pix2pix: Learning to follow image editing instruc- tions. 2023 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 18392-18402, 2022. URL https://api.semanticscholar. org/CorpusID:253581213.
  10. Burkov, E., Pasechnik, I., Grigorev, A., and Lempitsky, V. Neural head reenactment with latent pose descriptors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13786-13795, 2020.
  11. Carlini, N. and Farid, H. Evading deepfake-image detectors with white-and black-box attacks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 658-659, 2020.
  12. Chaitra, B. and Reddy, P. B. Digital image forgery: taxonomy, techniques, and tools-a com- prehensive study. International Journal of System Assurance Engineering and Management, 14:18-33, 2022. URL https://api.semanticscholar. org/CorpusID:255039112.
  13. Chen, L., Maddox, R. K., Duan, Z., and Xu, C. Hierarchical cross-modal talking face generation with dynamic pixel- wise loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7832- 7841, 2019.
  14. Chen, L., Cui, G., Kou, Z., Zheng, H., and Xu, C. What comprises a good talking-head video generation?: A sur- vey and benchmark. arXiv preprint arXiv:2005.03201, 2020. · arXiv
  15. Chen, Y., Haldar, N. A. H., Akhtar, N., and Mian, A. Text-image guided diffusion model for gener- ating deepfake celebrity interactions. 2023 Inter- national Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 348-355, 2023. URL https://api.semanticscholar. org/CorpusID:262825503.
  16. Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
  17. Chugh, K., Gupta, P., Dhall, A., and Subramanian, R. Not made for each other-audio-visual dissonance-based deep- fake detection and localization. pp. 439-447, 2020.
  18. Cohn, M. and Zellou, G. Perception of Concatenative vs. Neural Text-To-Speech (TTS): Differences in Intelligibil- ity in Noise and Language Attitudes. In Proc. Interspeech 2020, pp. 1733-1737, 2020. doi: 10.21437/Interspeech. 2020-1336. · DOI
  19. Corvi, R., Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., and Verdoliva, L. On the detection of synthetic images generated by diffusion models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
  20. Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., and Pfister, H. Video face replacement. Proceedings of the 2011 SIGGRAPH Asia Conference, 2011. URL https://api.semanticscholar. org/CorpusID:8692593.
  21. Fan, B., Wang, L., Soong, F. K., and Xie, L. Photo-real talking head with deep bidirectional lstm. In 2015 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 4884-4888. IEEE, 2015.
  22. Fan, Y., Lin, Z., Saito, J., Wang, W., and Komura, T. Face- former: Speech-driven 3d facial animation with trans- formers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18770- 18780, 2022.
  23. Garrido, P., Valgaerts, L., Rehmsen, O., Thormählen, T., Pérez, P., and Theobalt, C. Automatic face reenactment. CoRR, abs/1602.02651, 2016. URL http://arxiv. org/abs/1602.02651. · arXiv
  24. Gimeno, P., Viñals, I., Ortega, A., Miguel, A., and Lleida, E. Multiclass audio segmentation based on recurrent neural networks for broadcast domain data. EURASIP Journal on Audio, Speech, and Music Processing, 2020 (1), March 2020. ISSN 1687-4722. doi: 10.1186/ s13636-020-00172-6. URL http://dx.doi.org/ 10.1186/s13636-020-00172-6. · DOI
  25. Guera, David; Delp, E. J. Deepfake video detection using recurrent neural networks. IEEE 2018 15th IEEE Interna- tional Conference on Advanced Video and Signal Based Surveillance (AVSS), 2018.
  26. Guo, H., Hu, S., Wang, X., Chang, M.-C., and Lyu, S. Eyes tell all: Irregular pupil shapes reveal gan-generated faces. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2904-2908. IEEE, 2022.
  27. Guo, Y., Zhang, L., Hu, Y., He, X., and Gao, J. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp. 87-102. Springer, 2016.
  28. Guo, Y., Chen, K., Liang, S., Liu, Y., Bao, H., and Zhang, J. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  29. Hamza, A., Javed, A. R. R., Iqbal, F., Kryvinska, N., Al- madhor, A. S., Jalil, Z., and Borghol, R. Deepfake audio detection via mfcc features using machine learning. IEEE Access, 10:134018-134028, 2022. ISSN 2169-3536. doi: 10.1109/access.2022.3231480. URL http://dx.doi. org/10.1109/ACCESS.2022.3231480. · DOI · Paper page
  30. Hande, S. S. A review of concatenative text to speech synthesis. International Journal of Latest Trends in Engineering and Technology (IJLTEMAS), 3(9):12- 15, 2014. URL https://www.ijltemas.in/ DigitalLibrary/Vol.3Issue9/12-15.pdf.
  31. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
  32. Ho, J., Jain, A., and Abbeel, P. Denoising diffu- sion probabilistic models. ArXiv, abs/2006.11239, 2020. URL https://api.semanticscholar. org/CorpusID:219955663. · arXiv
  33. Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022. · arXiv
  34. Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205.15868, 2022. · arXiv
  35. Hou, Y., Guo, Q., Huang, Y., Xie, X., Ma, L., and Zhao, J. Evading deepfake detectors via adversarial statistical consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12271- 12280, 2023.
  36. Hu, C., Tian, Q., Li, T., Yuping, W., Wang, Y., and Zhao, H. Neural dubber: Dubbing for videos ac- cording to scripts. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 16582-16595.
  37. Curran Associates, Inc., 2021. URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ 8a9c8ac001d3ef9e4ce39b1177295e03-Paper. pdf.
  38. Jamaludin, A., Chung, J. S., and Zisserman, A. You said that?: Synthesising talking faces from audio. Interna- tional Journal of Computer Vision, 127(11):1767-1779, 2019.
  39. Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., and Xu, F. Audio-driven emotional video portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14080-14089, 2021.
  40. Kain, A. and Macon, M. W. Spectral voice conversion for text-to-speech synthesis. Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181), 1:285-288 vol.1, 1998. URL https://api. semanticscholar.org/CorpusID:1332456.
  41. Karras, T., Aila, T., Laine, S., Herva, A., and Lehtinen, J. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):1-12, 2017a.
  42. Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres- sive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017b. · arXiv
  43. Karras, T., Laine, S., and Aila, T. A style-based genera- tor architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
  44. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
  45. Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.
  46. Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H.- T., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion mod- els. 2023 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 6007-6017, 2022. URL https://api.semanticscholar. org/CorpusID:252918469.
  47. Korshunov, P. and Marcel, S. Deepfakes: A new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685, 2018. · arXiv
  48. Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., and Guo, B. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5001-5010, 2020a.
  49. Li, Y. and Lyu, S. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv:1811.00656, 2018. · arXiv
  50. Li, Y., Chang, M.-C., and Lyu, S. In ictu oculi: Exposing ai created fake videos by detecting eye blinking. pp. 1-7, 2018.
  51. Liang, B., Pan, Y., Guo, Z., Zhou, H., Hong, Z., Han, X., Han, J., Liu, J., Ding, E., and Wang, J. Expressive talking head generation with granular audio-visual control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3387-3396, 2022.
  52. Lim, Y., Lee, C., Kim, A., and Etzioni, O. Distildire: A small, fast, cheap and lightweight diffusion synthesized deepfake detection, 2024. URL https://arxiv. org/abs/2406.00856.
  53. Lin, Y., Wang, S., Lin, Q., and Tang, F. Face swap- ping under large pose variations: A 3d model based ap- proach. 2012 IEEE International Conference on Multime- dia and Expo, pp. 333-338, 2012. URL https://api. semanticscholar.org/CorpusID:30759641.
  54. Lin, Y., Shen, J., Wang, Y., and Pantic, M. Fp-age: Leverag- ing face parsing attention for facial age estimation in the wild. arXiv, 2021.
  55. Liu, S., Cao, Y., and Meng, H. M. Emotional voice conversion with cycle-consistent adversarial network. arXiv: Audio and Speech Processing, 2020a. URL https://api.semanticscholar. org/CorpusID:215416116.
  56. Liu, X., Xu, Y., Wu, Q., Zhou, H., Wu, W., and Zhou, B. Semantic-aware implicit neural audio-driven video portrait generation. ECCV, 2022.
  57. Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  58. Liu, Z., Qi, X., and Torr, P. H. Global texture enhancement for fake face detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8060-8069, 2020b.
  59. Liu, Z., Li, M., Zhang, Y., Wang, C., Zhang, Q., Wang, J., and Nie, Y. Fine-grained face swapping via regional gan inversion, 2023b. Lorenz, P., Durall, R. L., and Keuper, J. Detecting images generated by deep diffusion models using their local in- trinsic dimensionality. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 448- 459, 2023.
  60. Lu, Y. and Ebrahimi, T. Towards the detection of ai-synthesized human face images. arXiv preprint arXiv:2402.08750, 2024. · arXiv
  61. Lu, Y., Chai, J., and Cao, X. Live speech portraits: real-time photorealistic talking-head animation. ACM Transactions on Graphics (TOG), 40(6):1-17, 2021.
  62. Lv, L. Smart watermark to defend against deepfake image manipulation. In 2021 IEEE 6th international conference on computer and communication systems (ICCCS), pp. 380-384. IEEE, 2021.
  63. Marra, F., Gragnaniello, D., Verdoliva, L., and Poggi, G. Do gans leave artificial fingerprints? In 2019 IEEE confer- ence on multimedia information processing and retrieval (MIPR), pp. 506-511. IEEE, 2019.
  64. Matern, F., Riess, C., and Stamminger, M. Exploiting vi- sual artifacts to expose deepfakes and face manipulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pp. 83-92. IEEE, 2019.
  65. Mirsky, Y. and Lee, W. The creation and detection of deep- fakes: A survey. ACM Computing Surveys, 54(1):1-41, January 2021. ISSN 1557-7341. doi: 10.1145/3425780. URL http://dx.doi.org/10.1145/3425780. · DOI · Paper page
  66. Neekhara, P., Dolhansky, B., Bitton, J., and Ferrer, C. C. Adversarial threats to deepfake detection: A practical perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 923-932, 2021.
  67. Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., and Nguyen, C. M. Deep learning for deepfakes creation and detection: A survey. Computer Vision and Image Understanding, 223:103525, Octo- ber 2022. ISSN 1077-3142. doi: 10.1016/j.cviu.2022. 103525. URL http://dx.doi.org/10.1016/j. cviu.2022.103525. · DOI · Paper page
  68. Nirkin, Y., Keller, Y., and Hassner, T. FSGAN: sub- ject agnostic face swapping and reenactment. CoRR, abs/1908.05932, 2019. URL http://arxiv.org/ abs/1908.05932. · arXiv
  69. Ojha, U., Li, Y., and Lee, Y. J. Towards universal fake image detectors that generalize across generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24480-24489, 2023.
  70. Park, S. J., Kim, M., Hong, J., Choi, J., and Ro, Y. M. Synctalkface: Talking face generation with precise lip- syncing via audio-lip memory. In AAAI Conference on Artificial Intelligence. Association for the Advancement of Artificial Intelligence, 2022.
  71. Perov, I., Gao, D., Chervoniy, N., Liu, K., Marangonda, S., Umé, C., Dpfks, M., Facenheim, C. S., RP, L., Jiang, J., Zhang, S., Wu, P., Zhou, B., and Zhang, W. Deep- facelab: A simple, flexible and extensible face swap- ping framework. CoRR, abs/2005.05535, 2020. URL https://arxiv.org/abs/2005.05535. · arXiv
  72. Prajwal, K., Mukhopadhyay, R., Namboodiri, V. P., and Jawahar, C. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 484- 492, 2020.
  73. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
  74. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero- shot text-to-image generation. ArXiv, abs/2102.12092, 2021. URL https://api.semanticscholar. org/CorpusID:232035663. · arXiv
  75. Rana, M. S., Nobi, M. N., Murali, B., and Sung, A. H. Deepfake detection: A systematic literature review. IEEE Access, 10:25494-25513, 2022. doi: 10.1109/ACCESS. 2022.3154404. · DOI · Paper page
  76. Rashid, M. M., Lee, S.-H., and Kwon, K.-R. Blockchain technology for combating deepfake and protect video/image integrity. Journal of Korea Multimedia Soci- ety, 24(8):1044-1058, 2021.
  77. Reimao, R. and Tzerpos, V. For: A dataset for syn- thetic speech detection. In 2019 International Confer- ence on Speech Technology and Human-Computer Dia- logue (SpeD), pp. 1-10, 2019. doi: 10.1109/SPED.2019. 8906599. · DOI · Paper page
  78. Reisach, U. The responsibility of social media in times of societal and political manipulation. European journal of operational research, 291(3):906-917, 2021.
  79. Richard, A., Zollhöfer, M., Wen, Y., de la Torre, F., and Sheikh, Y. Meshtalk: 3d face animation from speech using cross-modality disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  80. Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., and Cohen-Or, D. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287-2296, 2021.
  81. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL https: //api.semanticscholar.org/CorpusID: 245335280.
  82. Rössler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Nießner, M. FaceForensics++: Learning to detect manipulated facial images. In International Conference on Computer Vision (ICCV), 2019.
  83. Rothe, R., Timofte, R., and Gool, L. V. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2-4):144-157, 2018.
  84. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text- to-image diffusion models for subject-driven genera- tion. 2023 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 22500-22510, 2022. URL https://api.semanticscholar. org/CorpusID:251800180.
  85. Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., and Natarajan, P. Recurrent convolutional strategies for face manipulation detection in videos. Interfaces (GUI), 3(1):80-87, 2019.
  86. Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Pho- torealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.11487, 2022. URL https://api.semanticscholar. org/CorpusID:248986576. · arXiv
  87. Scarpiniti, M., Comminiello, D., Uncini, A., and Lee, Y.-C. Deep recurrent neural networks for audio classification in construction sites. In 2020 28th European Signal Pro- cessing Conference (EUSIPCO), pp. 810-814, 2021. doi: 10.23919/Eusipco47968.2020.9287802. · DOI
  88. Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry- Ryan, R. J., Saurous, R. A., Agiomyrgiannakis, Y., and Wu, Y. Natural tts synthesis by condition- ing wavenet on mel spectrogram predictions. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783, 2017. URL https://api.semanticscholar. org/CorpusID:206742911.
  89. Shen, S., Li, W., Zhu, Z., Duan, Y., Zhou, J., and Lu, J. Learning dynamic facial radiance fields for few-shot talk- ing head synthesis. In ECCV, 2022.
  90. Stypułkowski, M., Vougioukas, K., He, S., Zieba, M., Petridis, S., and Pantic, M. Diffused heads: Diffusion models beat gans on talking-face generation. pp. 5091- 5100, 2024.
  91. Sun, Y., Zhang, Z., Echizen, I., Nguyen, H. H., Qiu, C., and Sun, L. Face forgery detection based on facial re- gion displacement trajectory series. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 633-642, 2023.
  92. Suwajanakorn, S., Seitz, S. M., and Kemelmacher- Shlizerman, I. Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (ToG), 36 (4):1-13, 2017.
  93. Swathi, P. and Sk, S. Deepfake creation and detection: A survey. In 2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA), pp. 584-588. IEEE, 2021.
  94. Tariang, D., Corvi, R., Cozzolino, D., Poggi, G., Nagano, K., and Verdoliva, L. Synthetic image verification in the era of generative ai: What works and what isn't there yet. arXiv preprint arXiv:2405.00196, 2024. · arXiv
  95. Thies, J., Zollhöfer, M., Nießner, M., Valgaerts, L., Stam- minger, M., and Theobalt, C. Real-time expression trans- fer for facial reenactment. ACM Transactions on Graph- ics (TOG), 34(6), 2015.
  96. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., and Nießner, M. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2387-2395, 2016.
  97. Thies, J., Zollhöfer, M., and Nießner, M. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
  98. Thies, J., Elgharib, M., Tewari, A., Theobalt, C., and Nießner, M. Neural voice puppetry: Audio-driven fa- cial reenactment. In European Conference on Computer Vision, pp. 716-731. Springer, 2020.
  99. Tokuda, K., Yoshimura, T., Masuko, T., Kobayashi, T., and Kitamura, T. Speech parameter generation algorithms for hmm-based speech synthesis. 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100), 3:1315-1318 vol.3, 2000. URL https://api.semanticscholar. org/CorpusID:9372431.
  100. Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., and Cohen-Or, D. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 2021.
  101. Trinh Van, L., Dao, T., Le, T., and Castelli, E. Emotional speech recognition using deep neural networks. Sensors, 22:1414, 02 2022. doi: 10.3390/s22041414. · DOI
  102. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Se- nior, A. W., and Kavukcuoglu, K. Wavenet: A gen- erative model for raw audio. ArXiv, abs/1609.03499, 2016. URL https://api.semanticscholar. org/CorpusID:6254678. · arXiv
  103. Vlasic, D., Brand, M., Pfister, H., and Popović, J. Face transfer with multilinear models. ACM Trans. Graph., 24(3):426-433, jul 2005. ISSN 0730-0301. doi: 10. 1145/1073204.1073209. URL https://doi.org/ 10.1145/1073204.1073209. · DOI
  104. Wang, K., Wu, Q., Song, L., Yang, Z., Wu, W., Qian, C., He, R., Qiao, Y., and Loy, C. C. Mead: A large-scale audio- visual dataset for emotional talking-face generation. In ECCV, August 2020a.
  105. Wang, R., Juefei-Xu, F., Ma, L., Xie, X., Huang, Y., Wang, J., and Liu, Y. Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces. arXiv preprint arXiv:1909.06122, 2019. · arXiv
  106. Wang, S.-Y., Wang, O., Zhang, R., Owens, A., and Efros, A. A. Cnn-generated images are surprisingly easy to spot...for now. In CVPR, 2020b.
  107. Wang, X., Li, Y., Zhang, H., and Shan, Y. Towards real- world blind face restoration with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  108. Wang, Z., Bao, J., Zhou, W., Wang, W., Hu, H., Chen, H., and Li, H. Dire for diffusion-generated image detection. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 22445-22455, 2023.
  109. Wang, Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H. Diffusiondb: A large- scale prompt gallery dataset for text-to-image gener- ative models. arXiv:2210.14896 [cs], 2022. URL https://arxiv.org/abs/2210.14896. · arXiv
  110. Wu, H., Kuo, H.-C., Zheng, N., Hung, K.-H., yi Lee, H., Tsao, Y., Wang, H.-M., and Meng, H. M. Partially fake audio detection by self-attention- based fake span discovery. ICASSP 2022 -2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9236-9240, 2022. URL https://api.semanticscholar. org/CorpusID:246823684.
  111. Xu, C., Zhang, J., Hua, M., He, Q., Yi, Z., and Liu, Y. Region-aware face swapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7632-7641, 2022a.
  112. Xu, Z., Zhou, H., Hong, Z., Liu, Z., Liu, J., Guo, Z., Han, J., Liu, J., Ding, E., and Wang, J. Styleswap: Style-based generator empowers robust face swapping. In European Conference on Computer Vision, pp. 661-677. Springer, 2022c.
  113. Yan, Z., Zhang, Y., Yuan, X., Lyu, S., and Wu, B. Deep- fakebench: A comprehensive benchmark of deepfake de- tection. In Thirty-seventh Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum? id=hizSx8pf0U.
  114. Yang, T., Ren, P., Xie, X., and Zhang, L. Gan prior em- bedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 672-681, 2021.
  115. Yang, X., Li, Y., and Lyu, S. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8261-8265. IEEE, 2019.
  116. Yang, Y., Shillingford, B., Assael, Y., Wang, M., Liu, W., Chen, Y., Zhang, Y., Sezener, E., Cobo, L. C., Denil, M., et al. Large-scale multilingual audio visual dubbing. arXiv preprint arXiv:2011.03530, 2020. · arXiv
  117. Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. · arXiv
  118. Yu, P., Xia, Z., Fei, J., and Lu, Y. A survey on deepfake video detection. IET Biometrics, 10(6):607-624, April 2021. ISSN 2047-4946. doi: 10.1049/bme2.12031. URL http://dx.doi.org/10.1049/bme2.12031. · DOI · Paper page
  119. Zaman, K., Sah, M., Direkoglu, C., and Unoki, M. A survey of audio classification using deep learning. IEEE Access, 11:106620-106649, 2023. doi: 10.1109/ACCESS.2023. 3318015. · DOI
  120. Zhang, X., Karaman, S., and Chang, S.-F. Detecting and simulating artifacts in gan fake images. In 2019 IEEE international workshop on information forensics and se- curity (WIFS), pp. 1-6. IEEE, 2019.
  121. Zhang, Z., Hu, Z., Deng, W., Fan, C., Lv, T., and Ding, Y. Dinet: Deformation inpainting network for realistic face visually dubbing on high resolution video. AAAI, 2023.
  122. Zhou, X. and Zafarani, R. A survey of fake news: Fun- damental theories, detection methods, and opportuni- ties, September 2020. ISSN 1557-7341. URL http: //dx.doi.org/10.1145/3395046. · DOI · Paper page
  123. Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., and Singh, K. Visemenet: Audio-driven animator-centric speech animation. ACM Transactions on Graphics (TOG), 37(4):1-10, 2018.
  124. Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kaloger- akis, E., and Li, D. Makelttalk: speaker-aware talking- head animation. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
  125. Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adver- sarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
  126. Zi, B., Chang, M., Chen, J., Ma, X., and Jiang, Y.-G. Wild- deepfake: A challenging real-world dataset for deepfake detection. In Proceedings of the 28th ACM international conference on multimedia, pp. 2382-2390, 2020.

Cited by

Sorted by citations

Authors
Lee, H., Lee, C., Farhat, K., Qiu, L., Geluso, S., Kim, A., & Etzioni, O.
Affiliations
University of Washington (US)
ORCID
Lee, Changyeon
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.06174
arXiv
2407.06174
OpenAlex
W4400488483
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
deepfakes, deepfake detection, deepfake generation, face swapping, generative adversarial networks, diffusion models, multimodal generative models, misinformation, deepfake datasets, evaluation benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review