CoChat
Sign Up
Cited by 22Misinformation and Its ImpactsDigital Media Forensic DetectionSpam and Phishing Detection

AI-Generated Faces in the Real World: A Large-Scale Case Study of Twitter Profile Images

Open access
Ricker, Jonas, Assenmacher, Dennis, Holz, Thorsten, Fischer, Asja, Quiring, Erwin·2024·The 27th International Symposium on Research in Attacks, Intrusions and Defenses
DOI 10.1145/3678890.3678922
Open this paper in CoChat — free account

Abstract

Recent advances in the field of generative artificial intelligence (AI) have blurred the lines between authentic and machine-generated content, making it almost impossible for humans to distinguish between such media. One notable consequence is the use of AI-generated images for fake profiles on social media. While several types of disinformation campaigns and similar incidents have been reported in the past, a systematic analysis has been lacking.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3678890.3678922

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 82

  1. Michael Albright and Scott McCloskey. 2019. Source Generator Attribution via Inversion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  2. Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. 2022. Dos and Don'ts of Machine Learning in Computer Security. In USENIX Security Symposium.
  3. Adam Badawy, Emilio Ferrara, and Kristina Lerman. 2018. Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign. In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).
  4. Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019. BlazeFace: Sub-millisecond Neural Face Detec- tion on Mobile GPUs. arXiv preprint (2019).
  5. Fabrício Benevenuto, Gabriel Magno, Tiago Rodrigues, and Virgílio Almeida. 2010. Detecting Spammers on Twitter. In Collaboration, Electronic messaging, Anti-Abuse and Spam Conference (CEAS).
  6. Giulia Boato, Cecilia Pasquini, Antonio L. Stefani, Sebastiano Verde, and Daniele Miorandi. 2022. TrueFace: A Dataset for the Detection of Synthetic Face Images from Social Networks. In IEEE International Joint Conference on Biometrics (IJCB).
  7. Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. 2020. What Makes Fake Images Detectable? Understanding Properties That Generalize. In European Conference on Computer Vision (ECCV).
  8. Yuanqi Chen, Ge Li, Cece Jin, Shan Liu, and Thomas Li. 2021. SSD-GAN: Measuring the Realness in the Spatial and Spectral Domains. In AAAI Conference on Artificial Intelligence (AAAI).
  9. Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. StarGAN: Unified Generative Adversarial Networks for Multi- Domain Image-to-Image Translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  10. Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva. 2023. On the Detection of Synthetic Images Generated by Diffusion Models. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  11. Davide Cozzolino, Diego Gragnaniello, Giovanni Poggi, and Luisa Verdoliva. 2021. Towards Universal GAN Image Detection. In International Conference on Visual Communications and Image Processing (VCIP).
  12. Davide Cozzolino, Justus Thies, Andreas Rössler, Matthias Niessner, and Luisa Verdoliva. 2021. SpoC: Spoofing Camera Fingerprints. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  13. Stefano Cresci. 2020. A Decade of Social Bot Detection. Commun. ACM (2020).
  14. Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems (NeurIPS).
  15. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR).
  16. Ricard Durall, Margret Keuper, and Janis Keuper. 2020. Watch Your Up-Convo- lution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  17. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A Density- Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In International Conference on Knowledge Discovery and Data Mining (KDD).
  18. Pavlos Fafalios, Vasileios Iosifidis, Eirini Ntoutsi, and Stefan Dietze. 2018. Tweet- sKB: A Public and Large-Scale RDF Corpus of Annotated Tweets. In European Semantic Web Conference (ESWC).
  19. Hany Farid. 2021. An Overview of Perceptual Hashing. Journal of Online Trust and Safety (2021).
  20. Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, and Teddy Furon. 2023. The Stable Signature: Rooting Watermarks in Latent Diffusion Models. In IEEE/CVF International Conference on Computer Vision (ICCV).
  21. Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. 2020. Leveraging Frequency Analysis for Deep Fake Image Recognition. In International Conference on Machine Learning (ICML).
  22. Joel Frank, Franziska Herbert, Jonas Ricker, Lea Schönherr, Thorsten Eisenhofer, Asja Fischer, Markus Dürmuth, and Thorsten Holz. 2024. A Representative Study on Human Detection of Artificially Generated Media Across Countries. In IEEE Symposium on Security and Privacy (S&P).
  23. Yang Gao, Rita Singh, and Bhiksha Raj. 2018. Voice Impersonation Using Generative Adversarial Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  24. Kiran Garimella and Dean Eckles. 2020. Images and Misinformation in Political Groups: Evidence from WhatsApp in India. Harvard Kennedy School (HKS) Misinformation Review (2020).
  25. Josh A. Goldstein and Renée DiResta. 2022. Research Note: This Salesperson Does Not Exist: How Tactics from Political Influence Operations on Social Media Are Deployed for Commercial Lead Generation. Harvard Kennedy School (HKS) Misinformation Review (2022).
  26. Josh A. Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative Language Models and Automated Influ- ence Operations: Emerging Threats and Potential Mitigations. arXiv preprint (2023).
  27. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Ad- versarial Nets. In Advances in Neural Information Processing Systems (NeurIPS).
  28. D. Gragnaniello, D. Cozzolino, F. Marra, G. Poggi, and L. Verdoliva. 2021. Are GAN Generated Images Easy to Detect? A Critical Analysis of the State-of-the- Art. In IEEE International Conference on Multimedia and Expo (ICME).
  29. Hui Guo, Shu Hu, Xin Wang, Ming-Ching Chang, and Siwei Lyu. 2022. Eyes Tell All: Irregular Pupil Shapes Reveal GAN-generated Faces. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  30. Supraja Gurajala, Joshua S. White, Brian Hudson, and Jeanna N. Matthews. 2015. Fake Twitter Accounts: Profile Characteristics Obtained Using an Activity- Based Pattern Detection Approach. In International Conference on Social Media & Society.
  31. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  32. Daniel Hickey, Matheus Schmitz, Daniel Fessler, Paul E. Smaldino, Goran Muric, and Keith Burghardt. 2023. Auditing Elon Musk's Impact on Hate Speech and Bots. In International AAAI Conference on Web and Social Media (ICWSM).
  33. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Proba- bilistic Models. In Advances in Neural Information Processing Systems (NeurIPS).
  34. Shu Hu, Yuezun Li, and Siwei Lyu. 2021. Exposing GAN-Generated Faces Using Inconsistent Corneal Specular Highlights. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  35. Nils Hulzebosch, Sarah Ibrahimi, and Marcel Worring. 2020. Detecting CNN- generated Facial Images in Real-World Scenarios. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  36. Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. 2023. Scaling up GANs for Text-to-Image Synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  37. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations (ICLR).
  38. Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021. Alias-Free Generative Adversarial Networks. In Advances in Neural Information Processing Systems (NeurIPS).
  39. Tero Karras, Samuli Laine, and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  40. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and Improving the Image Quality of StyleGAN. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  41. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).
  42. Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In International Conference on Learning Representations (ICLR).
  43. Federica Lago, Cecilia Pasquini, Rainer Böhme, Hélène Dumont, Valérie Gof- faux, and Giulia Boato. 2022. More Real than Real: A Study on Human Visual Perception of Synthetic Faces [Applications Corner]. IEEE Signal Processing Magazine (2022).
  44. Sara Mandelli, Nicolò Bonettini, Paolo Bestagini, and Stefano Tubaro. 2020. Training CNNs in Presence of JPEG Compression: Multimedia Forensics vs Computer Vision. In IEEE International Workshop on Information Forensics and Security (WIFS).
  45. Federico Marcon, Cecilia Pasquini, and Giulia Boato. 2021. Detection of Ma- nipulated Face Videos over Social Networks: A Large-Scale Study. Journal of Imaging (2021).
  46. Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. 2019. Do GANs Leave Artificial Fingerprints?. In IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).
  47. Falko Matern, Christian Riess, and Marc Stamminger. 2019. Exploiting Vi- sual Artifacts to Expose Deepfakes and Face Manipulations. In IEEE Winter Applications of Computer Vision (WACV) Workshops.
  48. Scott McCloskey and Michael Albright. 2019. Detecting GAN-generated Im- agery Using Saturation Cues. In IEEE International Conference on Image Process- ing (ICIP).
  49. Leland McInnes, John Healy, and James Melville. 2020. UMAP: Uniform Mani- fold Approximation and Projection for Dimension Reduction. arXiv preprint (2020).
  50. Jaron Mink, Licheng Luo, Natã M. Barbosa, Olivia Figueira, Yang Wang, and Gang Wang. 2022. DeepPhish: Understanding User Trust towards Artificially Generated Profiles in Online Social Networks. In USENIX Security Symposium.
  51. Sophie Nightingale, Shruti Agarwal, Erik Härkönen, Jaakko Lehtinen, and Hany Farid. 2021. Synthetic Faces: How Perceptually Convincing Are They? Journal of Vision (2021).
  52. Sophie J. Nightingale and Hany Farid. 2022. AI-synthesized Faces Are Indistin- guishable from Real Faces and More Trustworthy. Proceedings of the National Academy of Sciences (2022).
  53. Leonardo Nizzoli, Serena Tardelli, Marco Avvenuti, Stefano Cresci, Maurizio Tesconi, and Emilio Ferrara. 2020. Charting the Landscape of Online Cryptocur- rency Manipulation. IEEE Access (2020).
  54. Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. 2023. Towards Universal Fake Image Detectors That Generalize across Generative Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  55. Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. 2022. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  56. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research (JMLR) (2011).
  57. Jürgen Pfeffer, Daniel Matter, Kokil Jaidka, Onur Varol, Afra Mashhadi, Jana Lasser, Dennis Assenmacher, Siqi Wu, Diyi Yang, Cornelia Brantner, Daniel M. Romero, Jahna Otterbacher, Carsten Schwemmer, Kenneth Joseph, David Garcia, and Fred Morstatter. 2023. Just Another Day on Twitter: A Complete 24 Hours of Twitter Data. In International AAAI Conference on Web and Social Media (ICWSM).
  58. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models from Natural Language Supervision. In International Conference on Machine Learning (ICML).
  59. Jacob Ratkiewicz, Michael Conover, Mark Meiss, Bruno Goncalves, Alessandro Flammini, and Filippo Menczer. 2011. Detecting and Tracking Political Abuse in Social Media. In International AAAI Conference on Web and Social Media (ICWSM).
  60. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  61. Jonas Ricker, Denis Lukovnikov, and Asja Fischer. 2024. AEROBLADE: Training- Free Detection of Latent Diffusion Images Using Autoencoder Reconstruc- tion Error. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  62. Sippo Rossi, Youngjin Kwon, Odd Harald Auglend, Raghava Rao Mukkamala, Matti Rossi, and Jason Thatcher. 2023. Are Deep Learning-Generated Social Media Profiles Indistinguishable from Real Profiles?. In Hawaii International Conference on System Sciences (HICSS).
  63. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bern- stein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (2015).
  64. Johan Sabel and Harald Stiff. 2021. Detecting Generated Media: A Case Study on Twitter Data. NATO STO Meeting Proceedings (2021).
  65. Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. 2021. Projected GANs Converge Faster. In Advances in Neural Information Processing Systems (NeurIPS).
  66. Katja Schwarz, Yiyi Liao, and Andreas Geiger. 2021. On the Frequency Bias of Generative Models. In Advances in Neural Information Processing Systems (NeurIPS).
  67. Bingyu Shen, Brandon RichardWebster, Alice O'Toole, Kevin Bowyer, and Walter J. Scheirer. 2021. A Study of the Human Perception of Synthetic Faces. In IEEE International Conference on Automatic Face and Gesture Recognition (FG).
  68. Naman Singh, Tushar Sharma, Abha Thakral, and Tanupriya Choudhury. 2018. Detection of Fake Profile in Online Social Networks Using Machine Learning. In International Conference on Advances in Computing and Communication Engineering (ICACCE).
  69. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep Unsupervised Learning Using Nonequilibrium Thermodynamics. In International Conference on Machine Learning (ICML).
  70. Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. 2016. Conditional Image Generation with PixelCNN Decoders. In Advances in Neural Information Processing Systems (NeurIPS).
  71. Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel Recurrent Neural Networks. In International Conference on Machine Learning (ICML).
  72. Emily L. Wang, Luca Luceri, Francesco Pierri, and Emilio Ferrara. 2023. Identify- ing and Characterizing Behavioral Classes of Radicalization within the QAnon Conspiracy on Twitter. In International AAAI Conference on Web and Social Media (ICWSM).
  73. Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. 2020. CNN-generated Images Are Surprisingly Easy to Spot... for Now. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  74. Yuping Wang, Chen Ling, and Gianluca Stringhini. 2023. Understanding the Use of Images to Spread COVID-19 Misinformation on Twitter. Proc. ACM Hum.-Comput. Interact. (2023).
  75. Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming- Hsuan Yang. 2023. GAN Inversion: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  76. Kai-Cheng Yang and Filippo Menczer. 2024. Anatomy of an AI-powered mali- cious social botnet. Journal of Quantitative Description: Digital Media (2024).
  77. Kai-Cheng Yang, Danishjeet Singh, and Filippo Menczer. 2024. Characteristics and Prevalence of Fake Social Media Profiles with AI-generated Faces. Journal of Online Trust and Safety (2024).
  78. Kai-Cheng Yang, Onur Varol, Pik-Mai Hui, and Filippo Menczer. 2020. Scal- able and Generalizable Social Bot Detection through Data Selection. In AAAI Conference on Artificial Intelligence (AAAI).
  79. Yuanshun Yao, Bimal Viswanath, Jenna Cryan, Haitao Zheng, and Ben Y. Zhao. 2017. Automated Crowdturfing Attacks and Defenses in Online Review Systems. In ACM SIGSAC Conference on Computer and Communications Security (CCS).
  80. Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianx- iong Xiao. 2016. LSUN: Construction of a Large-Scale Image Dataset Using Deep Learning with Humans in the Loop. arXiv preprint (2016).
  81. Ning Yu, Larry S. Davis, and Mario Fritz. 2019. Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints. In IEEE/CVF International Conference on Computer Vision (ICCV).
  82. Savvas Zannettou, Tristan Caulfield, Jeremy Blackburn, Emiliano De Cristofaro, Michael Sirivianos, Gianluca Stringhini, and Guillermo Suarez-Tangil. 2018. On the Origins of Memes by Means of Fringe Web Communities. In Internet Measurement Conference (IMC).

Cited by

Sorted by citations

Authors
Ricker, J., Assenmacher, D., Holz, T., Fischer, A., & Quiring, E.
Affiliations
  • Ruhr University Bochum (DE)
  • GESIS - Leibniz Institute for the Social Sciences (DE)
  • Helmholtz Center for Information Security (DE)
  • International Computer Science Institute (US)
ORCID
Venue
The 27th International Symposium on Research in Attacks, Intrusions and Defenses
Publisher
ACM
Published
2024-09-30
Pages
513-530
DOI
10.1145/3678890.3678922
OpenAlex
W4402957943
Cited by
22
References
6
Citations by year
2026: 11 · 2025: 7 · 2024: 4
FWCI
22.86
Type
conference paper
Language
en
Open access
Licence
Locations
  • International Symposium on Research in Attacks, Intrusions and Defenses (RAID) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • arXiv (Cornell University)
  • CISPA Helmholtz Center (cc-by)
Funders
  • Deutsche Forschungsgemeinschaft (EXC 2092 CASA - 390781972)
  • Leibniz Association Competition (P101/2020)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
AI-generated images, generative AI, fake accounts, disinformation, deepfakes, synthetic media detection, large-scale image analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review