CoChat
Sign Up
Cited by 17Misinformation and Its ImpactsEthics and Social Impacts of AICybercrime and Law Enforcement Studies

Studying the Online Deepfake Community

Open access
Timmerman, Brian, Mehta, Pulak, Deb, Progga, Gallagher, Kevin, Dolan-Gavitt, Brendan, Garg, Siddharth, Greenstadt, Rachel·2023·Journal of Online Trust and Safety
DOI 10.54501/jots.v2i1.126
Open this paper in CoChat — free accountOpen PDF

Abstract

Deepfakes have become a dual-use technology with applications in the domains of art, science, and industry. However, the technology can also be leveraged maliciously in areas such as disinformation, identity fraud, and harassment. In response to the technology's dangerous potential many deepfake creation communities have been deplatformed, including the technology's originating community – r/deepfakes. Opening in February 2018, just eight days after the removal of r/deepfakes, MrDeepFakes (MDF) went online as a privately owned platform to fulfill the role of community hub, and has since grown into the largest dedicated deepfake creation and discussion platform currently online. This position of community hub is balanced against the site's other main purpose, which is the hosting of deepfake pornography depicting public figures- produced without consent. In this paper we explore the two largest deepfake communities that have existed via a mixed methods approach utilizing quantitative and qualitative analysis. We seek to identify how these platforms were and are used by their members, what opinions these deepfakers hold about the technology and how it is seen by society at large, and identify how deepfakes-as-disinformation is viewed by the community. We find that there is a large emphasis on technical discussion on these platforms, intermixed with potentially malicious content. Additionally, we find the deplatforming of deepfake communities early in the technology's life has significantly impacted trust regarding alternative community platforms.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.54501/jots.v2i1.126

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The increased scalability, multilingualism, and multimodality of AI-generated content further complicate its detection and mitigation.

    “Additionally, the increased scalability, multilingualism, and multimodality of AI-generated content further complicate its detection and mitigation (Feuerriegel et al. 2023; Timmerman et al. 2023).”

    From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Timmerman et al. 2023

    The Discussion (section 6; page unavailable) addresses moderation of deepfake communities and warns that suppressing discussion can amplify harmful uses. Our copy does not link scalability, multilingualism, or multimodality of AI-generated content to difficulties detecting or mitigating that content; searches for those terms and for detection at scale found no such passage. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Afroz, Sadia, Vaibhav Garg, Damon McCoy, and Rachel Greenstadt. 2013. "Honor among thieves: A common's analysis of cybercrime economies." In 2013 APWG eCrime Researchers Summit, 1-11. IEEE. https://doi.org/10.1109/ecrs.2013.6805778. · DOI
  2. Afroz, Sadia, Aylin Caliskan Islam, Ariel Stolerman, Rachel Greenstadt, and Damon McCoy. 2014. "Doppelgänger finder: Taking stylometry to the underground." In 2014 IEEE Symposium on Security and Privacy, 212-26. IEEE. https : //doi.org/10.1109/sp .2014. · DOI
  3. Allyn, Bobby. 2022. "Deepfake video of Zelenskyy could be 'tip of the iceberg' in info war, experts warn." NPR (March). https://www.npr.org/2022/03/16/1087062648/deepf ake-video-zelenskyy-experts-war-manipulation-ukraine-russia. · DOI
  4. Attard, Angelica, and Neil S Coulson. 2012. "A thematic analysis of patient communica- tion in Parkinson's disease online support group discussion forums." Computers in Human Behavior 28 (2): 500-506. https://doi.org/10.1016/j.chb.2011.10.022. · DOI
  5. Bode, Lisa. 2021. "Deepfaking Keanu: YouTube deepfakes, platform visual effects, and the complexity of reception." Convergence 27 (4): 919-34. https://doi.org/10.1177 /13548565211030454. · DOI
  6. Burkell, Jacquelyn, and Chandell Gosse. 2019. "Nothing new here: Emphasizing the social and cultural context of deepfakes." First Monday, https://doi.org/10.5210/fm .v24i12.10287. · DOI
  7. Caporusso, Nicholas. 2020. "Deepfakes for the good: A beneficial application of con- tentious artificial intelligence technology." In International Conference on Applied Human Factors and Ergonomics, 235-41. Springer. https://doi.org/10.1007/978-3 -030-51328-3_33. · DOI
  8. Cheng, Xu, Jiangchuan Liu, and Cameron Dale. 2013. "Understanding the characteristics of internet short video sharing: A youtube-based measurement study." IEEE Transac- tions on Multimedia 15 (5): 1184-94. https://doi.org/10.1109/tmm.2013.2265531. · DOI
  9. Chivers, Bonnie R, Rhonda M Garad, Jacqueline A Boyle, Helen Skouteris, Helena J Teede, and Cheryce L Harrison. 2020. "Perinatal distress during COVID-19: thematic analysis of an online parenting forum." Journal of Medical Internet Research 22 (9): e22002. https://doi.org/10.2196/22002. · DOI
  10. Chun, Hyunwoo, Haewoon Kwak, Young-Ho Eom, Yong-Yeol Ahn, Sue Moon, and Hawoong Jeong. 2008. "Comparison of online social relations in volume vs interaction: a case study of cyworld." In Proceedings of the 8th ACM SIGCOMM Conference on Internet Measurement, 57-70. https://doi.org/10.1145/1452520.1452528. · DOI
  11. DeCuir-Gunby, Jessica T, Patricia L Marshall, and Allison W McCulloch. 2011. "Devel- oping and using a codebook for the analysis of interview data: An example from a professional development research project." Field Methods 23 (2): 136-55. https: //doi.org/10.1177/1525822x10388468. · DOI · Paper page
  12. Etienne, Hubert. 2021. "The future of online trust (and why Deepfake is advancing it)." AI and Ethics, 1-10. https://doi.org/10.1007/s43681-021-00072-1. · DOI · Paper page
  13. Farid, Hany. 2022. "Creating, Using, Misusing, and Detecting Deep Fakes." Journal of Online Trust and Safety 1 (4). https://doi.org/10.54501/jots.v1i4. · DOI
  14. Fingers, Jon. 2018. "Reddit bans the 'deepfake' AI porn it helped spawn." Engadget (February). https://www.engadget.com/2018-02-07-reddit-bans-deepfake-ai-por n.html. · DOI
  15. Forte, Andrea, Nazanin Andalibi, and Rachel Greenstadt. 2017. "Privacy, Anonymity, and Perceived Risk in Open Collaboration: A Study of Tor Users and Wikipedians." In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 1800-1811. CSCW '17. Portland, Oregon, USA: Association for Computing Machinery. ISBN: 9781450343350. https://doi.org/10.1145/29981 81.2998273. · DOI
  16. Franklin, Jason, Adrian Perrig, Vern Paxson, and Stefan Savage. 2007. "An inquiry into the nature and causes of the wealth of internet miscreants." Ccs 7:375-88. https: //doi.org/doi.org/10.1145/1315245.1315292. · DOI
  17. Güera, David, and Edward J. Delp. 2018. "Deepfake Video Detection Using Recurrent Neural Networks." In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 1-6. https://doi.org/10.1109/AVSS.2018.86 39163. · DOI · Paper page
  18. Hine, Gabriel, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leon- tiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. 2017. "Kek, Cucks, and God Emperor Trump: A measurement study of 4chan's politically incor- rect forum and its effects on the web." In Proceedings of the International AAAI Conference on Web and Social Media, vol. 11. 1. https://doi.org/10.1609/icwsm.v 11i1.14893. · DOI
  19. Huang, Yihao, Felix Juefei-Xu, Run Wang, Qing Guo, Lei Ma, Xiaofei Xie, Jianwen Li, Weikai Miao, Yang Liu, and Geguang Pu. 2020. "Fakepolisher: Making deepfakes more detection-evasive by shallow reconstruction." In Proceedings of the 28th ACM International Conference on Multimedia, 1217-26. https://doi.org/10.1145/33941 71.3413732. · DOI · Paper page
  20. Kikerpill, Kristjan. 2020. "Choose your stars and studs: The rise of deepfake designer porn." Porn Studies 7 (4): 352-56. https://doi.org/10.1080/23268743.2020.17658 · DOI
  21. Koopman, Marissa, Andrea Macarulla Rodriguez, and Zeno Geradts. 2018. "Detection of deepfake video manipulation." In The 20th Irish Machine Vision and Image Processing Conference (IMVIP), 133-36. ISBN: 978-0-9934207-3-3. http : //hdl .handle.net/2262/89508. · DOI
  22. Kugler, Matthew B, and Carly Pace. 2021. "Deepfake Privacy: Attitudes and Regulation." Northwestern Public Law Research Paper, nos. 21-04, https://doi.org/10.2139/ssrn .3781968. · DOI
  23. Kwok, Andrei O. J., and Sharon G. M. Koh. 2021. "Deepfake: a social construction of technology perspective." Current Issues in Tourism 24 (13): 1798-802. https: //doi.org/10.1080/13683500.2020.1738357. · DOI
  24. Lee, YoungAh, Kuo-Ting Huang, Robin Blom, Rebecca Schriner, and Carl A Ciccarelli. 2021. "To believe or not to believe: framing analysis of content and audience response of top 10 deepfake videos on YouTube." Cyberpsychology, Behavior, and Social Networking 24 (3): 153-58. https://doi.org/10.1089/cyber.2020.0176. · DOI
  25. Li, Yuezun, and Siwei Lyu. 2018. "Exposing deepfake videos by detecting face warping artifacts." arXiv preprint arXiv:1811.00656, https://doi.org/10.48550/arXiv.1811.0 0656. · DOI · Paper page
  26. Lucas, Kweilin T. 2022. "Deepfakes and domestic violence: perpetrating intimate partner abuse using video technology." Victims & Offenders 17 (5): 647-59. https://doi.org /10.1080/15564886.2022.2036656. · DOI
  27. McDonald, Nora, Sarita Schoenebeck, and Andrea Forte. 2019. "Reliability and inter- rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice." Proceedings of the ACM on Human-Computer Interaction 3 (CSCW): 1-23. https://doi.org/10.1145/3359174. · DOI · Paper page
  28. Mehta, Pulak, Gauri Jagatap, Kevin Gallagher, Brian Timmerman, Progga Deb, Siddharth Garg, Rachel Greenstadt, and Brendan Dolan-Gavitt. 2023. "Can deepfakes be created on a whim?" In Companion Proceedings of the ACM Web Conference 2023, 1324-34. https://doi.org/10.1145/3543873.3587581. · DOI · Paper page
  29. Mondal, Mainack, Leandro Araújo Silva, and Fabrício Benevenuto. 2017. "A measurement study of hate speech in social media." In Proceedings of the 28th ACM Conference on Hypertext and Social Media, 85-94. https://doi.org/10.1145/3078714.3078723. · DOI
  30. Motoyama, Marti, Damon McCoy, Kirill Levchenko, Stefan Savage, and Geoffrey M Voelker. 2011. "An analysis of underground forums." In Proceedings of the 2011 ACM SIG- COMM Conference on Internet Measurement Conference, 71-80. https : //doi.org /10.1145/2068816.2068824. · DOI
  31. Neethirajan, Suresh. 2021. Beyond Deepfake Technology Fear: On its Positive Uses for Livestock Farming. Preprint. https://doi.org/10.20944/preprints202107.0326.v1. · DOI
  32. Popova, Milena. 2020. "Reading out of context: pornographic deepfakes, celebrity and intimacy." Porn Studies 7 (4): 367-81. https://doi.org/10.1080/23268743.2019.16 75090. Pushshift. Pushshift Reddit API Documentation. GitHub repository. Accessed August 16, 2023. https://github.com/pushshift/api. · DOI
  33. Smedley, Richard M, and Neil S Coulson. 2017. "A thematic analysis of messages posted by moderators within health-related asynchronous online support forums." Patient Education and Counseling 100 (9): 1688-93. https://doi.org/10.1016/j.pec.2017.0 4.008. · DOI
  34. Tseng, Emily, Rosanna Bellini, Nora McDonald, Matan Danos, Rachel Greenstadt, Damon McCoy, Nicola Dell, and Thomas Ristenpart. 2020. "The tools and tactics used in intimate partner surveillance: An analysis of online infidelity forums." In 29th USENIX Security Symposium (USENIX Security 20), 1893-909. ISBN: 978-1-939133-17-5. https://www.usenix.org/conference/usenixsecurity20/presentation/tseng.
  35. Westerlund, Mika. 2019. "The emergence of deepfake technology: A review." Technology Innovation Management Review 9 (11). https://doi.org/10.22215/timreview/1282. · DOI · Paper page
  36. Widdler, David Gray, Dawn Nafus, Laura Dabbish, and James Herbsleb. 2022. "Limits and Possibilities for "Ethical AI" in Open Source: A Study of Deepfakes." In 2022 ACM Conference on Fairness, Accountability, and Transparency. https : //doi.org /10.1145/3531146.3533779. https://www.youtube.com/watch?v=t59gRbpYMiY Yes 1,585,910 https://www.youtube.com/watch?v=lSM-9RBk3HQ Yes 786,179 https://www.youtube.com/watch?v=Zmutd9618Kk Yes 620,899 https://www.youtube.com/watch?v=wYSmp-nrJ7M Yes 277,330 https://www.youtube.com/watch?v=hP3njGbvvWc Yes 196,051 https://www.youtube.com/watch?v=tW7EENTWXRk Yes 195,374 https://www.youtube.com/watch?v=0p-nNSvB7KA Yes 117,565 https://www.youtube.com/watch?v=q44LPygdMxU Yes 97,360 https://www.youtube.com/watch?v=QSmHho1uHFM Yes 61,504 https://www.youtube.com/watch?v=_bc3SPbCdW8 Yes 50,173 https://www.youtube.com/watch?v=rw5F5lSvBLE Yes 46,926 https://www.youtube.com/watch?v=CHs3VuW7TtU Yes 38,157 https://www.youtube.com/watch?v=ljMXS8vovx4 Yes 37,301 https://www.youtube.com/watch?v=N6XN0fjHZSA Yes 36,514 https://www.youtube.com/watch?v=1Bt5wyGqdk4 Yes 32,432 https://www.youtube.com/watch?v=zpOclEB2-dk Yes 15,845 https://www.youtube.com/watch?v=k0Z-ZuQ1fi8 Yes 2,522 https://www.youtube.com/watch?v=qaqRLopz0wA No 309,593 https://www.youtube.com/watch?v=bQihTE3QBVk No 87,235 https://www.youtube.com/watch?v=pud_1PUoFXA No 78,782 https://www.youtube.com/watch?v=is347MG71yY No 50,703 https://www.youtube.com/watch?v=OI1LEN-SgLM No 50,016 https://www.youtube.com/watch?v=VBNCusXEhA8 No 25,945 https://www.youtube.com/watch?v=5CU81pJAPV4 No 16,364 https://www.youtube.com/watch?v=W8xaFW-2UR8 No 8,602 https://www.youtube.com/watch?v=ev1rhuBzBCw No 7,782 https://www.youtube.com/watch?v=rw0oavibpF8 No 6,736 https://www.youtube.com/watch?v=KLe7-3bMpg4 No 3,131 https://www.youtube.com/watch?v=MjyBswm3dG4 No 2,410 https://www.youtube.com/watch?v=DQGRD9KfdCw No 436 · DOI

Cited by

Sorted by citations

Authors
Timmerman, B., Mehta, P., Deb, P., Gallagher, K., Dolan-Gavitt, B., Garg, S., & Greenstadt, R.
Affiliations
  • New York University (US)
  • Universidade Nova de Lisboa (PT)
ORCID
Venue
Journal of Online Trust and Safety
Publisher
Stanford Internet Observatory
Published
2023-09-21
Volume
2
Issue
1
ISSN
2770-3142 (ISSN-L)
DOI
10.54501/jots.v2i1.126
OpenAlex
W4386965396
Cited by
17
References
3
Citations by year
2026: 8 · 2025: 7 · 2024: 2
FWCI
5.85
Type
journal article
Language
en
Open access
Licence
Locations
Journal of Online Trust and Safety (publishedVersion, cc-by-nc-sa)
Funders
  • National Science Foundation (2016061)
  • NOVA Laboratory for Computer Science and Informatics (UIDB/04516/2020)
  • Fundação para a Ciência e a Tecnologia (UIDB/04516/2020)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Cybercrime and Law Enforcement Studies
Keywords
deepfakes, deplatforming, disinformation, deepfake pornography, identity fraud, online harassment, community trust, dual-use technology, online communities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review