The Radicalization Risks of GPT-3 and Advanced Neural Language Models
Open accessAbstract
In this paper, we expand on our previous research of the potential for abuse of generative language models by assessing GPT-3. Experimenting with prompts representative of different types of extremist narrative, structures of social interaction, and radical ideologies, we find that GPT-3 demonstrates significant improvement over its predecessor, GPT-2, in generating extremist texts. We also show GPT-3's strength in generating text that accurately emulates interactive, informational, and influential content that could be utilized for radicalizing individuals into violent far-right extremist ideologies and behaviors. While OpenAI's preventative measures are strong, the possibility of unregulated copycat technology represents significant risk for large-scale online radicalization and recruitment; thus, in the absence of safeguards, successful and efficient weaponization that requires little experimentation is likely. AI stakeholders, the policymaking community, and governments should begin investing as soon as possible in building social norms, public policy, and educational initiatives to preempt an influx of machine-generated disinformation and propaganda. Mitigation will require effective policy and partnerships across industry, government, and civil society.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The ability of LLMs to generate human-like text could be used to create fake news articles or other seemingly legitimate but fabricated or misleading content without the reader realizing it is AI-produced.
“The ability of LLMs to generate texts similar to those composed by humans could be used to create fake news articles or other seemingly legitimate but actually fabricated or misleading content, (45, 46) without the reader realizing that the text is produced by AI (47).”
From ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health (2023) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Sageman, M. (2008) · University of Pennsylvania Press eBooks · doi:10.9783/9780812206784 · Cited by 710 · see citing works
Release Strategies and the Social Impacts of Language Models
Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.09203 · Cited by 292 · see citing works
Self-citationTHE RADICAL ONLINE: INDIVIDUAL RADICALIZATION PROCESSES AND THE ROLE OF THE INTERNET
Köehler, D. (2014) · DOAJ (DOAJ: Directory of Open Access Journals) · Cited by 134 · see citing works
Developing an explanatory model for the process of online radicalisation and terrorism
Torok, R. (2013) · Security Informatics · doi:10.1186/2190-8532-2-6 · Cited by 76 · see citing works
Purici, Ş. (2020) · Codrul Cosminului · doi:10.4316/cc.2020.02.009 · Cited by 63 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [4][4] Alexander Meleagrou-Hitchens and Nick Kaderbhai. Research perspectives on online radicalisation: A literature review, 2006-2016. International Centre for the Study of Radicalisation , 2017. -
- [8][8] Mehmet F. Bastug, Aziz Douai, and Davut Akca. Exploring the 'demand side' of online radicalization: Evidence from the canadian context. Studies in Conflict & Terrorism , 43(7):616-637, 2020. -
Cited by
Sorted by citations
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
“Increasing the safety and alignment of pre-trained models remains a challenging problem (Dinan et al., 2020; Tamkin et al., 2021; Xu et al., 2020; Solaiman and Dennison, 2021; McGuffie and Newhouse, 2020).”
BackgroundChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1166120 · Cited by 708 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Korngiebel, D. M., & Mooney, S. D. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00464-x · Cited by 259 · see citing works
ChatGPT in physics education: A pilot study on easy-to-implement activities
Bitzenbauer, P. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13176 · Cited by 181 · see citing works
Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.12867 · Cited by 164 · see citing works
Jeyaraman, M., Ramasubramanian, S., Balaji, S., Jeyaraman, N., Nallakumarasamy, A., & Sharma, S. (2023) · World Journal of Methodology · doi:10.5662/wjm.v13.i4.170 · Cited by 125 · see citing works
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150 · see citing works
Garrido-Muñoz, I., Montejo-Ráez, A., Martínez-Santiago, F., & Ureña-López, L. A. (2021) · Applied Sciences · doi:10.3390/app11073184 · Cited by 123 · see citing works
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Kang, D., Li, X., Stoica, I., Guestrin, C., Zaharia, M., & Hashimoto, T. (2024) · 2024 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw63631.2024.00018 · Cited by 114 · see citing works
ChatGPT and the Rise of Large Language Models: The New AI-Driven Infodemic Threat in Public Health
De Angelis, L., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023) · doi:10.2139/ssrn.4352931 · Cited by 708 · see citing works
The Gradient of Generative AI Release: Methods and Considerations
Solaiman, I. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593981 · Cited by 90 · see citing works
Makhortykh, M., Zucker, E. M., Simon, D. J., Bultmann, D., & Ulloa, R. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00072-6 · Cited by 48 · see citing works
Impacts of generative artificial intelligence on the future of labor market: a systematic review
Salari, N., Beiromvand, M., Hosseinian-Far, A., Habibi, J., Babajani, F., & Mohammadi, M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100652 · Cited by 49 · see citing works
“Together with LLMs, these models can produce texts resembling human-written content, though they may also generate other accurate and/or fake contents (Biswas, 2023; Gupta et al., 2022; McGuffie & Newhouse, 2020).”
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40 · see citing works
Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196 · see citing works
- Authors
- McGuffie, K., & Newhouse, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2009.06807
- OpenAlex
- W3086249591
- Cited by
- 86
- References
- 5
- Citations by year
- 2025: 5 · 2024: 16 · 2023: 29 · 2022: 19 · 2021: 14 · 2020: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Sociology and Political Science › Terrorism, Counterterrorism, and Political Violence
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- GPT-3, large language models, radicalization, far-right extremism, disinformation, propaganda, AI-generated text, GPT-2, online recruitment, AI policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
