CoChat
Sign Up
PreprintCited by 152Misinformation and Its ImpactsEthics and Social Impacts of AISpam and Phishing Detection

Generative language models and automated influence operations: Emerging threats and potential mitigations

Open access
Goldstein, Josh A., Sastry, Girish, Musser, Micah, DiResta, Renee, Gentzel, Matthew, Sedova, Katerina·2023·arXiv
DOI 10.48550/arxiv.2301.04246arXiv 2301.04246
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative language models have improved drastically, and can now produce realistic text outputs that are difficult to distinguish from human-written content. For malicious actors, these language models bring the promise of automating the creation of convincing and misleading text for use in influence operations. This report assesses how language models might change influence operations in the future, and what steps can be taken to mitigate this threat. We lay out possible changes to the actors, behaviors, and content of online influence operations, and provide a framework for stages of the language model-to-influence operations pipeline that mitigations could target (model construction, model access, content dissemination, and belief formation). While no reasonable mitigation can be expected to fully prevent the threat of AI-enabled influence operations, a combination of multiple mitigations may make an important difference.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2301.04246

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Algorithmic systems can customize fabricated content to match the biases of particular audiences.

    “Algorithmic systems can customize fabricated content to match the biases of particular audiences (Goldstein et al., 2023).”

    From Deepfakes, Democracy, and the Ethics of Synthetic Media: A Synthesis of the SOLARIS Project (2026) · cites this paper as Goldstein et al., 2023

    Section 4.1 says language models could generate articles tailored to audience demographics for propagandists, and section 5.5 discusses persuasive arguments that exploit viewers’ biases. But section 4.1 presents effective audience-specific tailoring as a future possibility whose persuasive payoff remains uncertain, rather than an established capability as the claim implies. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Language models bring the promise of automating the creation of convincing and misleading text for use in influence operations, rather than having to rely on human labor.

    “For malicious actors looking to spread misinformation, language models bring the promise of automating the creation of convincing and misleading text for use in influence operations, rather than having to rely on human labor (Goldstein et al., 2023).”

    From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Goldstein et al., 2023

  • AI-generated misinformation poses significant challenges to defense strategies previously employed by digital platforms and users.

    “This emerging form of digital misinformation poses significant challenges to the defense strategies previously employed by digital platforms and users (Goldstein et al. 2023; Feuerriegel et al. 2023).”

    From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Goldstein et al. 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 100

  1. AlQuraishi, Mohammed. "Machine learning in protein structure prediction." Current Opinion in Chemical Biology 65 (December 2021): 1–8. ISSN: 1367-5931. https://doi.org/10.1016/J.CBPA.2021.04. 005. · DOI
  2. Altay, Sacha, Anne Sophie Hacquin, Coralie Chevallier, and Hugo Mercier. "Information delivered by a chatbot has a positive impact on COVID-19 vaccines attitudes and intentions." Journal of Experimen- tal Psychology: Applied, October 28, 2021. ISSN: 1939-2192. https://doi.org/10.1037/XAP0000400. · DOI
  3. Bagdasaryan, Eugene, and Vitaly Shmatikov. "Spinning Language Models: Risks of Propaganda-As-A- Service and Countermeasures." 2022 IEEE Symposium on Security and Privacy, 2022, 769–786. htt ps://doi.org/10.1109/SP46214.2022.9833572. · DOI
  4. Bail, Christopher A., Brian Guay, Emily Maloney, Aidan Combs, D. Sunshine Hillygus, Friedolin Merhout, Deen Freelon, and Alexander Volfovsky. "Assessing the Russian Internet Research Agency's impact on the political attitudes and behaviors of American Twitter users in late 2017." PNAS 117, no. 1 (January 7, 2020). https://doi.org/10.1073/pnas.1906420116. · DOI · Paper page
  5. Balla, Steve, Reeve Bull, Bridget Dooling, Emily Hammond, Michael Herz, Michael Livermore, and Beth Simone Noveck. Mass, Computer-Generated, and Fraudulent Comments. Report to the Administrative Conference of the U.S., June 17, 2020. https :// regulatorystudies . columbian .gwu .edu/mass - computer-generated-and-fraudulent-comments-0.
  6. "Best Practices for Deploying Language Models." Cohere, June 2, 2022. https://txt.cohere.ai/best- practices-for-deploying-language-models/. https://www.npr.org/2022/03/16/1087062648/deepfake-video-zelenskyy-experts-war-manipulation-ukraine-russia https://www.npr.org/2022/03/16/1087062648/deepfake-video-zelenskyy-experts-war-manipulation-ukraine-russia https://doi.org/10.1016/J.CBPA.2021.04.005 https://doi.org/10.1016/J.CBPA.2021.04.005 https://doi.org/10.1037/XAP0000400 https://openai.com/api/ https://about.fb.com/news/2020/09/august-2020-cib-report/ https://about.fb.com/news/2020/09/august-2020-cib-report/ https://www.huffingtonpost.co.uk/entry/deepfake-porn_uk_5bf2c126e4b0f32bd58ba316 https://www.huffingtonpost.co.uk/entry/deepfake-porn_uk_5bf2c126e4b0f32bd58ba316 https://doi.org/10.1109/SP46214.2022.9833572 https://doi.org/10.1109/SP46214.2022.9833572 https://doi.org/10.1073/pnas.1906420116 https://openai.com/blog/vpt/ https://regulatorystudies.columbian.gwu.edu/mass-computer-generated-and-fraudulent-comments-0 https://regulatorystudies.columbian.gwu.edu/mass-computer-generated-and-fraudulent-comments-0 https://carnegieendowment.org/2021/06/28/measuring-effects-of-influence-operations-key-findings-and-gaps-from-empirical-research-pub-84824 https://carnegieendowment.org/2021/06/28/measuring-effects-of-influence-operations-key-findings-and-gaps-from-empirical-research-pub-84824 https://wudaoai.cn/model/ https://wudaoai.cn/model/ https://txt.cohere.ai/best-practices-for-deploying-language-models/ https://txt.cohere.ai/best-practices-for-deploying-language-models/ · DOI
  7. Bliss, Nadya, Elizabeth Bradley, Joshua Garland, Filippo Menczer, Scott W. Ruston, Kate Starbird, and Chris Wiggins. "An Agenda for Disinformation Research." arxiv:2012.08572 [cs.CY], December 2020. https://doi.org/10.48550/arxiv.2012.08572. · DOI
  8. Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, et al. "On the Opportunities and Risks of Foundation Models." arxiv:2108.07258 [cs.LG], August 2021. https://doi.org/10.48550/arxiv.2108.07258. · DOI · Paper page
  9. Borgeaud, Sebastian, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Van Den Driessche, et al. "Improving language models by retrieving from trillions of tokens." arxiv:2112.04426 [cs.CL], December 2021. https://doi.org/10.48550/arxiv.2112.04426. · DOI · Paper page
  10. Boucher, Nicholas, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. "Bad Characters: Impercep- tible NLP Attacks." 2022 IEEE Symposium on Security and Privacy, June 2022, 1987–2004. ISSN: 10816011. https://doi.org/10.48550/arxiv.2106.09898. · DOI
  11. Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. "Language Models are Few-Shot Learners." Advances in Neural Information Pro- cessing Systems 33 (May 2020). ISSN: 10495258. https://doi.org/10.48550/arxiv.2005.14165. · DOI · Paper page
  12. Brundage, Miles, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, et al. "Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims." arxiv:2004.07213 [cs.CY], April 2020. https://doi.org/10.48550/arxiv.2004.07213. · DOI · Paper page
  13. Buchanan, Ben, Andrew Lohn, Micah Musser, and Katerina Sedova. Truth, Lies, and Automation: How Language Models Could Change Disinformation. Center for Security and Emerging Technology, May 2021. https://doi.org/10.51593/2021CA003. · DOI · Paper page
  14. Buchanan, Ben, and Taylor Miller. Machine Learning for Policy Makers: What It Is and Why It Matters. Belfer Center for Science and International Affairs, June 2017. https://www.belfercenter.org/ sites/default/files/files/publication/MachineLearningforPolicymakers.pdf. https://www.unicef.org/eca/media/13636/file https://www.unicef.org/eca/media/13636/file https://about.fb.com/news/2020/01/enforcing-against-manipulated-media/ https://about.fb.com/news/2020/01/enforcing-against-manipulated-media/ https://doi.org/10.48550/arxiv.2012.08572 https://doi.org/10.48550/arxiv.2108.07258 https://www.npr.org/2022/12/15/1143114122/ai-generated-fake-faces-have-become-a-hallmark-of-online-influence-operations https://www.npr.org/2022/12/15/1143114122/ai-generated-fake-faces-have-become-a-hallmark-of-online-influence-operations https://en.unesco.org/publications/balanceact https://en.unesco.org/publications/balanceact https://doi.org/10.48550/arxiv.2112.04426 https://doi.org/10.48550/arxiv.2106.09898 https://www.wired.com/story/isps-funded-85-million-fake-comments-opposing-net-neutrality/ https://www.theatlantic.com/ideas/archive/2021/01/bigger-threat-was-always-domestic/617618/ https://www.theatlantic.com/ideas/archive/2021/01/bigger-threat-was-always-domestic/617618/ https://doi.org/10.48550/arxiv.2005.14165 https://doi.org/10.48550/arxiv.2004.07213 https://doi.org/10.51593/2021CA003 https://www.belfercenter.org/sites/default/files/files/publication/MachineLearningforPolicymakers.pdf https://www.belfercenter.org/sites/default/files/files/publication/MachineLearningforPolicymakers.pdf · DOI
  15. Chen, Huili, Cheng Fu, Bita Darvish Rouhani, Jishen Zhao, and Farinaz Koushanfar. "DeepAttest: An end-to-end attestation framework for deep neural networks." Proceedings of the 46th International Symposium on Computer Architecture, June 2019, 487–498. ISSN: 10636897. https://doi.org/10. 1145/3307650.3322251.
  16. Chen, Mark, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, et al. "Evaluating Large Language Models Trained on Code." arxiv:2107.03374 [cs.LG], July 14, 2021. https://doi.org/10.48550/arxiv.2107.03374. · DOI · Paper page
  17. Chesney, Robert, and Danielle Citron. "Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security." California Law Review 107, no. 6 (2019): 1753. https://doi.org/10.15779/ Z38RV0D15J.
  18. Chung, Hyung Won, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, et al. "Scaling Instruction-Finetuned Language Models." arxiv:2210.11416 [cs.LG], October 20, 2022. ht tps://doi.org/10.48550/arxiv.2210.11416. · DOI · Paper page
  19. "Curbing Misuse at Dall-E 2." OpenAI. Accessed June 27, 2022. https://openai.com/dall-e-2/. https://bigscience.huggingface.co/blog/building-a-tb-scale-multilingual-dataset-for-language-modeling https://bigscience.huggingface.co/blog/building-a-tb-scale-multilingual-dataset-for-language-modeling https://c2pa.org/specifications/specifications/1.0/security/Harms_Modelling.html https://c2pa.org/specifications/specifications/1.0/security/Harms_Modelling.html https://doi.org/10.1145/3307650.3322251 https://doi.org/10.1145/3307650.3322251 https://doi.org/10.48550/arxiv.2107.03374 https://doi.org/10.15779/Z38RV0D15J https://doi.org/10.15779/Z38RV0D15J https://www.atlanticcouncil.org/in-depth-research-reports/report/the-madcom-future/ https://www.atlanticcouncil.org/in-depth-research-reports/report/the-madcom-future/ https://www.economist.com/asia/2021/11/20/chinese-propagandists-court-south-east-asias-chinese-diaspora https://www.economist.com/asia/2021/11/20/chinese-propagandists-court-south-east-asias-chinese-diaspora https://www.economist.com/asia/2021/11/20/chinese-propagandists-court-south-east-asias-chinese-diaspora https://doi.org/10.48550/arxiv.2210.11416 https://doi.org/10.48550/arxiv.2210.11416 https://docs.cohere.ai/api-reference/ https://www.ftc.gov/reports/combatting-online-harms-through-innovation https://www.bis.doc.gov/index.php/documents/about-bis/newsroom/press-releases/3158-2022-10-07-bis-press-release-advanced-computing-and-semiconductor-manufacturing-controls-final/file https://www.bis.doc.gov/index.php/documents/about-bis/newsroom/press-releases/3158-2022-10-07-bis-press-release-advanced-computing-and-semiconductor-manufacturing-controls-final/file https://www.bis.doc.gov/index.php/documents/about-bis/newsroom/press-releases/3158-2022-10-07-bis-press-release-advanced-computing-and-semiconductor-manufacturing-controls-final/file https://public-inspection.federalregister.gov/2022-21658.pdf https://public-inspection.federalregister.gov/2022-21658.pdf https://int.nyt.com/data/documenttools/2021-intelligence-community-election-interference-assessment/abd0346ebdd93e1e/full.pdf https://int.nyt.com/data/documenttools/2021-intelligence-community-election-interference-assessment/abd0346ebdd93e1e/full.pdf https://openai.com/dall-e-2/ · DOI
  20. Dhingra, Bhuwan, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. "Time-Aware Language Models as Temporal Knowledge Bases." Transactions of the Asso- ciation for Computational Linguistics 10 (March 2022): 257–273. ISSN: 2307387X. https://doi.org/ 10.1162/tacl_a_00459. · DOI
  21. Dill, Janina, Scott D. Sagan, and Benjamin A. Valentino. "Kettles of Hawks: Public Opinion on the Nuclear Taboo and Noncombatant Immunity in the United States, United Kingdom, France, and Israel." Security Studies 31, no. 1 (2022): 1–31. ISSN: 15561852. https://doi.org/10.1080/09636412. 2022.2038663. · DOI
  22. DiResta, Renée, Shelby Grossman, and Alexandra Siegel. "In-House Vs. Outsourced Trolls: How Digital Mercenaries Shape State Influence Strategies." Political Communication 39, no. 2 (2021): 222–253. ISSN: 10917675. https://doi.org/10.1080/10584609.2021.1994065. · DOI
  23. Downs, Anthony. "An Economic Theory of Political Action in a Democracy." Journal of Political Economy 65, no. 2 (1957): 135–150. https://www.jstor.org/stable/1827369.
  24. Earl, Jennifer, Thomas V. Maher, and Jennifer Pan. "The digital repression of social movements, protest, and activism: A synthetic review." Science Advances 8 (October 2022): 8198. https://www.science. org/doi/pdf/10.1126/sciadv.abl8198. · DOI
  25. England, Paul, Henrique S. Malvar, Eric Horvitz, Jack W. Stokes, Cédric Fournet, Rebecca Burke-Aguero, Amaury Chamayou, et al. "AMP: Authentication of Media via Provenance." MMSys 2021 - Proceed- ings of the 2021 Multimedia Systems Conference, June 2021, 108–121. https://doi.org/10.48550/ arxiv.2001.07886. · arXiv
  26. Evans, Owain, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. "Truthful AI: Developing and governing AI that does not lie." arxiv:2110.06674, October 13, 2021. https://doi.org/10.48550/arxiv.2110.06674. https://www.theguardian.com/technology/2020/sep/04/russia-media-disinformation-fake-news-peacedata https://www.theguardian.com/technology/2020/sep/04/russia-media-disinformation-fake-news-peacedata https://doi.org/10.1162/tacl_a_00459 https://doi.org/10.1162/tacl_a_00459 https://doi.org/10.1080/09636412.2022.2038663 https://doi.org/10.1080/09636412.2022.2038663 https://cyber.fsi.stanford.edu/io/publication/potemkin-think-tanks https://cyber.fsi.stanford.edu/io/publication/potemkin-think-tanks https://cyber.fsi.stanford.edu/io/publication/bed-embeds https://www.washingtonpost.com/opinions/2019/11/15/heres-how-russia-will-attack-election-were-still-not-ready/ https://www.washingtonpost.com/opinions/2019/11/15/heres-how-russia-will-attack-election-were-still-not-ready/ https://doi.org/10.1080/10584609.2021.1994065 https://www.jstor.org/stable/1827369 https://www.science.org/doi/pdf/10.1126/sciadv.abl8198 https://www.science.org/doi/pdf/10.1126/sciadv.abl8198 https://github.com/ai-forever/ru-gpts#readme https://doi.org/10.48550/arxiv.2001.07886 https://doi.org/10.48550/arxiv.2001.07886 https://doi.org/10.48550/arxiv.2110.06674 · DOI · Paper page
  27. Farid, Hany. "Creating, Using, Misusing, and Detecting Deep Fakes." Journal of Online Trust and Safety 1, no. 4 (September 2022). ISSN: 2770-3142. https://doi.org/10.54501/JOTS.V1I4.56. · DOI
  28. Finnemore, Martha, and Kathryn Sikkink. "International Norm Dynamics and Political Change." Inter- national Organization 52, no. 4 (1998): 887–917. https://www.jstor.org/stable/2601361.
  29. Fröhling, Leon, and Arkaitz Zubiaga. "Feature-based detection of automated language models: Tackling GPT-2, GPT-3 and Grover." PeerJ Computer Science 7 (April 6, 2021): 1–23. ISSN: 23765992. https: //doi.org/10.7717/peerj-cs.443. · DOI · Paper page
  30. Ganguli, Deep, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, et al. "Predictability and Surprise in Large Generative Models." 2022 ACM Conference on Fairness, Accountability, and Transparency, June 2022, 1747–1764. https ://doi . org/10 .1145/ 3531146.3533229.
  31. Gehrmann, Sebastian, Hendrik Strobelt, and Alexander M. Rush. "GLTR: Statistical Detection and Visu- alization of Generated Text." Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics: System Demonstrations, July 2019, 111–116. https://doi.org/10.18653/V1/P19- 3019. · DOI
  32. Geller, Tom. "Overcoming the Uncanny Valley." IEEE Computer Graphics and Applications 28, no. 4 (July- Aug. 2008): 11–17. ISSN: 02721716. https://doi.org/10.1109/MCG.2008.79. · DOI
  33. Goldstein, Josh A. "Foreign Influence Operations in the Cyber Age." PhD diss., University of Oxford, 2021. https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.840171.
  34. Goldstein, Josh A., and Renée DiResta. "This salesperson does not exist: How tactics from political in- fluence operations on social media are deployed for commercial lead generation." Harvard Kennedy School Misinformation Review 3, no. 5 (September 2022). https://doi.org/10.37016/MR-2020-104. https://doi.org/10.54501/JOTS.V1I4.56 https://beta.openai.com/docs/guides/fine-tuning http://web.archive.org/web/20220621204451/https://docs.cohere.ai/finetuning-wiki/ http://web.archive.org/web/20220621204451/https://docs.cohere.ai/finetuning-wiki/ https://www.jstor.org/stable/2601361 https://www.nytimes.com/2021/07/25/world/europe/disinformation-social-media.html https://science.house.gov/download/francois-addendum https://science.house.gov/download/francois-addendum https://www.nytimes.com/2021/06/30/technology/disinformation-message-apps.html https://www.nytimes.com/2021/06/30/technology/disinformation-message-apps.html https://doi.org/10.7717/peerj-cs.443 https://doi.org/10.7717/peerj-cs.443 https://doi.org/10.1145/3531146.3533229 https://doi.org/10.1145/3531146.3533229 https://doi.org/10.18653/V1/P19-3019 https://doi.org/10.18653/V1/P19-3019 https://doi.org/10.1109/MCG.2008.79 https://about.fb.com/news/2021/05/influence-operations-threat-report/ https://about.fb.com/news/2021/05/influence-operations-threat-report/ https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.840171 https://doi.org/10.37016/MR-2020-104 · DOI
  35. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. https://www. deeplearningbook.org/.
  36. Grinberg, Nir, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. "Fake news on Twitter during the 2016 U.S. presidential election." Science 363, no. 6425 (January 25, 2019): 374–378. ISSN: 10959203. https://doi.org/10.1126/science.aau2706. · DOI · Paper page
  37. Guess, Andrew M., Michael Lerner, Benjamin Lyons, Jacob M. Montgomery, Brendan Nyhan, Jason Reifler, and Neelanjan Sircar. "A digital media literacy intervention increases discernment between mainstream and false news in the United States and India." PNAS 117, no. 27 (July 2020): 15536– 15545. ISSN: 10916490. https://doi.org/10.1073/pnas.1920498117. · DOI · Paper page
  38. Helmus, Todd C., and Marta Kepe. "A Compendium of Recommendations for Countering Russian and Other State-Sponsored Propaganda." RAND Corporation, June 2021. https://doi.org/10.7249/RR- A894-1. · DOI
  39. Helmus, Todd C., James V. Marrone, Marek N. Posard, and Danielle Schlang. "Russian Propaganda Hits Its Mark: Experimentally Testing the Impact of Russian Propaganda and Counter-Interventions." RAND Corporation, October 2020. https://doi.org/10.7249/RRA704-3. · DOI
  40. Hernandez, Danny, and Tom B. Brown. "Measuring the Algorithmic Efficiency of Neural Networks." arxiv:2005.04305 [cs.LG], May 2020. https://doi.org/10.48550/arxiv.2005.04305. · DOI · Paper page
  41. Herriman, Maguire, Elana Meer, Roy Rosin, Vivian Lee, Vindell Washington, and Kevin G. Volpp. "Asked and Answered: Building a Chatbot to Address Covid-19-Related Concerns." NEJM Catalyst Innova- tions in Care Delivery, June 18, 2020. https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0230. https://foreignpolicy.com/2021/12/15/china-twitter-trolls-ccp-influence-operations-astroturfing/ https://foreignpolicy.com/2021/12/15/china-twitter-trolls-ccp-influence-operations-astroturfing/ https://www.brookings.edu/techstream/how-disinformation-evolved-in-2020/ https://www.deeplearningbook.org/ https://www.deeplearningbook.org/ https://graphika.com/reports/posing-as-patriots https://doi.org/10.1126/science.aau2706 https://purl.stanford.edu/tk756wp5109 https://stacks.stanford.edu/file/druid:bt877dz8024/20211202-tz-twitter-takedown.pdf https://stacks.stanford.edu/file/druid:bt877dz8024/20211202-tz-twitter-takedown.pdf https://stacks.stanford.edu/file/druid:hp643wc2962/twitter-SA-202009.pdf https://doi.org/10.1073/pnas.1920498117 https://www.technologyreview.com/2019/10/10/132667/the-biggest-threat-of-deepfakes-isnt-the-deepfakes-themselves/ https://www.technologyreview.com/2019/10/10/132667/the-biggest-threat-of-deepfakes-isnt-the-deepfakes-themselves/ https://blog.heim.xyz/palm-training-cost/ https://blog.heim.xyz/palm-training-cost/ https://doi.org/10.7249/RR-A894-1 https://doi.org/10.7249/RR-A894-1 https://doi.org/10.7249/RRA704-3 https://doi.org/10.48550/arxiv.2005.04305 https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0230 · DOI
  42. Holtzman, Ari, Jan Buys, Leo Du, Maxwell Forbes, and Yejin Choi. "The Curious Case of Neural Text Degeneration." arxiv:1904.09751 [cs.CL], February 19, 2019. ISSN: 16130073. https://doi.org/10. 48550/arxiv.1904.09751. · arXiv
  43. Hwang, Tim. Deepfakes: A Grounded Threat Assessment. Center for Security and Emerging Technology, July 2020. https://doi.org/10.51593/20190030. · DOI · Paper page
  44. Ippolito, Daphne, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. "Automatic Detection of Generated Text is Easiest when Humans are Fooled." arXiv:1911.00650 [cs.CL], July 2020, 1808– 1822. https://doi.org/10.48550/arXiv.1911.00650. · DOI · Paper page
  45. Jakesch, Maurice, Megan French, Xiao Ma, Jeffrey T. Hancock, and Mor Naaman. "AI-mediated commu- nication: How the perception that profile text was written by AI affects trustworthiness." CHI '19: Proceedings of CHI Conference on Human Factors in Computing Systems, May 2019. https://doi.org/ 10.1145/3290605.3300469. · DOI · Paper page
  46. Jeong, Se Hoon, Hyunyi Cho, and Yoori Hwang. "Media Literacy Interventions: A Meta-Analytic Review." Journal of Communication 62, no. 3 (June 2012): 454–472. ISSN: 0021-9916. https://doi.org/10. 1111/J.1460-2466.2012.01643.X.
  47. Johnson, Christian, and William Marcellino. Bad Actors in News Reporting: Tracking News Manipulation by State Actors. RAND Corporation, November 2021. https://doi.org/10.7249/RRA112-21. · DOI
  48. Jowett, Garth, and Victoria O'Donnell. Propaganda & Persuasion. 6th ed. SAGE Publications, 2014. ISBN: 1483323528.
  49. Kahembwe, Emmanuel, and Subramanian Ramamoorthy. "Lower Dimensional Kernels for Video Dis- criminators." Neural Networks 132 (December 2020): 506–520. https://doi.org/10.1016/j.neunet. 2020.09.016. · DOI
  50. Keskar, Nitish Shirish, Bryan Mccann, Lav R Varshney, Caiming Xiong, Richard Socher, and Salesforce Re- search. "CTRL: A Conditional Transformer Language Model for Controllable Generation." arxiv:1909.05858 [cs.CL], September 2019. https://doi.org/10.48550/arxiv.1909.05858. · DOI · Paper page
  51. Khan, Saif M., and Carrick Flynn. Maintaining China's Dependence on Democracies for Advanced Computer Chips. Center for Security and Emerging Technology, April 2020. https://cset.georgetown.edu/ publication/maintaining-chinas-dependence-on-democracies-for-advanced-computer-chips/. https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/ https://blog.twitter.com/en_us/topics/product/2017/an-update-on-safety https://blog.twitter.com/en_us/topics/product/2017/an-update-on-safety https://doi.org/10.48550/arxiv.1904.09751 https://doi.org/10.48550/arxiv.1904.09751 https://www.fast.ai/posts/2019-02-15-openai-gp2.html https://www.ggfutures.net/analysis/humor-over-rumor-combating-disinformation-around-covid-19-in-taiwan https://www.ggfutures.net/analysis/humor-over-rumor-combating-disinformation-around-covid-19-in-taiwan https://doi.org/10.51593/20190030 https://doi.org/10.48550/arXiv.1911.00650 https://doi.org/10.1145/3290605.3300469 https://doi.org/10.1145/3290605.3300469 https://doi.org/10.1111/J.1460-2466.2012.01643.X https://doi.org/10.1111/J.1460-2466.2012.01643.X https://doi.org/10.7249/RRA112-21 https://doi.org/10.1016/j.neunet.2020.09.016 https://doi.org/10.1016/j.neunet.2020.09.016 https://www.c4isrnet.com/opinion/2021/08/23/us-military-must-prepare-for-pow-concerns-in-the-deepfake-era/ https://www.c4isrnet.com/opinion/2021/08/23/us-military-must-prepare-for-pow-concerns-in-the-deepfake-era/ https://doi.org/10.48550/arxiv.1909.05858 https://cset.georgetown.edu/publication/maintaining-chinas-dependence-on-democracies-for-advanced-computer-chips/ https://cset.georgetown.edu/publication/maintaining-chinas-dependence-on-democracies-for-advanced-computer-chips/ · DOI · Paper page
  52. King, Gary, Jennifer Pan, and Margaret E. Roberts. "How the Chinese Government Fabricates Social Media Posts for Strategic Distraction, Not Engaged Argument." American Political Science Review 111, no. 3 (2017): 484–501. https://doi.org/10.1017/S0003055417000144. · DOI · Paper page
  53. Kreps, Sarah, R. Miles McCain, and Miles Brundage. "All the News That's Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation." Journal of Experimental Political Science 9, no. 1 (November 2022): 104–117. ISSN: 2052-2630. https://doi.org/10.1017/XPS.2020.37. · DOI · Paper page
  54. Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. "Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing." ACM Computing Surveys, September 2021. https://doi.org/10.1145/3560815. · DOI · Paper page
  55. Lohn, Andrew. Hacking AI: A Primer for Policymakers on Machine Learning Cybersecurity. Center for Se- curity and Emerging Technology, December 2020. https://doi.org/10.51593/2020CA006. · DOI
  56. Lohn, Andrew, and Micah Musser. AI and Compute: How Much Longer Can Computing Power Drive Ar- tificial Intelligence Progress? Center for Security and Emerging Technology, January 2022. https: //doi.org/10.51593/2021CA009. · DOI · Paper page
  57. Lohn, Andrew J., and Krystal A. Jackson. Will AI Make Cyber Swords or Shields? Center for Security and Emerging Technology, August 2022. https://doi.org/10.51593/2022CA002. https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6 https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6 https://doi.org/10.1017/S0003055417000144 https://www.nytimes.com/wirecutter/blog/state-of-privacy-laws-in-us/ https://www.nytimes.com/wirecutter/blog/state-of-privacy-laws-in-us/ https://www.wired.com/story/ai-powered-text-program-could-fool-government/ https://doi.org/10.1017/XPS.2020.37 https://screenrant.com/ai-deepfake-cloned-voice-bank-scam-theft-millions/ https://screenrant.com/ai-deepfake-cloned-voice-bank-scam-theft-millions/ https://thegradient.pub/gpt-4chan-lessons/ https://thegradient.pub/gpt-4chan-lessons/ https://openai.com/blog/critiques/ https://crfm.stanford.edu/2022/05/17/community-norms.html https://crfm.stanford.edu/2022/05/17/community-norms.html https://docs.google.com/forms/d/e/1FAIpQLSdh3Pgh0sGrYtRihBu-GPN7FSQoODBLvF7dVAFLZk2iuMgoLw/viewform?fbzx=1650213417672418119 https://docs.google.com/forms/d/e/1FAIpQLSdh3Pgh0sGrYtRihBu-GPN7FSQoODBLvF7dVAFLZk2iuMgoLw/viewform?fbzx=1650213417672418119 https://doi.org/10.1145/3560815 https://doi.org/10.51593/2020CA006 https://doi.org/10.51593/2021CA009 https://doi.org/10.51593/2021CA009 https://doi.org/10.51593/2022CA002 · DOI
  58. Loureiro, Daniel, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-Collados. "TimeLMs: Diachronic Language Models from Twitter." arxiv.2202.03829 [cs.CL], February 2022, 251–260. https://doi.org/10.48550/arxiv.2202.03829. · DOI
  59. Mansimov, Elman, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. "Generating Images from Captions with Attention." 4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings, November 9, 2015. https://doi.org/10.48550/arxiv.1511.02793. · DOI
  60. Mazarr, Michael J., Ryan Michael Bauer, Abigail Casey, Sarah Anita Heintz, and Luke J. Matthews. The Emerging Risk of Virtual Societal Warfare: Social Manipulation in a Changing Information Environ- ment. RAND Corporation, October 2019. https://doi.org/10.7249/RR2714. · DOI
  61. Merton, Robert K., and Norman W. Storer. The Sociology of Science: Theoretical and Empirical Investiga- tions. Univ. of Chicago Press, 1973.
  62. Mooney, Austin. "Spotlight On Sensitive Personal Data As Foreign Investment Rules Take Force." National Law Review 11, no. 163 (February 18, 2020). https://www.natlawreview.com/article/spotlight- sensitive-personal-data-foreign-investment-rules-take-force.
  63. Mu, Zhaoxi, Xinyu Yang, and Yizhuo Dong. "Review of end-to-end speech synthesis technology based on deep learning." arxiv:2104.09995 [cs.SD], April 2021. https://doi.org/10.48550/arxiv.2104. 09995. · DOI
  64. Nakano, Reiichiro, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, et al. "WebGPT: Browser-assisted question-answering with human feedback." arxiv:2112.09332 [cs.CL], June 1, 2022. https://doi.org/10.48550/arxiv.2112.09332. · DOI · Paper page
  65. "NAVER Unveils HyperCLOVA, Korea's First Hyperscale 'Al to Empower Everyone'." Naver Corp. Press Releases, May 25, 2021. https://www.navercorp.com/en/promotion/pressReleasesView/30686.
  66. Nightingale, Sophie J., and Hany Farid. "AI-synthesized faces are indistinguishable from real faces and more trustworthy." PNAS 119, no. 8 (February 2022). ISSN: 10916490. https://doi.org/10.1073/ PNAS.2120481119.
  67. Ovadya, Aviv, and Jess Whittlestone. "Reducing malicious use of synthetic media research: Considera- tions and potential release practices for machine learning." arxiv:1907.11274 [cs.CY], July 2019. https://doi.org/10.48550/arxiv.1907.11274. · DOI · Paper page
  68. Pajola, Luca, and Mauro Conti. "Fall of Giants: How popular text-based MLaaS fall against a simple evasion attack." Proceedings - 2021 IEEE European Symposium on Security and Privacy, Euro S and P 2021, April 2021, 198–211. https://doi.org/10.48550/arxiv.2104.05996. https://doi.org/10.48550/arxiv.2112.09332 https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/ https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/ https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/ https://www.navercorp.com/en/promotion/pressReleasesView/30686 https://doi.org/10.1073/PNAS.2120481119 https://doi.org/10.1073/PNAS.2120481119 https://www.brookings.edu/research/the-breakout-scale-measuring-the-impact-of-influence-operations/ https://www.brookings.edu/research/the-breakout-scale-measuring-the-impact-of-influence-operations/ https://www.approximatelycorrect.com/2019/02/17/openai-trains-language-model-mass-hysteria-ensues/ https://www.approximatelycorrect.com/2019/02/17/openai-trains-language-model-mass-hysteria-ensues/ https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/MODEL_LICENSE.md https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/MODEL_LICENSE.md https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf https://aviv.medium.com/contextualization-engines-can-fight-misinformation-without-censorship-c5c47222a3b7 https://aviv.medium.com/contextualization-engines-can-fight-misinformation-without-censorship-c5c47222a3b7 https://doi.org/10.48550/arxiv.1907.11274 https://doi.org/10.48550/arxiv.2104.05996 · DOI
  69. Papernot, Nicolas, Ian Goodfellow, Martín Abadi, Kunal Talwar, and Úlfar Erlingsson. "Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data." 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, October 2016. https://doi.org/ 10.48550/arxiv.1610.05755. · DOI · Paper page
  70. Park, Hee Sun, Timothy R. Levine, Catherine Y.Kingsley Westerman, Tierney Orfgen, and Sarah Foregger. "The Effects of Argument Quality and Involvement Type on Attitude Formation and Attitude Change: A Test of Dual-Process and Social Judgment Predictions." Human Communication Research 33, no. 1 (January 2007): 81–102. ISSN: 0360-3989. https://doi.org/10.1111/J.1468-2958.2007.00290.X. · DOI
  71. Percy, Sarah. Mercenaries: The History of a Norm in International Relations. 1–280. Oxford University Press, October 2007. ISBN: 9780191706608.
  72. Ramesh, Aditya, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. "Hierarchical Text-Conditional Image Generation with CLIP Latents." arxiv:2204.06125 [cs.CV], April 2022. https://doi.org/10. 48550/arxiv.2204.06125. · arXiv
  73. Rashkin, Hannah, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. "Measuring Attribution in Natural Lan- guage Generation Models." arxiv:2112.12870 [cs.CL], August 2, 2022. https://doi.org/10.48550/ arxiv.2112.12870. · arXiv
  74. "Responsible AI Licenses (RAIL)." Responsible AI Licenses (RAIL). Accessed September 14, 2022. https: //www.licenses.ai/. https://doi.org/10.48550/arxiv.1610.05755 https://doi.org/10.48550/arxiv.1610.05755 https://doi.org/10.1111/J.1468-2958.2007.00290.X https://partnershiponai.org/ https://doi.org/10.1038/s41586-021-03344-2 https://www.pewresearch.org/internet/2017/11/29/public-comments-to-the-federal-communications-commission-about-net-neutrality-contain-many-inaccuracies-and-duplicates/ https://www.pewresearch.org/internet/2017/11/29/public-comments-to-the-federal-communications-commission-about-net-neutrality-contain-many-inaccuracies-and-duplicates/ https://www.pewresearch.org/internet/2017/11/29/public-comments-to-the-federal-communications-commission-about-net-neutrality-contain-many-inaccuracies-and-duplicates/ https://www.state.gov/russias-pillars-of-disinformation-and-propaganda-report/ https://blog.eprivacy.eu/?p=544 https://blog.eprivacy.eu/?p=544 https://docs.cohere.ai/docs/prompt-engineering https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/ https://doi.org/10.48550/arxiv.2204.06125 https://doi.org/10.48550/arxiv.2204.06125 https://doi.org/10.48550/arxiv.2112.12870 https://doi.org/10.48550/arxiv.2112.12870 https://www.thedailybeast.com/right-wing-media-outlets-duped-by-a-middle-east-propaganda-campaign https://www.thedailybeast.com/right-wing-media-outlets-duped-by-a-middle-east-propaganda-campaign https://eshoo.house.gov/sites/eshoo.house.gov/files/9.20.22LettertoNSCandOSTPonStabilityAI.pdf https://eshoo.house.gov/sites/eshoo.house.gov/files/9.20.22LettertoNSCandOSTPonStabilityAI.pdf https://www.licenses.ai/ https://www.licenses.ai/ · DOI · Paper page
  75. Riedl, Martin J., Sharon Strover, Tiancheng Cao, Jaewon R. Choi, Brad Limov, and Mackenzie Schnell. "Reverse-engineering political protest: the Russian Internet Research Agency in the Heart of Texas." Information, Communication, and Society 25, no. 15 (2021). ISSN: 14684462. https://doi.org/10. 1080/1369118X.2021.1934066.
  76. Roozenbeek, Jon, Sander van der Linden, and Thomas Nygren. "Prebunking interventions based on "inoculation" theory can reduce susceptibility to misinformation across cultures." Harvard Kennedy School Misinformation Review 1, no. 2 (February 2020). https://doi.org/10.37016//MR-2020-008. · DOI
  77. Sablayrolles, Alexandre, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. "Radioactive data: tracing through training." 37th International Conference on Machine Learning, ICML 2020 PartF168147-11 (February 3, 2020): 8296–8305. https://doi.org/10.48550/arxiv.2002.00937. · DOI
  78. Schwartz, Oscar. "In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation." IEEE Spectrum, November 25, 2019. https://spectrum.ieee.org/in-2016-microsofts-racist-chatbot- revealed-the-dangers-of-online-conversation.
  79. Sedova, Katerina, Christine McNeill, Aurora Johnson, Aditi Joshi, and Ido Wulkan. AI and the Future of Disinformation Campaigns: Part 2: A Threat Model. Center for Security and Emerging Technology, December 2021. https://doi.org/10.51593/2021CA011. · DOI
  80. Seger, Elizabeth, Shahar Avin, Gavin Pearson, Mark Briers, Seán Ó Heigeartaigh, and Helena Bacon. Tackling threats to informed decision-making in democratic societies: Promoting epistemic security in a technologically-advanced world. The Alan Turing Institute, October 14, 2020. https://doi.org/10. 17863/CAM.64183. https://us.macmillan.com/books/9780374287269/activemeasures https://doi.org/10.1080/1369118X.2021.1934066 https://doi.org/10.1080/1369118X.2021.1934066 https://doi.org/10.37016//MR-2020-008 https://ruder.io/recent-advances-lm-fine-tuning/ https://doi.org/10.48550/arxiv.2002.00937 https://imagen.research.google/ https://crsreports.congress.gov https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf#page=52 https://www.nscai.gov/wp-content/uploads/2021/03/Full-Report-Digital-1.pdf#page=52 https://www.chinatalk.media/p/new-chip-export-controls-explained https://www.chinatalk.media/p/new-chip-export-controls-explained https://www.lawfareblog.com/toward-information-operations-kill-chain https://www.lawfareblog.com/toward-information-operations-kill-chain https://ndupress.ndu.edu/Portals/68/Documents/stratperspective/inss/Strategic-Perspectives-11.pdf https://ndupress.ndu.edu/Portals/68/Documents/stratperspective/inss/Strategic-Perspectives-11.pdf https://spectrum.ieee.org/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation https://spectrum.ieee.org/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation https://doi.org/10.51593/2021CA011 https://doi.org/10.17863/CAM.64183 https://doi.org/10.17863/CAM.64183 · DOI
  81. Sevilla, Jaime, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. "Compute Trends Across Three Eras of Machine Learning." Proceedings of the International Joint Conference on Neural Networks, March 9, 2022. https://doi.org/10.48550/arxiv.2202.05924. · DOI
  82. Shannon, Vaughn P. "Norms Are What States Make of Them: The Political Psychology of Norm Violation." International Studies Quarterly 44, no. 2 (June 2000): 293–316. ISSN: 0020-8833. https://doi.org/ 10.1111/0020-8833.00159. · DOI
  83. Shazeer, Noam, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, Jan- uary 2017. https://doi.org/10.48550/arxiv.1701.06538. · DOI · Paper page
  84. Shokri, Reza, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. "Membership Inference Attacks against Machine Learning Models." Proceedings - IEEE Symposium on Security and Privacy, October 2016, 3–18. ISSN: 10816011. https://doi.org/10.48550/arxiv.1610.05820. · DOI
  85. Solaiman, Irene, Miles Brundage, Openai Jack, Clark Openai, Amanda Askell Openai, Ariel Herbert-Voss, Jeff Wu Openai, et al. "Release Strategies and the Social Impacts of Language Models." arxiv:1908.09203 [cs.CL], August 2019. https://doi.org/10.48550/arxiv.1908.09203. · DOI · Paper page
  86. Starbird, Kate, Ahmer Arif, and Tom Wilson. "Disinformation as collaborative work: Surfacing the partic- ipatory nature of strategic information operations." Proceedings of the ACM on Human-Computer In- teraction Vol: CSCW, Article 127, CSCW 2019. ISSN: 25730142. https://doi.org/10.1145/3359229. · DOI · Paper page
  87. Sunstein, Cass R. "Social Norms and Social Roles." Columbia Law Review 44 (1996): 909. https :// chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=12456&context=journal_articles. https://www.intelligence.senate.gov/sites/default/files/documents/Report_Volume2.pdf https://www.intelligence.senate.gov/sites/default/files/documents/Report_Volume2.pdf https://doi.org/10.48550/arxiv.2202.05924 https://docs.google.com/spreadsheets/d/1AAIebjNsnJj_uKALHbXNfn3_YsT6sHXtCU0q7OIPuc4/edit#gid=0 https://docs.google.com/spreadsheets/d/1AAIebjNsnJj_uKALHbXNfn3_YsT6sHXtCU0q7OIPuc4/edit#gid=0 https://doi.org/10.1111/0020-8833.00159 https://doi.org/10.1111/0020-8833.00159 https://doi.org/10.48550/arxiv.1701.06538 https://doi.org/10.48550/arxiv.1610.05820 https://commoncrawl.org/the-data/get-started/ https://commoncrawl.org/the-data/get-started/ https://doi.org/10.48550/arxiv.1908.09203 https://spotthetroll.org/ https://doi.org/10.1145/3359229 http://gltr.io/ https://www.reuters.com/article/usa-election-russia-disinformation/exclusive-russian-operation-masqueraded-as-right-wing-news-site-to-target-u-s-voters-sources-idUSKBN26M5OP https://www.reuters.com/article/usa-election-russia-disinformation/exclusive-russian-operation-masqueraded-as-right-wing-news-site-to-target-u-s-voters-sources-idUSKBN26M5OP https://www.reuters.com/article/usa-election-russia-disinformation/exclusive-russian-operation-masqueraded-as-right-wing-news-site-to-target-u-s-voters-sources-idUSKBN26M5OP https://www.reuters.com/article/us-facebook-disinformation-brazil/facebook-suspends-disinformation-network-tied-to-staff-of-brazils-bolsonaro-idUSKBN2492Y5 https://www.reuters.com/article/us-facebook-disinformation-brazil/facebook-suspends-disinformation-network-tied-to-staff-of-brazils-bolsonaro-idUSKBN2492Y5 https://www.reuters.com/article/us-facebook-disinformation-brazil/facebook-suspends-disinformation-network-tied-to-staff-of-brazils-bolsonaro-idUSKBN2492Y5 https://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=12456&context=journal_articles https://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=12456&context=journal_articles · DOI
  88. Sutskever, Ilya, James Martens, and Geoffrey Hinton. "Generating Text with Recurrent Neural Networks." Edited by Lisa Gooter and Tobias Scheffer. Proceedings of the 28th International Conference on Ma- chine Learning, 2011. https://icml.cc/2011/papers/524_icmlpaper.pdf.
  89. Tannenwald, Nina. "The Nuclear Taboo: The United States and the Normative Basis of Nuclear Non-Use." International Organization 53, no. 3 (1999): 433–468. https://www.jstor.org/stable/2601286.
  90. Taylor, Philip M. Munitions of the mind: a history of propaganda from the ancient world to the present era. Manchester University Press, 2003. ISBN: 978-1-84779-092-7.
  91. Ternovski, John, Joshua Kalla, and Peter Aronow. "The Negative Consequences of Informing Voters about Deepfakes: Evidence from Two Survey Experiments." Journal of Online Trust and Safety 1, no. 2 (February 2022). ISSN: 2770-3142. https://doi.org/10.54501/JOTS.V1I2.28. · DOI
  92. Tramer, Florian, Fan Zheng, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. "Stealing Machine Learning Models via Prediction APIs." 25th USENIX Security Symposium (Austin, TX; USENIX Security 16), 2016, 601–618. https://www.usenix.org/conference/usenixsecurity16/technical-sessions/ presentation/tramer.
  93. Verdoliva, Luisa. "Media Forensics and DeepFakes: An Overview." IEEE Journal on Selected Topics in Signal Processing 14, no. 5 (January 2020): 910–932. ISSN: 19410484. https://doi.org/10.1109/JSTSP. 2020.3002101. https://icml.cc/2011/papers/524_icmlpaper.pdf https://www.jstor.org/stable/2601286 https://doi.org/10.54501/JOTS.V1I2.28 https://www.proofofhumanity.id/ https://www.proofofhumanity.id/ https://fsi.stanford.edu/publication/contours-and-controversies-parler https://fsi.stanford.edu/publication/contours-and-controversies-parler https://cyber.fsi.stanford.edu/publication/gabufacturing-dissent-an-in-depth-analysis-of-gab https://cyber.fsi.stanford.edu/publication/gabufacturing-dissent-an-in-depth-analysis-of-gab https://lab.witness.org/ticks-or-it-didnt-happen/ https://lab.witness.org/ticks-or-it-didnt-happen/ https://www.washingtonpost.com/technology/2022/06/11/google-ai-lamda-blake-lemoine/ https://www.washingtonpost.com/technology/2022/06/11/google-ai-lamda-blake-lemoine/ https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer https://www.gao.gov/products/gao-21-103181 https://tinyurl.com/5bamt57s https://tinyurl.com/5bamt57s https://www.supremecourt.gov/opinions/20pdf/19-783_k53l.pdf https://www.supremecourt.gov/opinions/20pdf/19-783_k53l.pdf https://www.mosaicml.com/blog/gpt-3-quality-for-500k https://doi.org/10.1109/JSTSP.2020.3002101 https://doi.org/10.1109/JSTSP.2020.3002101 · DOI
  94. Wallace, Eric, Tony Z. Zhao, Shi Feng, and Sameer Singh. "Concealed Data Poisoning Attacks on NLP Models." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2021, 139–150. https://doi.org/ 10.48550/arxiv.2010.12563. · DOI
  95. Wanless, Alicia, and James Pamment. "How Do You Define a Problem Like Influence?" Journal of Infor- mation Warfare 18, no. 3 (2019): 1–14. https://www.jstor.org/stable/26894679.
  96. Wei, Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, et al. "Emergent Abilities of Large Language Models." arxiv:2206.07682 [cs.CL], June 2022. https: //doi.org/10.48550/arxiv.2206.07682. · DOI · Paper page
  97. Wu, Tongshuang, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J. Cai. "PromptChainer: Chaining Large Language Model Prompts through Visual Programming." Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, April 2022. https://doi.org/10.1145/3491101.3519729. · DOI · Paper page
  98. Wu, Xingjiao, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. "A Survey of Human- in-the-loop for Machine Learning." Future Generation Computer Systems 135 (August 2021): 364– https://doi.org/10.1016/j.future.2022.05.014. · DOI · Paper page
  99. Xiang, Tao, Chunlong Xie, Shangwei Guo, Jiwei Li, and Tianwei Zhang. "Protecting Your NLG Models with Semantic and Robust Watermarks." arxiv:2112.05428 [cs.MM], December 10, 2021. https: //doi.org/10.48550/arxiv.2112.05428. https://www.w3.org/TR/vc-data-model/ https://www.w3.org/TR/vc-data-model/ https://www.theverge.com/2022/6/8/23159465/youtuber-ai-bot-pol-gpt-4chan-yannic-kilcher-ethics https://www.theverge.com/2022/6/8/23159465/youtuber-ai-bot-pol-gpt-4chan-yannic-kilcher-ethics https://doi.org/10.48550/arxiv.2010.12563 https://doi.org/10.48550/arxiv.2010.12563 https://www.axios.com/2021/11/18/openai-gpt-3-waiting-list-api https://www.axios.com/2021/11/18/openai-gpt-3-waiting-list-api https://www.jstor.org/stable/26894679 https://www.lawfareblog.com/media-has-overcorrected-foreign-influence https://www.lawfareblog.com/media-has-overcorrected-foreign-influence https://www.nytimes.com/2019/08/14/opinion/deepfakes-adele-disinformation.html https://www.nytimes.com/2019/08/14/opinion/deepfakes-adele-disinformation.html https://doi.org/10.48550/arxiv.2206.07682 https://doi.org/10.48550/arxiv.2206.07682 https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/ https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/ https://m.facebook.com/help/1053588012132894 https://www.ai21.com/blog/announcing-ai21-studio-and-jurassic-1 https://venturebeat.com/ai/huawei-trained-the-chinese-language-equivalent-of-gpt-3/ https://venturebeat.com/ai/huawei-trained-the-chinese-language-equivalent-of-gpt-3/ https://doi.org/10.1145/3491101.3519729 https://doi.org/10.1016/j.future.2022.05.014 https://doi.org/10.48550/arxiv.2112.05428 https://doi.org/10.48550/arxiv.2112.05428 · DOI
  100. Zeng, Wei, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, et al. "PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation." arxiv:2104.12369 [cs.CL], April 2021. https://doi.org/10.48550/arxiv.2104.12369. https://parti.research.google/ https://doi.org/10.48550/arxiv.2104.12369 · DOI · Paper page

Cited by

Sorted by citations

Authors
Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., & Sedova, K.
Affiliations
  • Georgetown University (US)
  • OpenAI (United States) (US)
  • Stanford Internet Observatory
  • Georgetown's Center for Security and Emerging Technology
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2301.04246
arXiv
2301.04246
OpenAlex
W4315881236
Cited by
152
References
0
Citations by year
2026: 9 · 2025: 32 · 2024: 49 · 2023: 60 · 2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
generative language models, influence operations, disinformation, mitigation measures, belief formation, content dissemination, online manipulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review