CoChat
Sign Up
PreprintCited by 196Authorship Attribution and ProfilingDigital Media Forensic DetectionSpam and Phishing Detection

Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

Open access
Crothers, Evan, Japkowicz, Nathalie, Viktor, Herna·2022·arXiv
DOI 10.48550/arxiv.2210.07321arXiv 2210.07321
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine generated text is increasingly difficult to distinguish from human authored text. Powerful open-source models are freely available, and user-friendly tools that democratize access to generative models are proliferating. ChatGPT, which was released shortly after the first edition of this survey, epitomizes these trends. The great potential of state-of-the-art natural language generation (NLG) systems is tempered by the multitude of avenues for abuse. Detection of machine generated text is a key countermeasure for reducing abuse of NLG models, with significant technical challenges and numerous open problems. We provide a survey that includes both 1) an extensive analysis of threat models posed by contemporary NLG systems, and 2) the most complete review of machine generated text detection methods to date. This survey places machine generated text within its cybersecurity and social context, and provides strong guidance for future work addressing the most critical threat models, and ensuring detection systems themselves demonstrate trustworthiness through fairness, robustness, and accountability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.07321

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Machine-generated content has long been implicated in cyber-social threats, such as phishing, mis/disinformation, and harmful content.

    “Machine-generated content has long been implicated in cyber-social threats, such as phishing, mis/disinformation, and harmful content (Crothers et al., 2022).”

    From Anatomy of an AI-powered malicious social botnet (2024) · cites this paper as Crothers et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 164 · Cited by 196

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 89

  1. Open Cover Letter, "Open cover letter: Generate cover letters with ai," 2022.
  2. A. Kanapala, S. Pal, and R. Pamula, "Text summarization from legal documents: a survey," Artificial Intelligence Review, vol. 51, no. 3, pp. 371-402, 2019.
  3. M. Wang, M. Wang, F. Yu, Y. Yang, J. Walker, and J. Mostafa, "A system- atic review of automatic text summarization for biomedical literature and ehrs," Journal of the American Medical Informatics Association, vol. 28, no. 10, pp. 2287-2297, 2021.
  4. S. Kim, J. Lee, and G. Gweon, "Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality," in Proc. 2019 CHI conference on human factors in computing systems, pp. 1-12, 2019.
  5. A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero-shot text-to-image generation," in ICML, pp. 8821-8831, PMLR, 2021.
  6. G. Jawahar, M. Abdul-Mageed, and V. Laks Lakshmanan, "Automatic detection of machine generated text: A critical survey," in Proc. 28th International Conference on Computational Linguistics, pp. 2296-2309, 2020.
  7. D. Beresneva, "Computer-generated text detection using machine learn- ing: A systematic review," in International Conference on Applications of Natural Language to Information Systems, pp. 421-426, Springer, 2016.
  8. H.-Q. Nguyen-Son, N.-D. T. Tieu, H. H. Nguyen, J. Yamagishi, and I. E. Zen, "Identifying computer-generated text using statistical analysis," in 2017 APSIPA ASC, pp. 1504-1511, 2017.
  9. L. Fröhling and A. Zubiaga, "Feature-based detection of automated language models: tackling gpt-2, gpt-3 and grover," PeerJ Computer Science, vol. 7, p. e443, 2021.
  10. D. Kaur, S. Uslu, K. J. Rittichier, and A. Durresi, "Trustworthy artificial intelligence: A review," ACM Comput. Surv., vol. 55, jan 2022.
  11. C. Dong, Y. Li, H. Gong, M. Chen, J. Li, Y. Shen, and M. Yang, "A survey of natural language generation," ACM Comput. Surv., jul 2022. Just Accepted.
  12. A. Gatt and E. Krahmer, "Survey of the state of the art in natural language generation: Core tasks, applications and evaluation," Journal of Artificial Intelligence Research, vol. 61, pp. 65-170, 2018.
  13. R. Perera and P. Nand, "Recent advances in natural language generation: A survey and classification of the empirical literature," Computing and Informatics, vol. 36, no. 1, pp. 1-32, 2017.
  14. J. Li, T. Tang, W. X. Zhao, and J.-R. Wen, "Pretrained language model for text generation: A survey," in Proc. 30th IJCAI (Z.-H. Zhou, ed.), pp. 4492-4499, IJCAI Organization, 8 2021. Survey Track.
  15. E. Reiter and R. Dale, "Building applied natural language generation systems," Natural Language Engineering, vol. 3, 03 2002.
  16. J. Lyons, Natural Language and Universal Grammar: Volume 1: Essays in Linguistic Theory, vol. 1. Cambridge University Press, 1991.
  17. L. Zhang and J.-T. Sun, Text Generation, pp. 3048-3051. Boston, MA: Springer US, 2009.
  18. S. Biderman and E. Raff, "Fooling moss detection with pretrained lan- guage models," in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 2933-2943, 2022.
  19. W. Wang, B. Tang, R. Wang, L. Wang, and A. Ye, "A survey on adversarial attacks and defenses in text," CoRR, vol. abs/1902.07285, 2019. · arXiv
  20. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," J. of Mach. Learn. Res., vol. 21, no. 140, pp. 1-67, 2020.
  21. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, "fairseq: A fast, extensible toolkit for sequence modeling," in Proc. of NAACL-HLT 2019: Demo., 2019.
  22. F. A. Sheikha and D. Inkpen, "Generation of formal and informal sentences," in Proc. 13th European Workshop on Natural Language Generation, pp. 187-193, 2011.
  23. A. Sudhakar, B. Upadhyay, and A. Maheswaran, ""transforming" delete, retrieve, generate approach for controlled text style transfer," in Proc. 2019 Conf. EMNLP-IJCNLP, pp. 3269-3279, 2019.
  24. A. Aghajanyan, A. Shrivastava, A. Gupta, N. Goyal, L. Zettlemoyer, and S. Gupta, "Better fine-tuning by reducing representational collapse," in ICLR, 2021.
  25. H. P. Luhn, "The automatic creation of literature abstracts," IBM Journal of research and development, vol. 2, no. 2, pp. 159-165, 1958.
  26. H. Harkous, I. Groves, and A. Saffari, "Have your text and use it too! end- to-end neural data-to-text generation with semantic fidelity," in COLING 2020, 2020.
  27. L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, "Unified vision-language pre-training for image captioning and vqa," in Proc. AAAI Conference on Artificial Intelligence, vol. 34, pp. 13041-13049, 2020.
  28. P. A. Duboue and K. R. McKeown, "Statistical acquisition of content selection rules for natural language generation," in Proc. 2003 Confer- ence on Empirical Methods in Natural Language Processing, pp. 121- 128, 2003.
  29. I. Langkilde and K. Knight, "Generation that exploits corpus-based sta- tistical knowledge," in COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics, 1998.
  30. R. Kondadadi, B. Howald, and F. Schilder, "A statistical nlg framework for aggregated planning and realization," in Proc. 51st Annual Meeting of the ACL (Volume 1: Long Papers), pp. 1406-1415, 2013.
  31. N. Dethlefs and H. Cuayáhuitl, "Hierarchical reinforcement learning for adaptive text generation," in Proc. 6th INLG, ACL, July 2010.
  32. T. Mikolov et al., "Stat. lang. models based on neural networks," Presen- tation at Google, Mountain View, 2nd April, vol. 80, no. 26, 2012.
  33. M. Berglund, T. Raiko, M. Honkala, L. Kärkkäinen, A. Vetek, and J. Karhunen, "Bidirectional recurrent neural networks as generative mod- els," in NIPS, 2015.
  34. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension," in Proc. 58th Annu. Meet. ACL Sys. Demo., (Online), pp. 7871-7880, ACL, July 2020.
  35. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, "Mass: Masked sequence to sequence pre-training for language generation," in ICML, pp. 5926- 5936, 2019.
  36. S. Bromander, A. Jøsang, and M. Eian, "Semantic cyberthreat mod- elling.," in STIDS, pp. 74-78, 2016.
  37. L. Kohnfelder and P. Garg, "The threats to our products," Microsoft Interface, Microsoft Corporation, vol. 33, 1999.
  38. T. UcedaVelez and M. M. Morana, Risk Centric Threat Modeling: process for attack simulation and threat analysis. John Wiley & Sons, 2015.
  39. A. Shostack, Threat modeling: Designing for security. John Wiley & Sons, 2014.
  40. K. L. Chiew, K. S. C. Yong, and C. L. Tan, "A survey of phishing attacks: Their types, vectors and technical approaches," Expert Systems with Applications, vol. 106, pp. 1-20, 2018.
  41. Z. Alkhalil, C. Hewage, L. Nawaf, and I. Khan, "Phishing attacks: A recent comprehensive study and a new anatomy," Frontiers in Computer Science, vol. 3, 2021.
  42. A. J. Burns, M. E. Johnson, and D. D. Caputo, "Spear phishing in a barrel: Insights from a targeted phishing campaign," J. Organ. Comput. Electron. Commer., vol. 29, no. 1, pp. 24-39, 2019.
  43. S. Abu-Nimeh, D. Nappa, X. Wang, and S. Nair, "A comparison of machine learning techniques for phishing detection," in Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit, pp. 60-69, 2007.
  44. R. Schuster, C. Song, E. Tromer, and V. Shmatikov, "You autocom- plete me: Poisoning vulnerabilities in neural code completion," in 30th USENIX Security Symposium (USENIX Security 21), pp. 1559-1575, 2021.
  45. P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin, "Generating fake cyber threat intelligence using transformer-based models," in 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9, 2021.
  46. N. Baracaldo, B. Chen, H. Ludwig, and J. A. Safavi, "Mitigating poi- soning attacks on machine learning models: A data provenance based approach," in Proc. 10th ACM Workshop on Artificial Intelligence and Security, pp. 103-110, 2017.
  47. M. Gabielkov, A. Ramachandran, A. Chaintreau, and A. Legout, "Social Clicks: What and Who Gets Read on Twitter?," in ACM SIGMETRICS / IFIP Performance 2016, (Antibes Juan-les-Pins, France), June 2016.
  48. American Press Institute, "How americans get their news," Mar 2014.
  49. J. Mink, L. Luo, N. M. Barbosa, O. Figueira, Y. Wang, and G. Wang, "DeepPhish: Understanding user trust towards artificially generated pro- files in online social networks," in 31st USENIX Security Symposium (USENIX Security 22), (Boston, MA), pp. 1669-1686, USENIX Associ- ation, Aug. 2022.
  50. E. Crothers, N. Japkowicz, and H. L. Viktor, "Towards ethical content- based detection of online influence campaigns," in 2019 IEEE 29th Inter- national Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6, IEEE, 2019.
  51. C. Labbé and D. Labbé, "Duplicate and fake publications in the scientific literature: how many scigen papers in computer science?," Scientomet- rics, vol. 94, no. 1, pp. 379-396, 2013.
  52. G. Cabanac and C. Labbé, "Prevalence of nonsensical algorithmically generated papers in the scientific literature," Journal of the Association for Information Science and Technology, vol. 72, no. 12, pp. 1461-1476, 2021.
  53. Z. Zhuang, E. Elmacioglu, D. Lee, and C. L. Giles, "Measuring confer- ence quality by mining program committee characteristics," in Proc. 7th JCDL, JCDL '07, (New York, NY, USA), p. 225-234, ACM, 2007.
  54. Harth, V. Presutti, R. Troncy, M. Acosta, A. Polleres, J. D. Fernández, J. Xavier Parreira, O. Hartig, K. Hose, and M. Cochez, eds.), (Cham), pp. 158-163, Springer International Publishing, 2020.
  55. J. Rodriguez, T. Hay, D. Gros, Z. Shamsi, and R. Srinivasan, "Cross- domain detection of GPT-2-generated technical text," in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (Seattle, United States), pp. 1213-1233, Association for Computational Linguis- tics, July 2022.
  56. B. Lufkin, "Why do cover letters still exist?," Oct 2021.
  57. D. Ippolito, D. Duckworth, C. Callison-Burch, and D. Eck, "Automatic detection of generated text is easiest when humans are fooled," in ACL, 2020.
  58. A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, and J. Vertesi, "Fairness and abstraction in sociotechnical systems," in Proc. Conference on Fairness, Accountability, and Transparency, FAT* '19, (New York, NY, USA), p. 59-68, ACM, 2019.
  59. L. v. Ahn, M. Blum, N. J. Hopper, and J. Langford, "Captcha: Using hard ai problems for security," in International conference on the theory and applications of cryptographic techniques, pp. 294-311, Springer, 2003.
  60. M. Guerar, L. Verderame, M. Migliardi, F. Palmieri, and A. Merlo, "Gotta captcha 'em all: A survey of 20 years of the human-or-computer dilemma," ACM Comput. Surv., vol. 54, oct 2021.
  61. S. Munir, B. Batool, Z. Shafiq, P. Srinivasan, and F. Zaffar, "Through the looking glass: Learning to attribute synthetic text generated by language models," in Proc. 16th Conf. Euro. Chap. ACL: Main Vol., (Online), pp. 1811-1822, ACL, Apr. 2021.
  62. A. Uchendu, T. Le, K. Shu, and D. Lee, "Authorship attribution for neural text generation," in Proc. 2020 Conf. EMNLP, (Online), pp. 8384-8395, ACL, Nov. 2020.
  63. S. Gehrmann, H. Strobelt, and A. Rush, "GLTR: Statistical detection and visualization of generated text," in Proc. 57th Annual Meeting of the ACL: System Demonstrations, (Florence, Italy), pp. 111-116, ACL, July 2019.
  64. A. See, A. Pappu, R. Saxena, A. Yerukola, and C. D. Manning, "Do massively pretrained language models make better storytellers?," in Pro- ceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 843-861, 2019.
  65. X. Chen, P. Jin, S. Jing, and C. Xie, "Automatic detection of chinese generated essayss based on pre-trained bert," in 2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Confer- ence (ITAIC), vol. 10, pp. 2257-2260, IEEE, 2022.
  66. Y. Cui, W. Che, T. Liu, B. Qin, and Z. Yang, "Pre-training with whole word masking for chinese bert," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3504-3514, 2021.
  67. W. Antoun, F. Baly, and H. M. Hajj, "Arabert: Transformer-based model for arabic language understanding," ArXiv, 2020.
  68. S. Dadas, M. Perełkiewicz, and R. Poświata, "Pre-training polish transformer-based language models at scale," in International Conference on Artificial Intelligence and Soft Computing, pp. 301-314, Springer, 2020.
  69. Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang, "A tutorial on energy-based learning," Predicting structured data, vol. 1, no. 0, 2006.
  70. A. Bakhtin, Y. Deng, S. Gross, M. Ott, M. Ranzato, and A. Szlam, "Residual energy-based models for text," J. of Mach. Learn. Res., vol. 22, pp. 40-1, 2021.
  71. T. Fagni, F. Falchi, M. Gambini, A. Martella, and M. Tesconi, "Tweep- fake: About detecting deepfake tweets," PLOS ONE, vol. 16, pp. 1-16, 05 2021.
  72. S. G. Tesfagergish, R. Damaševičius, and J. Kapočiūtė-Dzikienė, "Deep fake recognition in tweets using text augmentation, word embeddings and deep learning," in International Conference on Computational Science and Its Applications, pp. 523-538, Springer, 2021.
  73. J. Tourille, B. Sow, and A. Popescu, "Automatic detection of bot- generated tweets," in Proc. 1st International Workshop on Multimedia AI against Disinformation, MAD '22, (New York, NY, USA), p. 44-51, ACM, 2022.
  74. M. Latah, "Detection of malicious social bots: A survey and a refined taxonomy," Expert Systems with Applications, vol. 151, p. 113383, 2020.
  75. J. Salminen, C. Kandpal, A. M. Kamel, S. gyo Jung, and B. J. Jansen, "Creating and detecting fake reviews of online products," Journal of Retailing and Consumer Services, vol. 64, p. 102771, 2022.
  76. M. M. Bhat and S. Parthasarathy, "How effectively can machines defend against machine-generated fake news? an empirical study," in Proc. First Workshop on Insights from Negative Results in NLP, (Online), pp. 48- 53, ACL, Nov. 2020.
  77. E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith, "All that's 'human' is not gold: Evaluating human evaluation of generated text," in Proc. 59th Annual Meeting of the ACL and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), (Online), pp. 7282-7296, ACL, Aug. 2021.
  78. Y. Dou, M. Forbes, R. Koncel-Kedziorski, N. A. Smith, and Y. Choi, "Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text," in Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7250-7274, 2022.
  79. P. J. Phillips, A. N. Yates, Y. Hu, C. A. Hahn, E. Noyes, K. Jackson, J. G. Cavazos, G. Jeckeln, R. Ranjan, S. Sankaranarayanan, et al., "Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms," Proc. National Academy of Sciences, vol. 115, no. 24, pp. 6171-6176, 2018.
  80. A. Uchendu, Z. Ma, T. Le, R. Zhang, and D. Lee, "TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation," in Findings of the Association for Computational Linguis- tics: EMNLP 2021, (Punta Cana, Dominican Republic), pp. 2001-2016, Association for Computational Linguistics, Nov. 2021.
  81. N. Japkowicz et al., "Learning from imbalanced data sets: a comparison of various strategies," in AAAI workshop on learning from imbalanced data sets, vol. 68, pp. 10-15, AAAI Press Menlo Park, CA, 2000.
  82. C. Bellinger, S. Sharma, and N. Japkowicz, "One-class versus binary classification: Which and when?," in 2012 11th International Conference on Machine Learning and Applications, vol. 2, pp. 102-106, 2012.
  83. Y. Tay, D. Bahri, C. Zheng, C. Brunk, D. Metzler, and A. Tomkins, "Reverse engineering configurations of neural text generation models," in Proc. 58th Annu. Meet. ACL Sys. Demo., pp. 275-279, 2020.
  84. J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi, "Black-box generation of adversarial text sequences to evade deep learning classifiers," in 2018 IEEE Security and Privacy Workshops (SPW), pp. 50-56, IEEE, 2018.
  85. R. Gagiano, M. M.-H. Kim, X. J. Zhang, and J. Biggs, "Robustness anal- ysis of grover for machine-generated news detection," in Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association, pp. 119-127, 2021.
  86. S. M. Lundberg and S.-I. Lee, "A unified approach to interpreting model predictions," Advances in neural information processing systems, vol. 30, 2017.
  87. E. Crothers, H. Viktor, and N. Japkowicz, "Mean user-text agglomeration (muta): Practical user representation and visualization for detection of on- line influence operations," in International Conference on Computational Data and Social Networks, pp. 305-318, Springer, 2021.
  88. S. Stieglitz, F. Brachten, D. Berthelé, M. Schlaus, C. Venetopoulou, and D. Veutgen, "Do social bots (still) act different to humans?-comparing metrics of social bots with those of humans," in International conference on social computing and social media, pp. 379-395, Springer, 2017.
  89. J. Berger and K. L. Milkman, "What makes online content viral?," Journal of marketing research, vol. 49, no. 2, pp. 192-205, 2012.

Authors
Crothers, E., Japkowicz, N., & Viktor, H.
Affiliations
  • University of Ottawa (CA)
  • American University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Volume
11
Pages
70977-71002
ISSN
2169-3536 (ISSN-L)
DOI
10.48550/arxiv.2210.07321
arXiv
2210.07321
OpenAlex
W4383751019
Cited by
196
References
164
Citations by year
2026: 52 · 2025: 66 · 2024: 54 · 2023: 24
FWCI
18.48
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
Natural Sciences and Engineering Research Council of Canada (rgpin-2018-04047)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
AI-generated text detection, natural language generation, threat model, ChatGPT, large language models, cybersecurity, fairness, robustness, accountability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review