CoChat
Sign Up
Cited by 196Authorship Attribution and ProfilingDigital Media Forensic DetectionSpam and Phishing Detection

Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

Open access
Crothers, Evan N., Japkowicz, Nathalie, Viktor, Herna L.·2023·IEEE Access
DOI 10.1109/access.2023.3294090
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine-generated text is increasingly difficult to distinguish from text authored by humans. Powerful open-source models are freely available, and user-friendly tools that democratize access to generative models are proliferating. ChatGPT, which was released shortly after the first edition of this survey, epitomizes these trends. The great potential of state-of-the-art natural language generation (NLG) systems is tempered by the multitude of avenues for abuse. Detection of machine-generated text is a key countermeasure for reducing the abuse of NLG models, and presents significant technical challenges and numerous open problems. We provide a survey that includes 1) an extensive analysis of threat models posed by contemporary NLG systems and 2) the most complete review of machine-generated text detection methods to date. This survey places machine-generated text within its cybersecurity and social context, and provides strong guidance for future work addressing the most critical threat models. While doing so, we highlight the importance that detection systems themselves demonstrate trustworthiness through fairness, robustness, and accountability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2023.3294090

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some studies categorize any text generated, modified, or extended by a machine as LLM-generated, simplifying the task to LLM-generated Text Detection or LLM-generated Text Attribution.

    “Some studies categorize any text that is generated, modified, or extended by a machine as LLM-generated.This simplifies the task to either LLM-generated Text Detection or LLMgenerated Text Attribution [Yang et al., 2023c;Crothers et al., 2023].”

    From Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges (2025) · cites this paper as Crothers et al., 2023

    The Definition and Scope section defines machine-generated text as text produced, modified, or extended by a machine. Section IV describes detection as distinguishing machine-generated from human text, and Section V-B discusses attribution to a generative model. The paper uses the broader category “machine-generated text,” however; it does not categorize all such text as LLM-generated. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 164 · Cited by 196

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 131

  1. S. Baki, R. Verma, A. Mukherjee, and O. Gnawali, "Scaling and ef- fectiveness of email masquerade attacks: Exploiting natural language generation," in Proc. 2017 ACM ASIA CCS '17, (New York, NY, USA), p. 469-482, ACM, 2017.
  2. A. Giaretta and N. Dragoni, "Community targeted phishing," in Proceed- ings of 6th International Conference in Software Engineering for Defence Applications (P. Ciancarini, M. Mazzara, A. Messina, A. Sillitti, and G. Succi, eds.), (Cham), pp. 86-93, Springer International Publishing, 2020.
  3. K. Shu, S. Wang, D. Lee, and H. Liu, "Mining disinformation and fake news: Concepts, methods, and recent advancements," in Disinformation, misinformation, and fake news in social media, pp. 1-19, Springer, 2020.
  4. H. Stiff and F. Johansson, "Detecting computer-generated disinforma- tion," Int. J. Data Sci. Anal., vol. 13, no. 4, pp. 363-383, 2022.
  5. R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi, "Defending against neural fake news," NeurIPS, vol. 32, 2019.
  6. D. I. Adelani, H. Mai, F. Fang, H. H. Nguyen, J. Yamagishi, and I. Echizen, "Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detec- tion," in Advanced Information Networking and Applications (L. Barolli, F. Amato, F. Moscato, T. Enokido, and M. Takizawa, eds.), (Cham), pp. 1341-1354, Springer International Publishing, 2020.
  7. A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," ArXiv, vol. abs/1706.03762, 2017. · arXiv
  8. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few-shot learners," ArXiv, vol. abs/2005.14165, 2020. · arXiv
  9. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al., "Opt: Open pre-trained transformer language models," arXiv preprint arXiv:2205.01068, 2022. · arXiv
  10. W. Zeng, X. Ren, T. Su, H. Wang, Y. Liao, Z. Wang, X. Jiang, Z. Yang, K. Wang, X. Zhang, C. Li, Z. Gong, Y. Yao, X. Huang, J. Wang, J. Yu, Q. Guo, Y. Yu, Y. Zhang, J. Wang, H. Tao, D. Yan, Z. Yi, F. Peng, F. Jiang, H. Zhang, L. Deng, Y. Zhang, Z. Lin, C. Zhang, S. Zhang, M. Guo, S. Gu, G. Fan, Y. Wang, X. Jin, Q. Liu, and Y. Tian, "Pangu- α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation," CoRR, vol. abs/2104.12369, 2021. · arXiv
  11. Open Cover Letter, "Open cover letter: Generate cover letters with ai," 2022.
  12. A. Kanapala, S. Pal, and R. Pamula, "Text summarization from legal documents: a survey," Artificial Intelligence Review, vol. 51, no. 3, pp. 371-402, 2019.
  13. M. Wang, M. Wang, F. Yu, Y. Yang, J. Walker, and J. Mostafa, "A system- atic review of automatic text summarization for biomedical literature and ehrs," Journal of the American Medical Informatics Association, vol. 28, no. 10, pp. 2287-2297, 2021.
  14. S. Kim, J. Lee, and G. Gweon, "Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality," in Proc. 2019 CHI conference on human factors in computing systems, pp. 1-12, 2019.
  15. A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero-shot text-to-image generation," in ICML, pp. 8821-8831, PMLR, 2021.
  16. G. Jawahar, M. Abdul-Mageed, and V. Laks Lakshmanan, "Automatic detection of machine generated text: A critical survey," in Proc. 28th International Conference on Computational Linguistics, pp. 2296-2309, 2020.
  17. D. Beresneva, "Computer-generated text detection using machine learn- ing: A systematic review," in International Conference on Applications of Natural Language to Information Systems, pp. 421-426, Springer, 2016.
  18. H.-Q. Nguyen-Son, N.-D. T. Tieu, H. H. Nguyen, J. Yamagishi, and I. E. Zen, "Identifying computer-generated text using statistical analysis," in 2017 APSIPA ASC, pp. 1504-1511, 2017.
  19. A. Lavoie and M. S. Krishnamoorthy, "Algorithmic detection of com- puter generated text," ArXiv, vol. abs/1008.0706, 2010. · arXiv
  20. E. Crothers, N. Japkowicz, H. Viktor, and P. Branco, "Adversarial robust- ness of neural-statistical features in detection of generative transformers," arXiv preprint arXiv:2203.07983, 2022. · arXiv
  21. R. Bitton, N. Maman, I. Singh, S. Momiyama, Y. Elovici, and A. Shabtai, "A framework for evaluating the cybersecurity risk of real world, machine learning production systems," CoRR, vol. abs/2107.01806, 2021. · arXiv
  22. D. Kaur, S. Uslu, K. J. Rittichier, and A. Durresi, "Trustworthy artificial intelligence: A review," ACM Comput. Surv., vol. 55, jan 2022.
  23. C. Dong, Y. Li, H. Gong, M. Chen, J. Li, Y. Shen, and M. Yang, "A survey of natural language generation," ACM Comput. Surv., jul 2022. Just Accepted.
  24. A. Gatt and E. Krahmer, "Survey of the state of the art in natural language generation: Core tasks, applications and evaluation," Journal of Artificial Intelligence Research, vol. 61, pp. 65-170, 2018.
  25. R. Perera and P. Nand, "Recent advances in natural language generation: A survey and classification of the empirical literature," Computing and Informatics, vol. 36, no. 1, pp. 1-32, 2017.
  26. S. Santhanam and S. Shaikh, "A survey of natural language generation techniques with a focus on dialogue systems-past, present and future directions," arXiv preprint arXiv:1906.00500, 2019. · arXiv
  27. J. Li, T. Tang, W. X. Zhao, and J.-R. Wen, "Pretrained language model for text generation: A survey," in Proc. 30th IJCAI (Z.-H. Zhou, ed.), pp. 4492-4499, IJCAI Organization, 8 2021. Survey Track.
  28. E. Reiter and R. Dale, "Building applied natural language generation systems," Natural Language Engineering, vol. 3, 03 2002.
  29. J. Lyons, Natural Language and Universal Grammar: Volume 1: Essays in Linguistic Theory, vol. 1. Cambridge University Press, 1991.
  30. L. Zhang and J.-T. Sun, Text Generation, pp. 3048-3051. Boston, MA: Springer US, 2009.
  31. S. Biderman and E. Raff, "Fooling moss detection with pretrained lan- guage models," in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 2933-2943, 2022.
  32. A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, and D. Mukhopad- hyay, "Adversarial attacks and defences: A survey," arXiv preprint arXiv:1810.00069, 2018. · arXiv
  33. W. Wang, B. Tang, R. Wang, L. Wang, and A. Ye, "A survey on adversarial attacks and defenses in text," CoRR, vol. abs/1902.07285, 2019. · arXiv
  34. A. Huq and M. T. Pervin, "Adversarial attacks and defense on texts: A survey," CoRR, vol. abs/2005.14108, 2020. · arXiv
  35. M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, "fairseq: A fast, extensible toolkit for sequence modeling," in Proc. of NAACL-HLT 2019: Demo., 2019.
  36. F. A. Sheikha and D. Inkpen, "Generation of formal and informal sentences," in Proc. 13th European Workshop on Natural Language Generation, pp. 187-193, 2011.
  37. A. Sudhakar, B. Upadhyay, and A. Maheswaran, ""transforming" delete, retrieve, generate approach for controlled text style transfer," in Proc. 2019 Conf. EMNLP-IJCNLP, pp. 3269-3279, 2019.
  38. A. Aghajanyan, A. Shrivastava, A. Gupta, N. Goyal, L. Zettlemoyer, and S. Gupta, "Better fine-tuning by reducing representational collapse," in ICLR, 2021.
  39. H. P. Luhn, "The automatic creation of literature abstracts," IBM Journal of research and development, vol. 2, no. 2, pp. 159-165, 1958.
  40. Z. Li, J. Kiseleva, and M. de Rijke, "Dialogue generation: From imitation learning to inverse reinforcement learning," CoRR, vol. abs/1812.03509, 2018. · arXiv
  41. Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, and W. B. Dolan, "Dialogpt: Large-scale generative pre-training for conversational response generation," pp. 270-278, 2020.
  42. X. Feng, M. Liu, J. Liu, B. Qin, Y. Sun, and T. Liu, "Topic-to-essay generation with neural networks.," in IJCAI, pp. 4078-4084, 2018.
  43. S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosin- ski, and R. Liu, "Plug and play language models: A simple approach to controlled text generation," in International Conference on Learning Representations, 2020.
  44. N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, "Ctrl: A conditional transformer language model for controllable generation," ArXiv, vol. abs/1909.05858, 2019. · arXiv
  45. H. Harkous, I. Groves, and A. Saffari, "Have your text and use it too! end- to-end neural data-to-text generation with semantic fidelity," in COLING 2020, 2020.
  46. J. Clive, K. Cao, and M. Rei, "Control prefixes for text generation," arXiv preprint arXiv:2110.08329, 2021. · arXiv
  47. J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang, "Git: A generative image-to-text transformer for vision and language," arXiv preprint arXiv:2205.14100, 2022. · arXiv
  48. G. Li, L. Zhu, P. Liu, and Y. Yang, "Entangled transformer for image captioning," in Proc. IEEE/CVF, pp. 8928-8937, 2019.
  49. H. Li, J. Zhu, C. Ma, J. Zhang, and C. Zong, "Multi-modal summarization for asynchronous collection of text, image, audio and video," in Proc. 2017 Conf. EMNLP, (Copenhagen, Denmark), pp. 1092-1102, ACL, Sept. 2017.
  50. S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko, "Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition," in 2013 IEEE ICCV, pp. 2712-2719, 2013.
  51. J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, "Attention-based models for speech recognition," NeurIPS, vol. 28, 2015.
  52. A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli, "Unsupervised speech recognition," NeurIPS, vol. 34, pp. 27826-27839, 2021.
  53. D. Jin, Z. Jin, Z. Hu, O. Vechtomova, and R. Mihalcea, "Deep learning for text style transfer: A survey," Computational Linguistics, vol. 48, pp. 155-205, Mar. 2022.
  54. L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, "Unified vision-language pre-training for image captioning and vqa," in Proc. AAAI Conference on Artificial Intelligence, vol. 34, pp. 13041-13049, 2020.
  55. P. A. Duboue and K. R. McKeown, "Statistical acquisition of content selection rules for natural language generation," in Proc. 2003 Confer- ence on Empirical Methods in Natural Language Processing, pp. 121- 128, 2003.
  56. I. Langkilde and K. Knight, "Generation that exploits corpus-based sta- tistical knowledge," in COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics, 1998.
  57. R. Kondadadi, B. Howald, and F. Schilder, "A statistical nlg framework for aggregated planning and realization," in Proc. 51st Annual Meeting of the ACL (Volume 1: Long Papers), pp. 1406-1415, 2013.
  58. L. E. Baum and T. Petrie, "Statistical inference for probabilistic functions of finite state markov chains," The annals of mathematical statistics, vol. 37, no. 6, pp. 1554-1563, 1966.
  59. S. Janarthanam and O. Lemon, "Learning lexical alignment policies for generating referring expressions for spoken dialogue systems," in Proc. 12th ENLG, pp. 74-81, 2009.
  60. N. Dethlefs and H. Cuayáhuitl, "Hierarchical reinforcement learning for adaptive text generation," in Proc. 6th INLG, ACL, July 2010.
  61. N. Kalchbrenner and P. Blunsom, "Recurrent continuous translation mod- els," in Proc. 2013 conference on empirical methods in natural language processing, pp. 1700-1709, 2013.
  62. T. Mikolov et al., "Stat. lang. models based on neural networks," Presen- tation at Google, Mountain View, 2nd April, vol. 80, no. 26, 2012.
  63. M. Berglund, T. Raiko, M. Honkala, L. Kärkkäinen, A. Vetek, and J. Karhunen, "Bidirectional recurrent neural networks as generative mod- els," in NIPS, 2015.
  64. D. Pawade, A. M. Sakhapara, M. Jain, N. Jain, and K. Gada, "Story scrambler -automatic text generation using word level rnn-lstm," Interna- tional Journal of Information Technology and Computer Science, 2018.
  65. M. O. Topal, A. Bas, and I. van Heerden, "Exploring transform- ers in natural language generation: Gpt, bert, and xlnet," ArXiv, vol. abs/2102.08036, 2021. · arXiv
  66. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, pp. 139-144, 2020.
  67. K. Lin, D. Li, X. He, M.-T. Sun, and Z. Zhang, "Adversarial ranking for language generation," in NIPS, 2017.
  68. L. Yu, W. Zhang, J. Wang, and Y. Yu, "Seqgan: Sequence generative adversarial nets with policy gradient," in AAAI, 2017.
  69. J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao, "Deep reinforcement learning for dialogue generation," in EMNLP, 2016.
  70. Z. Shi, X. Chen, X. Qiu, and X. Huang, "Toward diverse text generation with inverse reinforcement learning," in IJCAI, 2018.
  71. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, "The pile: An 800gb dataset of diverse text for language modeling," CoRR, vol. abs/2101.00027, 2021. · arXiv
  72. C. Meister, T. Pimentel, G. Wiher, and R. Cotterell, "Locally Typical Sampling," Transactions of the Association for Computational Linguis- tics, vol. 11, pp. 102-121, 01 2023.
  73. J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional transformers," CoRR, vol. abs/1810.04805, 2018. · arXiv
  74. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu, "Mass: Masked sequence to sequence pre-training for language generation," in ICML, pp. 5926- 5936, 2019.
  75. J. Howard and S. Ruder, "Universal language model fine-tuning for text classification," arXiv preprint arXiv:1801.06146, 2018. · arXiv
  76. B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani, "Gedi: Generative discriminator guided sequence generation," arXiv preprint arXiv:2009.06367, 2020. · arXiv
  77. S. Bromander, A. Jøsang, and M. Eian, "Semantic cyberthreat mod- elling.," in STIDS, pp. 74-78, 2016.
  78. L. Kohnfelder and P. Garg, "The threats to our products," Microsoft Interface, Microsoft Corporation, vol. 33, 1999.
  79. T. UcedaVelez and M. M. Morana, Risk Centric Threat Modeling: process for attack simulation and threat analysis. John Wiley & Sons, 2015.
  80. A. Shostack, Threat modeling: Designing for security. John Wiley & Sons, 2014.
  81. K. L. Chiew, K. S. C. Yong, and C. L. Tan, "A survey of phishing attacks: Their types, vectors and technical approaches," Expert Systems with Applications, vol. 106, pp. 1-20, 2018.
  82. Z. Alkhalil, C. Hewage, L. Nawaf, and I. Khan, "Phishing attacks: A recent comprehensive study and a new anatomy," Frontiers in Computer Science, vol. 3, 2021.
  83. A. J. Burns, M. E. Johnson, and D. D. Caputo, "Spear phishing in a barrel: Insights from a targeted phishing campaign," J. Organ. Comput. Electron. Commer., vol. 29, no. 1, pp. 24-39, 2019.
  84. S. Abu-Nimeh, D. Nappa, X. Wang, and S. Nair, "A comparison of machine learning techniques for phishing detection," in Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit, pp. 60-69, 2007.
  85. R. Schuster, C. Song, E. Tromer, and V. Shmatikov, "You autocom- plete me: Poisoning vulnerabilities in neural code completion," in 30th USENIX Security Symposium (USENIX Security 21), pp. 1559-1575, 2021.
  86. P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin, "Generating fake cyber threat intelligence using transformer-based models," in 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9, 2021.
  87. N. Baracaldo, B. Chen, H. Ludwig, and J. A. Safavi, "Mitigating poi- soning attacks on machine learning models: A data provenance based approach," in Proc. 10th ACM Workshop on Artificial Intelligence and Security, pp. 103-110, 2017.
  88. M. Gabielkov, A. Ramachandran, A. Chaintreau, and A. Legout, "Social Clicks: What and Who Gets Read on Twitter?," in ACM SIGMETRICS / IFIP Performance 2016, (Antibes Juan-les-Pins, France), June 2016.
  89. American Press Institute, "How americans get their news," Mar 2014.
  90. K. McGuffie and A. Newhouse, "The radicalization risks of GPT-3 and advanced neural language models," CoRR, vol. abs/2009.06807, 2020. · arXiv
  91. J. Mink, L. Luo, N. M. Barbosa, O. Figueira, Y. Wang, and G. Wang, "DeepPhish: Understanding user trust towards artificially generated pro- files in online social networks," in 31st USENIX Security Symposium (USENIX Security 22), (Boston, MA), pp. 1669-1686, USENIX Associ- ation, Aug. 2022.
  92. E. Crothers, N. Japkowicz, and H. L. Viktor, "Towards ethical content- based detection of online influence campaigns," in 2019 IEEE 29th Inter- national Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6, IEEE, 2019.
  93. I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, G. Krueger, J. W. Kim, S. Kreps, et al., "Release strategies and the social impacts of language models," arXiv preprint arXiv:1908.09203, 2019. · arXiv
  94. C. Labbé and D. Labbé, "Duplicate and fake publications in the scientific literature: how many scigen papers in computer science?," Scientomet- rics, vol. 94, no. 1, pp. 379-396, 2013.
  95. G. Cabanac and C. Labbé, "Prevalence of nonsensical algorithmically generated papers in the scientific literature," Journal of the Association for Information Science and Technology, vol. 72, no. 12, pp. 1461-1476, 2021.
  96. Z. Zhuang, E. Elmacioglu, D. Lee, and C. L. Giles, "Measuring confer- ence quality by mining program committee characteristics," in Proc. 7th JCDL, JCDL '07, (New York, NY, USA), p. 225-234, ACM, 2007.
  97. Harth, V. Presutti, R. Troncy, M. Acosta, A. Polleres, J. D. Fernández, J. Xavier Parreira, O. Hartig, K. Hose, and M. Cochez, eds.), (Cham), pp. 158-163, Springer International Publishing, 2020.
  98. B. Lufkin, "Why do cover letters still exist?," Oct 2021.
  99. A. D. Selbst, D. Boyd, S. A. Friedler, S. Venkatasubramanian, and J. Vertesi, "Fairness and abstraction in sociotechnical systems," in Proc. Conference on Fairness, Accountability, and Transparency, FAT* '19, (New York, NY, USA), p. 59-68, ACM, 2019.
  100. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, "Extracting training data from large language models," CoRR, vol. abs/2012.07805, 2020. · arXiv
  101. L. v. Ahn, M. Blum, N. J. Hopper, and J. Langford, "Captcha: Using hard ai problems for security," in International conference on the theory and applications of cryptographic techniques, pp. 294-311, Springer, 2003.
  102. M. Guerar, L. Verderame, M. Migliardi, F. Palmieri, and A. Merlo, "Gotta captcha 'em all: A survey of 20 years of the human-or-computer dilemma," ACM Comput. Surv., vol. 54, oct 2021.
  103. A. Uchendu, T. Le, K. Shu, and D. Lee, "Authorship attribution for neural text generation," in Proc. 2020 Conf. EMNLP, (Online), pp. 8384-8395, ACL, Nov. 2020.
  104. L. Kushnareva, D. Cherniavskii, V. Mikhailov, E. Artemova, S. Baran- nikov, A. Bernstein, I. Piontkovskaya, D. Piontkovski, and E. Burnaev, "Artificial text detection via examining the topology of attention maps," CoRR, vol. abs/2109.04825, 2021. · arXiv
  105. T. Shamardina, V. Mikhailov, D. Chernianskii, A. Fenogenova, M. Saidov, A. Valeeva, T. Shavrina, I. Smurov, E. Tutubalina, and E. Artemova, "Findings of the the ruatd shared task 2022 on artificial text detection in russian," arXiv preprint arXiv:2206.01583, 2022. · arXiv
  106. X. Chen, P. Jin, S. Jing, and C. Xie, "Automatic detection of chinese generated essayss based on pre-trained bert," in 2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Confer- ence (ITAIC), vol. 10, pp. 2257-2260, IEEE, 2022.
  107. Y. Cui, W. Che, T. Liu, B. Qin, and Z. Yang, "Pre-training with whole word masking for chinese bert," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3504-3514, 2021.
  108. L. Martin, B. Muller, P. J. O. Suárez, Y. Dupont, L. Romary, E. V. de la Clergerie, D. Seddah, and B. Sagot, "Camembert: a tasty french language model," ArXiv, vol. abs/1911.03894, 2019. · arXiv
  109. W. Antoun, F. Baly, and H. M. Hajj, "Arabert: Transformer-based model for arabic language understanding," ArXiv, 2020.
  110. S. Dadas, M. Perełkiewicz, and R. Poświata, "Pre-training polish transformer-based language models at scale," in International Conference on Artificial Intelligence and Soft Computing, pp. 301-314, Springer, 2020.
  111. Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang, "A tutorial on energy-based learning," Predicting structured data, vol. 1, no. 0, 2006.
  112. A. Bakhtin, Y. Deng, S. Gross, M. Ott, M. Ranzato, and A. Szlam, "Residual energy-based models for text," J. of Mach. Learn. Res., vol. 22, pp. 40-1, 2021.
  113. A. Bakhtin, S. Gross, M. Ott, Y. Deng, M. Ranzato, and A. Szlam, "Real or fake? learning to discriminate machine from human generated text," CoRR, vol. abs/1906.03351, 2019. · arXiv
  114. S. G. Tesfagergish, R. Damaševičius, and J. Kapočiūtė-Dzikienė, "Deep fake recognition in tweets using text augmentation, word embeddings and deep learning," in International Conference on Computational Science and Its Applications, pp. 523-538, Springer, 2021.
  115. J. Tourille, B. Sow, and A. Popescu, "Automatic detection of bot- generated tweets," in Proc. 1st International Workshop on Multimedia AI against Disinformation, MAD '22, (New York, NY, USA), p. 44-51, ACM, 2022.
  116. J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn, "The pushshift reddit dataset," CoRR, vol. abs/2001.08435, 2020. · arXiv
  117. P. Bhatt and A. Rios, "Detecting bot-generated text by character- izing linguistic accommodation in human-bot interactions," CoRR, vol. abs/2106.01170, 2021. · arXiv
  118. M. Latah, "Detection of malicious social bots: A survey and a refined taxonomy," Expert Systems with Applications, vol. 151, p. 113383, 2020.
  119. J. Salminen, C. Kandpal, A. M. Kamel, S. gyo Jung, and B. J. Jansen, "Creating and detecting fake reviews of online products," Journal of Retailing and Consumer Services, vol. 64, p. 102771, 2022.
  120. M. M. Bhat and S. Parthasarathy, "How effectively can machines defend against machine-generated fake news? an empirical study," in Proc. First Workshop on Insights from Negative Results in NLP, (Online), pp. 48- 53, ACL, Nov. 2020.
  121. P. J. Phillips, A. N. Yates, Y. Hu, C. A. Hahn, E. Noyes, K. Jackson, J. G. Cavazos, G. Jeckeln, R. Ranjan, S. Sankaranarayanan, et al., "Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms," Proc. National Academy of Sciences, vol. 115, no. 24, pp. 6171-6176, 2018.
  122. L. Dugan, D. Ippolito, A. Kirubarajan, and C. Callison-Burch, "Roft: A tool for evaluating human detection of machine-generated text," ArXiv, vol. abs/2010.03070, 2020. · arXiv
  123. A. Uchendu, Z. Ma, T. Le, R. Zhang, and D. Lee, "TURINGBENCH: A benchmark environment for Turing test in the age of neural text generation," in Findings of the Association for Computational Linguis- tics: EMNLP 2021, (Punta Cana, Dominican Republic), pp. 2001-2016, Association for Computational Linguistics, Nov. 2021.
  124. N. Japkowicz et al., "Learning from imbalanced data sets: a comparison of various strategies," in AAAI workshop on learning from imbalanced data sets, vol. 68, pp. 10-15, AAAI Press Menlo Park, CA, 2000.
  125. C. Bellinger, S. Sharma, and N. Japkowicz, "One-class versus binary classification: Which and when?," in 2012 11th International Conference on Machine Learning and Applications, vol. 2, pp. 102-106, 2012.
  126. Y. Tay, D. Bahri, C. Zheng, C. Brunk, D. Metzler, and A. Tomkins, "Reverse engineering configurations of neural text generation models," in Proc. 58th Annu. Meet. ACL Sys. Demo., pp. 275-279, 2020.
  127. D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, "Is BERT really robust? natural language attack on text classification and entailment," CoRR, vol. abs/1907.11932, 2019. · arXiv
  128. J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi, "Black-box generation of adversarial text sequences to evade deep learning classifiers," in 2018 IEEE Security and Privacy Workshops (SPW), pp. 50-56, IEEE, 2018.
  129. R. Gagiano, M. M.-H. Kim, X. J. Zhang, and J. Biggs, "Robustness anal- ysis of grover for machine-generated news detection," in Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association, pp. 119-127, 2021.
  130. E. Crothers, H. Viktor, and N. Japkowicz, "Mean user-text agglomeration (muta): Practical user representation and visualization for detection of on- line influence operations," in International Conference on Computational Data and Social Networks, pp. 305-318, Springer, 2021.
  131. S. Stieglitz, F. Brachten, D. Berthelé, M. Schlaus, C. Venetopoulou, and D. Veutgen, "Do social bots (still) act different to humans?-comparing metrics of social bots with those of humans," in International conference on social computing and social media, pp. 379-395, Springer, 2017.

Authors
Crothers, E. N., Japkowicz, N., & Viktor, H. L.
Affiliations
  • University of Ottawa (CA)
  • American University (US)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023
Volume
11
Pages
70977-71002
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2023.3294090
OpenAlex
W4383751019
Cited by
196
References
164
Citations by year
2026: 52 · 2025: 66 · 2024: 54 · 2023: 24
FWCI
18.48
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
Natural Sciences and Engineering Research Council of Canada (RGPIN-2018-04047)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
AI-generated text detection, natural language generation, threat model, ChatGPT, large language models, cybersecurity, fairness, robustness, accountability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review