Detecting computer-generated disinformation
Open accessAbstract
Modern neural language models can be used by malicious actors to automatically produce textual content looking as it has been written by genuine human users. Due to progress in the controllability of computer-generated text, there is a risk that state-sponsored actors may start using such methods for conducting large-scale information operations. Various detection algorithms have been suggested in the research literature to identify texts produced by language model-based generators, but these are often mainly evaluated on test data from the same distribution as they have been trained on. We evaluate promising Transformer-based detection algorithms in a large variety of experiments involving both in-distribution and out-of-distribution test data, as well as evaluation on more realistic in-the-wild data. It is shown that the generalizability of the detectors can be questioned, especially when applied to short social media posts. Moreover, the best performing (RoBERTa-based) detector is shown to be non-robust also to basic adversarial attacks, illustrating how easy it is for malicious actors to avoid detection by the current state-of-the-art detection algorithms.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine-generated text detection methods remain limited in their precision and scope.
“However, these methods remain limited in their precision and scope.”
From On the Risk of Misinformation Pollution with Large Language Models (2023) · cites this paper as Stiff and Johansson, 2022
Character-level perturbations (e.g., DeepWordBug) resulted in more than a halving of the performance of the OpenAI large detector.
“Stiff and Johansson (2022) utilized the DeepWordBug (Gao et al. 2018b) adversarial attack algorithm to introduce character-level perturbations to generated texts, including adjacent character swaps, character substitutions, deletions, and insertions, which resulted in more than a halving of the performance of the OpenAI large detector.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Stiff and Johansson (2022)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
GoEmotions: A Dataset of Fine-Grained Emotions
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., & Ravi, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.372 · Cited by 873 · see citing works
Detecting Automation of Twitter Accounts: Are You a Human, Bot, or Cyborg?
Chu, Z., Gianvecchio, S., Wang, H., & Jajodia, S. (2012) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2012.75 · Cited by 634 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · cited 4× · see citing works
“GLTR relies on that text generation methods tend to sample from a truncated head of the full probability distribution.”
BackgroundInfluentialGlobal Texture Enhancement for Fake Face Detection in the Wild
Liu, Z., Qi, X., & Torr, P. H. S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00808 · Cited by 401 · see citing works
The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race
Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., & Tesconi, M. (2017) · Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion · doi:10.1145/3041021.3055135 · Cited by 381 · cited 1× · see citing works
“Early machine learning-based approaches to detecting automation of social media accounts were often based on relatively simple measurements related to posting behavior, posted content, and account properties of individual accounts [6], but such approaches are not working as well today, due to newer generations of bots…”
BackgroundA decade of social bot detection
Cresci, S. (2020) · Communications of the ACM · doi:10.1145/3409116 · Cited by 323 · see citing works
Automatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1911.00650 · Cited by 264 · cited 2×
“Averages over the calculated values can also be used as input features to shallow classifiers, which has been tested with limited success [26].”
BackgroundAutomatic Detection of Generated Text is Easiest when Humans are Fooled
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.164 · Cited by 264 · see citing works
Detecting Deep-Fake Videos from Appearance and Behavior
Agarwal, S., Farid, H., El-Gaaly, T., & Lim, S.-N. (2020) · 2020 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs49906.2020.9360904 · Cited by 242 · cited 1× · see citing works
“However, compared to detection of images [11,31,45,46] and videos [2,7,32] being synthesized or manipulated using generative models, its text counterpart is under-researched.”
BackgroundTweepFake: about detecting deepfake tweets
Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251415 · Cited by 185 · see citing works
The False positive problem of automatic bot detection in social science research
Rauchfleisch, A., & Kaiser, J. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241045 · Cited by 184 · see citing works
Flagged first, then by citations
Research works printed but not linked: 38
- Adelani, D.I., Mai, H., Fang, F., Nguyen, H.H., Yamagishi, J., Echizen, I.: Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In: International Conference on Advanced Information Networking and Applications, Springer, pp. 1341-1354 (2020)
- Bastos, M., Farkas, J.: "Donald Trump is my president!": the inter- net research agency propaganda machine. Social Media+Society, pp. 1-13 (2019)
- Bradshaw, S., Bailey, H., Howard, P.N.: Industrialized disinforma- tion: 2020 global inventory of organized social media manipulation. Technical Report, Oxford Internet Institute (2021)
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Lan- guage models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.), Advances in Neural Information Processing Systems, vol. 33, Curran Associates, Inc., pp. 1877-1901 (2020). https://proceedings.neurips.cc/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
- Ciftci, U.A., Demir, I., Yin, L.: Fakecatcher: detection of syn- thetic portrait videos using biological signals. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020. 3009287 · DOI · Paper page
- Cohen, D., Croft, W.B.: End to end long short term memory net- works for non-factoid question answering. In: Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, pp. 143-146 (2016)
- Dang, H., Liu, F., Stehouwer, J., Liu, X., Jain, A.K.: On the detec- tion of digital face manipulation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Com- puter Society, Los Alamitos, CA, USA, pp. 5780-5789 (2020). https://doi.org/10.1109/CVPR42600.2020.00582 · DOI · Paper page
- Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., Liu, R.: Plug and play language models: a simple approach to controlled text generation. In: International Confer- ence on Learning Representations (2020). https://openreview.net/ forum?id=H1edEyBKDS
- Dawson, A., Innes, M.: How Russia's internet research agency built its disinformation campaign. Political Q. 90(2), 245-256 (2019)
- Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre- training of deep bidirectional transformers for language under- standing. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, vol. 1, Association for Computational Linguistics, Minneapolis, Minnesota, pp. 4171- 4186 (2019)
- Fan, A., Lewis, M., Dauphin, Y.: Hierarchical neural story genera- tion. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1, no. long papers, Association for Computational Linguistics, Melbourne, Australia, pp. 889-898 (2018)
- Foster, D.: Generative Deep Learning: Teaching Machines to Paint, Write, Compose, and Play, O'Reilly Media (2019)
- Gao, J., Lanchantin, J., Soffa, M.L., Qi, Y.: Black-box generation of adversarial text sequences to evade deep learning classifiers. In: 2018 IEEE Security and Privacy Workshops (SPW), IEEE, pp. 50-56 (2018)
- Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford, vol. 1, no. 12 (2009)
- Goodfellow, I., McDaniel, P., Papernot, N.: Making machine learn- ing robust against adversarial inputs. Commun. ACM 61(7), 56-66 (2018). https://doi.org/10.1145/3134599 · DOI · Paper page
- Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Bengio, Y., LeCun, Y. (eds.), 3rd Interna- tional Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, Conference Track Proceedings (2015). arXiv:abs/1412.6572
- Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. In: 8th International Confer- ence on Learning Representations (2020). https://openreview.net/ forum?id=rygGQyrFvH
- Jawahar, G., Abdul-Mageed, M., Lakshmanan V.S., L.: Automatic detection of machine generated text: a critical survey. In: Pro- ceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguis- tics, pp. 2296-2309 (2020)
- Keskar, N.S., McCann, B., Varshney, L.R., Xiong, C., Socher, R.: CTRL: a conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 (2019) · arXiv
- Krause, B., Gotmare, A.D., McCann, B., Keskar, N.S., Joty, S., Socher, R., Rajani, N.F.: GeDi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367 (2020) · arXiv
- Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019) · arXiv
- Nguyen, H.H., Fang, F., Yamagishi, J., Echizen, I.: Multi-task learn- ing for detecting and segmenting manipulated facial images and videos. In: 2019 IEEE 10th International Conference on Biomet- rics Theory, Applications and Systems (BTAS), pp. 1-8 (2019). https://doi.org/10.1109/BTAS46853.2019.9185974 · DOI · Paper page
- Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural net- works. In: 2016 IEEE Symposium on Security and Privacy (SP), IEEE Computer Society, Los Alamitos, CA, USA, pp. 582-597 (2016). https://doi.org/10.1109/SP.2016.41 · DOI
- Pauls, A., Klein, D.: Faster and smaller n-gram language models. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Asso- ciation for Computational Linguistics, Portland, Oregon, USA, pp. 258-267 (2011)
- Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
- Shao, C., Ciampaglia, G.L., Varol, O., Yang, K.C., Flammini, A., Menczer, F.: The spread of low-credibility content by social bots. Nat. Commun. (2018). https://doi.org/10.1038/s41467-018- 06930-7 · DOI · Paper page
- Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic composi- tionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Process- ing, pp. 1631-1642 (2013)
- Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J.W., Kreps, S., et al.: Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203 (2019) · arXiv
- Soutner, D., Müller, L.: Application of LSTM neural networks in language modelling. In: Habernal, I., Matoušek, V. (eds.) Text, Speech, and Dialogue, pp. 105-112. Springer, Berlin (2013)
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fergus, R.: Intriguing properties of neural networks. In: Bengio, Y., LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR 2014 (2014). arXiv:abs/1312.6199
- Torusdag, M.B., Kutlu, M., Selşuk, A.A.: Are we secure from bots? Investigating vulnerabilities of botometer. In: 2020 5th Interna- tional Conference on Computer Science and Engineering (UBMK), pp. 343-348 (2020). https://doi.org/10.1109/UBMK50275.2020. 9219433 · DOI
- Uchendu, A., Le, T., Shu, K., Lee, D.: Authorship attribution for neural text generation. In: Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, pp. 8384- 8395 (2020). https://doi.org/10.18653/v1/2020.emnlp-main.673. https://aclanthology.org/2020.emnlp-main.673 · DOI · Paper page
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, U., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, Curran Associates Inc., Red Hook, NY, USA, pp. 6000-6010 (2017)
- Wang, R., Juefei-Xu, F., Ma, L., Xie, X., Huang, Y., Wang, J., Liu, Y.: Fakespotter: a simple yet robust baseline for spotting ai- synthesized fake faces. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI) (2020)
- Wang, S.Y., Wang, O., Zhang, R., Owens, A., Efros, A.A.: Cnn- generated images are surprisingly easy to spot... for now. In: Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR) (2020)
- Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roes- ner, F., Choi, Y.: Defending against neural fake news. In: Advances in Neural Information Processing Systems, pp. 9054-9065 (2019)
- Zhang, W.E., Sheng, Q.Z., Alhazmi, A., Li, C.: Adversarial attacks on deep-learning models in natural language processing: a survey. ACM Trans. Intell. Syst. Technol. (2020). https://doi.org/10.1145/ 3374217 · DOI
- Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.), Advances in Neu- ral Information Processing Systems, vol. 28. Curran Asso- ciates, Inc. (2015). https://proceedings.neurips.cc/paper/2015/file/ 250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf
Cited by
Sorted by citations
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543 · see citing works
Menon, D., & Shilpa, K. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e20962 · Cited by 359 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
On the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 70 · see citing works
RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
Dugan, L., Hwang, A., Trhlík, F., Zhu, A., Ludan, J. M., Xu, H., Ippolito, D., & Callison-Burch, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.674 · Cited by 48 · see citing works
Cross-domain detection of GPT-2-generated technical text
Rodriguez, J., Hay, T., Gros, D., Shamsi, Z., & Srinivasan, R. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.88 · Cited by 40 · see citing works
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and Challenges
Huang, B., Chen, C., & Shu, K. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3715073.3715076 · Cited by 42 · see citing works
Toward Mitigating Misinformation and Social Media Manipulation in LLM Era
Zhang, Y., Sharma, K., Du, L., & Liu, Y. (2024) · Companion Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589335.3641256 · Cited by 25 · see citing works
Fact Check: Assessing the Response of ChatGPT to Alzheimer's Disease Myths
Huang, S. S., Song, Q., Beiting, K. J., Duggan, M. C., Hines, K., Murff, H., Leung, V., Powers, J., Harvey, T. S., Malin, B., et al. (2024) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2024.105178 · Cited by 9 · see citing works
Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196 · see citing works
- Authors
- Stiff, H., & Johansson, F.
- Affiliations
- Swedish Defence Research Agency (SE)
- Corresponding author
- Johansson, Fredrik
- ORCID
- Johansson, Fredrik
- Venue
- International Journal of Data Science and Analytics
- Published
- 2021-12-23
- Volume
- 13
- Issue
- 4
- Pages
- 363-383
- ISSN
- 2364-415X (ISSN-L)
- 2364-4168
- OpenAlex
- W4200594764
- Cited by
- 79
- References
- 12
- Citations by year
- 2026: 9 · 2025: 22 · 2024: 21 · 2023: 22 · 2022: 4 · 2021: 1
- FWCI
- 16.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- International Journal of Data Science and Analytics (publishedVersion, cc-by)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- Funders
- Horizon 2020 Framework Programme (832921)
- Försvarsmakten
- Swedish Defence Research Agency
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
- Keywords
- disinformation detection, AI-generated text detection, large language models, transformer-based detectors, out-of-distribution generalization, adversarial robustness, social media posts, information operations, RoBERTa, in-the-wild evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
