Generating Natural Language Adversarial Examples
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Deep neural networks (DNNs) are vulnerable to adversarial examples, perturbations to correctly classified examples which can cause the model to misclassify.In the image domain, these perturbations are often virtually indistinguishable to human perception, causing humans and state-of-the-art models to disagree.However, in the natural language domain, small perturbations are clearly perceptible, and the replacement of a single word can drastically alter the semantics of the document.Given these challenges, we use a black-box population-based optimization algorithm to generate semantically and syntactically similar adversarial examples that fool well-trained sentiment analysis and textual entailment models with success rates of 97% and 70%, respectively.We additionally demonstrate that 92.3% of the successful sentiment analysis adversarial examples are classified to their original label by 20 human annotators, and that the examples are perceptibly quite similar.Finally, we discuss an attempt to use adversarial training as a defense, but fail to yield improvement, demonstrating the strength and diversity of our adversarial examples.We hope our findings encourage researchers to pursue improving the robustness of DNNs in the natural language domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,960
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
On Assessing ML Model Robustness: A Methodological Framework (Academic Track)
Awadid, A., & Robert, B. (2025) · OASIcs : OpenAccess Series in Informatics · doi:10.4230/oasics.saia.2024.1 · Cited by 4,702
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495 · cited 3×
“Minimizing the number of word replacements necessary to induce misclassification has been studied in previous work (Papernot et al., 2016b), however without consideration given to semantics or syntactics, yielding incoherent generated examples.”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 5×
“Textual Entailment: We trained a textual entailment model using the Stanford Natural Language Inference (SNLI) corpus (Bowman et al., 2015).”
MethodInfluentialAdversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325
Cited by
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
“Alzantot et al. [153] proposed a method to generate adversarial examples for text while using a population-based genetic algorithm.”
Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6311 · Cited by 918
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
Ren, S., Deng, Y., He, K., & Che, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1103 · Cited by 635
BERT-ATTACK: Adversarial Attack Against BERT Using BERT
Li, L., Ma, R., Guo, Q., Xue, X., & Qiu, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.500 · Cited by 574
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
Morris, J., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., & Qi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.16 · Cited by 556
Recent Advances in Adversarial Training for Adversarial Robustness
Bai, T., Luo, J., Zhao, J., Wen, B., & Wang, Q. (2021) · arXiv · doi:10.48550/arxiv.2102.01356 · Cited by 490
BAE: BERT-based Adversarial Examples for Text Classification
Garg, S., & Ramakrishnan, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.498 · Cited by 478
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 477
A Survey on Data Augmentation for Text Classification
Bayer, M., Kaufhold, M.-A., & Reuter, C. (2023) · ACM Computing Surveys · doi:10.1145/3544558 · Cited by 425
Adversarial Attacks on Deep-learning Models in Natural Language Processing: A Survey
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3374217 · Cited by 407
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · arXiv · doi:10.48550/arxiv.2305.01937 · Cited by 331
“A special type of adversarial attack is called synonym substitution attacks (SSAs) (Alzantot et al., 2018), where the adversarial sample is created by replacing some words with their synonyms in the benign sample.”
BackgroundAdversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain
Rosenberg, I., Shabtai, A., Elovici, Y., & Rokach, L. (2022) · ACM Computing Surveys · doi:10.1145/3453158 · Cited by 295
“For spam detection, Alzantot et al. (2018) evaluated the attack-specific robustness defense methods of adversarial retraining against RNN classifiers, showing that such methods improve robustness against adversarial attacks.”
MethodInfluentialIntriguing Properties of Adversarial ML Attacks in the Problem Space
Pierazzi, F., Pendlebury, F., Cortellazzi, J., & Cavallaro, L. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00073 · Cited by 287
RobustBench: a standardized adversarial robustness benchmark
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., & Hein, M. (2020) · arXiv · doi:10.48550/arxiv.2010.09670 · Cited by 284
“, natural language processing [2], malware detection [51]) where adversarial robustness can be of interest.”
BackgroundProgress in Neural NLP: Modeling, Learning, and Reasoning
Zhou, M., Duan, N., Liu, S., & Shum, H.-Y. (2020) · Engineering · doi:10.1016/j.eng.2019.12.014 · Cited by 281
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06796 · Cited by 248
Human-level play in the game of Diplomacy by combining language models with strategic reasoning
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. (2022) · Science · doi:10.1126/science.ade9097 · Cited by 241
Robust Neural Machine Translation with Doubly Adversarial Inputs
Cheng, Y., Jiang, L., & Macherey, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1425 · Cited by 227
“A few studies have been proposed in other natural language processing (NLP) tasks aiming to tackle this issue in classification tasks, e.g. in (Miyato et al., 2017; Alzantot et al., 2018; Ebrahimi et al., 2018b; Zhao et al., 2018).”
BackgroundDisentangling Adversarial Robustness and Generalization
Stutz, D., Hein, M., & Schiele, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00714 · Cited by 223
“Finally, adversarial examples have been applied to a wide variety of tasks, also beyond computer vision, e.g., [26, 18, 93, 47, 35, 54, 3, 16].”
BackgroundCertified Robustness to Adversarial Word Substitutions
Jia, R., Raghunathan, A., Göksel, K., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1423 · Cited by 222
“We also use a different language model6 from Alzantot et al. (2018) that achieves perplexity of 50.79 on the One Billion Word dataset (Chelba et al., 2013).”
BackgroundMethodInfluentialVisual Adversarial Examples Jailbreak Aligned Large Language Models
Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., & Mittal, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30150 · Cited by 150
A Holistic Approach to Undesired Content Detection in the Real World
Markov, T., Zhang, C., Agarwal, S., Eloundou, T., Lee, T., Adler, S., Jiang, A., & Weng, L. (2022) · arXiv · doi:10.48550/arxiv.2208.03274 · Cited by 129
“It is widely known that ML models are vulnerable to adversarial inputs and may make predictions based on seemingly irrelevant features (Szegedy et al. 2013; Goodfellow, Shlens, and Szegedy 2015; Alzantot et al. 2018; Zhang et al. 2021).”
Adversarial Attacks on Time Series
Karim, F., Majumdar, S., & Darabi, H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2986319 · Cited by 118
On the Robustness of Self-Attentive Models
Hsieh, Y.-L., Cheng, M., Juan, D.-C., Wei, W., Hsu, W.-L., & Hsieh, C.-J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1147 · Cited by 96
“Unlike computer vision models, generating an effective, textual adversarial example that misleads a model but can go unnoticed by humans is a challenging and thriving research problem (Alzantot et al., 2018).”
BackgroundMeasure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90
Adversarial Attacks Against Deep Generative Models on Data: A Survey
Sun, H., Zhu, T., Zhang, Z., Jin, D., Xiong, P., & Zhou, W. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130903 · Cited by 90
“removing words [99], replacing words arbitrarily [100], or substituting words with synonyms [101].”
BackgroundBad Characters: Imperceptible NLP Attacks
Boucher, N., Shumailov, I., Anderson, R., & Papernot, N. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833641 · Cited by 102
Sentiment spin: Attacking financial sentiment with GPT-3
Leippold, M. (2023) · Finance Research Letters · doi:10.1016/j.frl.2023.103957 · Cited by 78
Shilling Black-Box Recommender Systems by Learning to Generate Fake User Profiles
Lin, C., Chen, S., Zeng, M., Zhang, S., Gao, M., & Li, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3183210 · Cited by 62
DynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56
“These efforts seek to determine whether models have met specific learning targets (Alzantot et al., 2018; Glockner et al., 2018; Naik et al., 2018; Nie et al., 2019), exploit relatively superficial properties of their training data, (Jia and Liang, 2017; Kaushik and Lipton, 2018; Zhang et al.”
BackgroundGrey-box Adversarial Attack And Defence For Sentiment Classification
Xu, Y., Zhong, X., Jimeno Yepes, A., & Lau, J. H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.321 · Cited by 54
FENCE: Feasible Evasion Attacks on Neural Networks in Constrained Environments
Chernikova, A., & Oprea, A. (2022) · ACM Transactions on Privacy and Security · doi:10.1145/3544746 · Cited by 52
“There is work on designing attacks in other domains, such as audio: [26], [15], [85], [64], [60], [79], [12] [56]; text: [54], [21], [42], [24], [3]; and video: [40], [33], [75].”
Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
Raina, V., Liusie, A., & Gales, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14016 · Cited by 40
Zhuo, T. Y., Li, Z., Huang, Y., Shiri, F., Wang, W., Haffari, G., & Li, Y.-F. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.77 · Cited by 31
Red Teaming Language Model Detectors with Language Models
Shi, Z., Wang, Y., Yin, F., Chen, X., Chang, K.-W., & Hsieh, C.-J. (2023) · arXiv · doi:10.48550/arxiv.2305.19713 · Cited by 30
Self-citationFinding Generalizable Evidence by Learning to Convince Q&A Models
Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., & Cho, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1244 · Cited by 27
“Neural networks are susceptible to adversarial examples—changes to the input which do or do not change the network’s prediction in surprising ways (Szegedy et al., 2014; Jia and Liang, 2017; Ribeiro et al., 2018; Alzantot et al., 2018).”
BackgroundMultimodal Approaches for Alzheimer’s Detection Using Patients’ Speech and Transcript
Cai, H., Huang, X., Liu, Z., Liao, W., Dai, H., Wu, Z., Zhu, D., Ren, H., Li, Q., Liu, T., et al. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43075-6_34 · Cited by 22
Improving the robustness and accuracy of biomedical language models through adversarial training
Moradi, M., & Samwald, M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104114 · Cited by 18
Silent guardian: protecting text from malicious exploitation by large language models
Zhao, J., Chen, K., Yuan, X., Qi, Y., Zhang, W., & Yu, N. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3455775 · Cited by 12
Towards Fair and Robust Classification
Sun, H., Wu, K., Wang, T., & Wang, W. H. (2022) · 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp53844.2022.00030 · Cited by 3
Model Editing with Canonical Examples
Hewitt, J., Chen, S., Xie, L. L., Adams, E., Liang, P., & Manning, C. D. (2024) · arXiv · doi:10.48550/arxiv.2402.06155 · Cited by 1
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
“A key NLP line of research in text adversarial attacks involves synonym substitution attacks (SSAs) (Alzantot et al., 2018) generate adversarial samples by replacing words with synonyms in benign text, relying on sophisticated methods such as TextFooler (Jin et al., 2020), PWWS (Ren et al., 2019),…”
Xian, R. P., Lee, A. J., Lolla, S., Wang, V., Cui, Q., Ro, R., & Abbasi-Asl, R. (2024) · arXiv · doi:10.48550/arxiv.2402.10527
Gradient-based adversarial attacks against text transformers
Guo, C., Sablayrolles, A., Jégou, H., & Kiela, D. (2021) · arXiv · doi:10.48550/arxiv.2104.13733
Metadata
- Authors
- Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M., & Chang, K.-W.
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 2890-2896
- OpenAlex
- W2962718684
- Cited by
- 787
- References
- 27
- FWCI
- 35.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.