Generating Natural Adversarial Examples
Open accessAbstract
Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zhao et al. used a GAN to generate sentence-level adversarial perturbations that are natural and legible.
“Zhao et al. [106] exploited a generative adversarial network to generate sentence-level adversarial perturbations.Their method can generate natural and legible adversarial samples to deceive the classifier.”
From Enhancing the Robustness of AI-Generated Text Detectors: A Survey (2025) · cites this paper as [106]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Improved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800 · see citing works
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680 · see citing works
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1× · see citing works
“l network through different levels of representation erasure. However, erasure of words or phrases directly often harms text integrity, resulting in semantically or grammatically incorrect sentences. Ribeiro et al. (2018) replace tokens by random words of the same POS tag with probability proportional to embedding sim…”
BackgroundSelf-citation
Flagged first, then by citations
Research works printed but not linked: 27
- David Alvarez-Melis and Tommi S. Jaakkola. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Empirical Methods in Natural Language Processing (EMNLP) , 2017.
- Martin Arjovsky and L´ eon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations (ICLR) , 2017.
- Martin Arjovsky, Soumith Chintala, and L´ eon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning (ICML) , 2017.
- Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study. arXiv preprint 1706.08224 , 2017. · arXiv
- Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations (ICLR) , 2018.
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP) , 2015.
- Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced lstm for natural language inference. In Association for Computational Linguistics (ACL) , 2017.
- Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, and Deepak Verma. Adversarial classification. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2004.
- Ian Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. arXiv preprint 1701.00160 , 2016. · arXiv
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS) , 2014.
- Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR) , 2015.
- Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward controlled generation of text. In International Conference on Machine Learning (ICML) , 2017.
- Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP) , 2017.
- Diederik P. Kingma and Max Welling. Stochastic gradient vb and the variational auto-encoder. In International Conference on Learning Representations (ICLR) , 2014.
- Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 1607.02533 , 2016. · arXiv
- Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.
- Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint 1612.08220 , 2016. · arXiv
- Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint 1605.07277 , 2016a. · arXiv
- Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy , 2016b.
- Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the ACM on Asia Conference on Computer and Communications Security , 2017.
- Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Deepxplore: Automated whitebox testing of deep learning systems. In the ACM Symposium on Operating Systems Principles , 2017.
- Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint 1706.04987 , 2017. · arXiv
- Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NIPS) , 2016.
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR) , 2014.
- Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR) , 2018.
- Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506.03365 , 2015. · arXiv
- Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. Adversarially regularized autoencoders. arXiv preprint 1706.04223 , 2017. · arXiv
Cited by
Sorted by citations
Adversarial Examples: Attacks and Defenses for Deep Learning
Yuan, X., He, P., Zhu, Q., & Li, X. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2886017 · Cited by 1,733 · see citing works
Generative Adversarial Networks: A Survey Toward Private and Secure Applications
Cai, Z., Xiong, Z., Xu, H., Wang, P., Li, W., & Pan, Y. (2022) · ACM Computing Surveys · doi:10.1145/3459992 · Cited by 376 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
On the Robustness of Self-Attentive Models
Hsieh, Y.-L., Cheng, M., Juan, D.-C., Wei, W., Hsu, W.-L., & Hsieh, C.-J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1147 · Cited by 96 · see citing works
Measure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
Bad Characters: Imperceptible NLP Attacks
Boucher, N., Shumailov, I., Anderson, R., & Papernot, N. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833641 · Cited by 102 · see citing works
Zhuo, T. Y., Li, Z., Huang, Y., Shiri, F., Wang, W., Haffari, G., & Li, Y.-F. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.77 · Cited by 31 · see citing works
Enhancing the Robustness of AI-Generated Text Detectors: A Survey
Liu, X., Li, Y., & Li, K. (2025) · Mathematics · doi:10.3390/math13132145 · Cited by 10 · see citing works
- Authors
- Zhao, Z., Dua, D., & Singh, S.
- Affiliations
- University of California, Irvine (US)
- ORCID
- Singh, Sameer
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1710.11342
- OpenAlex
- W2766108848
- Cited by
- 508
- References
- 5
- Citations by year
- 2026: 5 · 2025: 18 · 2024: 47 · 2023: 49 · 2022: 71 · 2021: 106 · 2020: 104 · 2019: 70 · 2018: 38
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- adversarial examples, data manifold, generative adversarial networks, black-box classifiers, robustness analysis, image classification, natural language inference, machine translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
