Bad Characters: Imperceptible NLP Attacks
Open accessThe DOI matches its Crossref record; 69 of the 73 references it cites were found in a registry or an index.
Abstract
Several years of research have shown that machine-learning systems are vulnerable to adversarial examples, both in theory and in practice. Until now, such attacks have primarily targeted visual models, exploiting the gap between human and machine perception. Although text-based models have also been attacked with adversarial examples, such attacks struggled to preserve semantic meaning and indistinguishability. In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs. We use encoding-specific perturbations that are imperceptible to the human eye to manipulate the outputs of a wide range of Natural Language Processing (NLP) systems from neural machine-translation pipelines to web search engines. We find that with a single imperceptible encoding injection – representing one invisible character, homoglyph, reordering, or deletion – an attacker can significantly reduce the performance of vulnerable models, and with three injections most models can be functionally broken. Our attacks work against currently-deployed commercial systems, including those produced by Microsoft and Google, in addition to open source models published by Facebook, IBM, and HuggingFace. This novel series of attacks presents a significant threat to many language processing systems: an attacker can affect systems in a targeted manner without any assumptions about the underlying model. We conclude that text-based NLP systems require careful input sanitization, just like conventional applications, and that given such systems are now being deployed rapidly at scale, the urgent attention of architects and operators is required.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Storn, R., & Price, K. (1997) · Journal of Global Optimization · doi:10.1023/a:1008202821328 · Cited by 29,569
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495
Self-citationA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,486
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,644
An Overview of the Tesseract OCR Engine
Smith, R. (2007) · Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) Vol 2 · doi:10.1109/icdar.2007.4376991 · Cited by 2,376
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4009 · Cited by 2,344
Evasion attacks against machine learning at test time
Biggio, B., Corona, I., Maiorca, D., Nelson, B. A., Srndic, N., Laskov, P., Giacinto, G., & Roli, F. (2017) · CINECA IRIS Institutial Research Information System (University of Genoa) · Cited by 1,872
Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., & Hsieh, C.-J. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140448 · Cited by 1,656
Wild patterns: Ten years after the rise of adversarial machine learning
Biggio, B., & Roli, F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.023 · Cited by 1,409
A Survey on Hate Speech Detection using Natural Language Processing
Schmidt, A., & Wiegand, M. (2017) · Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media · doi:10.18653/v1/w17-1101 · Cited by 1,395
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1176 · Cited by 1,232
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
HotFlip: White-Box Adversarial Examples for Text Classification
Ebrahimi, J., Rao, A., Lowd, D., & Dou, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2006 · Cited by 922
Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861
Generating Natural Language Adversarial Examples
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivastava, M., & Chang, K.-W. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1316 · Cited by 787
CARER: Contextualized Affect Representations for Emotion Recognition
Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., & Chen, Y.-S. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1404 · Cited by 698
Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers
Gao, J., Lanchantin, J., Soffa, M. L., & Qi, Y. (2018) · 2018 IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00016 · Cited by 693
Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency
Ren, S., Deng, Y., He, K., & Che, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1103 · Cited by 635
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
Iyyer, M., Wieting, J., Gimpel, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1170 · Cited by 615
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
Morris, J., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., & Qi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.16 · Cited by 556
Scaling Neural Machine Translation
Ott, M., Edunov, S., Grangier, D., & Auli, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6301 · Cited by 544
Generating Natural Adversarial Examples
Zhao, Z., Dua, D., & Singh, S. (2017) · arXiv · doi:10.48550/arxiv.1710.11342 · Cited by 508
SoK: Security and Privacy in Machine Learning
Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2018) · 2018 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2018.00035 · Cited by 502
Self-citationExploiting machine learning to subvert your spam filter
Nelson, B. A., Barreno, M., Chi, F. J., Joseph, A. D., Rubinstein, B. I. P., Saini, U., Sutton, C. A., Tygar, J. D., & Xia, K. (2008) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 481
SoK: Towards the Science of security and privacy in machine learning
Papernot, N., McDaniel, P. D., Sinha, A., & Wellman, M. P. (2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 427
Self-citationCrafting adversarial input sequences for recurrent neural networks
Papernot, N., McDaniel, P., Swami, A., & Harang, R. (2016) · MILCOM 2016 - 2016 IEEE Military Communications Conference · doi:10.1109/milcom.2016.7795300 · Cited by 419
Self-citationSynthetic and Natural Noise Both Break Neural Machine Translation
Belinkov, Y., & Bisk, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02173 · Cited by 386
TextBugger: Generating Adversarial Text Against Real-world Applications
Li, J., Ji, S., Du, T., Li, B., & Wang, T. (2019) · Proceedings 2019 Network and Distributed System Security Symposium · doi:10.14722/ndss.2019.23138 · Cited by 343
(2018) · International Conference on Learning Representations · Cited by 285
Certified Robustness to Adversarial Word Substitutions
Jia, R., Raghunathan, A., Göksel, K., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1423 · Cited by 222
Towards the Science of Security and Privacy in Machine Learning
Papernot, N., McDaniel, P. D., Sinha, A., & Wellman, M. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03814 · Cited by 192
Self-citationGabrilovich, E., & Gontmakher, A. (2002) · Communications of the ACM · doi:10.1145/503124.503156 · Cited by 153
A taxonomy and terminology of adversarial machine learningPreprint
Tabassi, E., Burns, K. J., Hadjimichael, M., Molina–Markham, A., & Sexton, J. T. (2019) · doi:10.6028/nist.ir.8269-draft · Cited by 135
UniSpaCh: A text-based data hiding method using Unicode space characters
Por, L. Y., Wong, K., & Chee, K. O. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2011.12.023 · Cited by 134
Sponge Examples: Energy-Latency Attacks on Neural Networks
Shumailov, I., Zhao, Y., Bates, D., Papernot, N., Mullins, R., & Anderson, R. (2021) · 2021 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp51992.2021.00024 · Cited by 125
Self-citationA survey of current paradigms in machine translation
Dorr, B., Jordan, P., & Benoit, J. (1999) · Advances in Computing · doi:10.1016/s0065-2458(08)60282-x · Cited by 121
On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models
Michel, P., Li, X., Neubig, G., & Pino, J. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-1314 · Cited by 106
Hong, S., Frigo, P., Kaya, Y., Giuffrida, C., & Dumitraş, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01017 · Cited by 100
Label-Only Membership Inference Attacks
Choquette-Choo, C. A., Tramèr, F., Carlini, N., & Papernot, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14321 · Cited by 96
Self-citationA Reinforced Generation of Adversarial Examples for Neural Machine Translation
zou, W., Huang, S., Xie, J., Dai, X., & Chen, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.319 · Cited by 91
Cutting through the confusion: a measurement study of homograph attacks
Holgers, T., Watson, D. E., & Gribble, S. D. (2006) · USENIX Annual Technical Conference · Cited by 89
TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPPreprint
Morris, J. X., Lifland, E., Yoo, J. Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05909 · Cited by 59
ShamFinder: An Automated Framework for Detecting IDN Homographs
Hiroaki, S., Chiba, D., Yoneya, Y., Mori, T., & Goto, S. (2019) · ACM Internet Measurement Conference (IMC) · doi:10.1145/3355369.3355587 · Cited by 52
Detecting and modeling local text reuse
Smith, D. A., Cordell, R., Dillon, E. M., Stramp, N., & Wilkerson, J. (2014) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2014.6970166 · Cited by 48
Hong, S., Frigo, P., Kaya, Y., Giuffrida, C., & Dumitraş, T. (2019) · arXiv (Cornell University) · doi:10.5555/3361338.3361373 · Cited by 43
A Taxonomy and Terminology of Adversarial Machine Learning
Tabassi, E. (2023) · doi:10.6028/nist.ai.100-2 · Cited by 39
Detecting and Modeling Local Text Reuse
Smith, D. A., Cordell, R., Dillon, E. M., Stramp, N., & Wilkerson, J. (2015) · Cited by 26
Wikipedia Talk Labels: Toxicity
Thain, N., Dixon, L., & Wulczyn, E. (2017) · Figshare · doi:10.6084/m9.figshare.4563973.v2 · Cited by 16
Wikipedia Talk Labels: Toxicity
Thain, N., Dixon, L., & Wulczyn, E. (2017) · Figshare · doi:10.6084/m9.figshare.4563973 · Cited by 16
Ten years of attacks on companies using visual impersonation of domain names
Simpson, G., Moore, T., & Clayton, R. (2020) · APWG Symposium on Electronic Crime Research (eCrime) · doi:10.1109/ecrime51433.2020.9493251 · Cited by 10
Research works printed but not linked: 10
- 2020, Unicode Bidirectional Algorithm, The Unicode Consortium
- 2014, Unicode Security Considerations, The Unicode Consortium
- 2015, CAPEC-632: Homograph Attack via Homoglyphs (Version 3.4), MITRE Common Attack Pattern Enumeration and Classification 632
- knight, 2018, Evasion with unicode format characters, SpamAssassin - Dev
- 10.18653/v1/D17-1026 · DOI · Paper page
- savani, 2021, DistilBERT Base Uncased Emotion
- weaver, 1949, Translation, Translation Machine Translation of Languages Fourteen Essays
- 2020, Münchener Digitalisierungszentrum (MDZ) - Bayerische Staatsbibliothek, BERT Large Cased Finetuned CoNLL03 English
- frenkel, 2020, Facebook is failing in global disinformation fight, says former worker, New York Times
- 2020, Unicode Security Considerations, The Unicode Consortium
Cited by
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
Adversarial machine learning: a taxonomy and terminology of attacks and mitigations
Vassilev, A., Oprea, A., Fordyce, A., & Anderson, H. (2024) · doi:10.6028/nist.ai.100-2e2023 · Cited by 147
A Watermark for Large Language Models
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2301.10226 · Cited by 194
“Like homoglyphs, these characters must be removed through canonicalization (Pajola & Conti, 2021; Boucher et al., 2022).”
Metadata
- Authors
- Boucher, N., Shumailov, I., Anderson, R., & Papernot, N.
- Affiliations
- University of Cambridge (GB)
- Venue
- 2022 IEEE Symposium on Security and Privacy (SP)
- Publisher
- IEEE
- Published
- 2022-05
- Pages
- 1987-2004
- ISSN
- 1081-6011 (ISSN-L)
- OpenAlex
- W3176393001
- Cited by
- 102
- References
- 73
- Citations by year
- 2026: 13 · 2025: 26 · 2024: 23 · 2023: 32 · 2022: 5
- FWCI
- 11.89
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Symposium on Security and Privacy (S&P) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Bosch-Forschungsstiftung
- Defense Advanced Research Projects Agency
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- adversarial examples, imperceptible attacks, black-box attacks, homoglyphs, neural machine translation, search engines, input sanitization, semantic preservation, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.