Black-box Adversarial Attacks with Limited Queries and Information
Open accessAbstract
Current neural network-based classifiers are susceptible to adversarial examples even in the black-box setting, where the attacker only has query access to the model. In practice, the threat model for real-world systems is often more restrictive than the typical black-box model where the adversary can observe the full output of the network on arbitrarily many chosen inputs. We define three realistic threat models that more accurately characterize many real-world classifiers: the query-limited setting, the partial-information setting, and the label-only setting. We develop new attacks that fool classifiers under these more restrictive threat models, where previous methods would be impractical or ineffective. We demonstrate that our methods are effective against an ImageNet classifier under our proposed threat models. We also demonstrate a targeted black-box attack against a commercial classifier, overcoming the challenges of limited query access, partial information, and other practical issues to break the Google Cloud Vision API.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Natural evolutionary strategies (NES) for gradient estimation is one of the state-of-the-art methods in black-box adversarial ML.
“Natural evolutionary strategies (NES) for gradient estimation is one of the state-of-the-art methods in black-box adversarial ML [10, 25, 38].”
From Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning (2021) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 30 · Cited by 806
Research works printed but not linked: 26
- Athalye, A., Engstrom, L., Ilyas, A., and Kwok, K. Synthesizing robust adversarial examples. 2017. URL https://arxiv. org/abs/1707.07397 .
- Biggio, B., Nelson, B., and Laskov, P. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML'12, pp. 1467-1474, 2012. ISBN 978-1-45031285-1. URL http://dl.acm.org/citation.cfm? id=3042573.3042761 .
- Biggio, B., Corona, I., Maiorca, D., Nelson, B., ˇ Srndi´ c, N., Laskov, P., Giacinto, G., and Roli, F. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pp. 387-402. Springer, 2013.
- Brendel, W., Rauber, J., and Bethge, M. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018. URL https: //arxiv.org/abs/1712.04248 .
- Carlini, N., Mishra, P., Vaidya, T., Zhang, Y., Sherr, M., Shields, C., Wagner, D., and Zhou, W. Hidden voice commands. In 25th USENIX Security Symposium (USENIX Security 16), Austin, TX , 2016.
- Chen, P.-Y., Zhang, H., Sharma, Y., Yi, J., and Hsieh, C.J. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security , AISec '17, pp. 15-26, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-5202-4. doi: 10.1145/3128572.3140448. URL http://doi.acm.org/ 10.1145/3128572.3140448 . · DOI
- Dasgupta, S., Hsu, D., and Verma, N. A concentration theorem for projections. In Conference on Uncertainty in Artificial Intelligence , 2006.
- Evtimov, I., Eykholt, K., Fernandes, E., Kohno, T., Li, B., Prakash, A., Rahmati, A., and Song, D. Robust physical-world attacks on machine learning models. CoRR , abs/1707.08945, 2017. · arXiv
- Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR) , 2015.
- Gorban, A. N., Tyukin, I. Y., Prokhorov, D. V., and Sofeikov, K. I. Approximation with random bases. Inf. Sci. , 364(C): 129-145, October 2016. ISSN 0020-0255. doi: 10.1016/j.ins. 2015.09.021. URL http://dx.doi.org/10.1016/j. ins.2015.09.021 . · DOI
- Hayes, J. and Danezis, G. Machine learning as an adversarial service: Learning black-box adversarial examples. CoRR , abs/1708.05207, 2017. · arXiv
- Hosseini, H., Xiao, B., Jaiswal, M., and Poovendran, R. On the limitation of convolutional neural networks in recognizing negative images. 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) , pp. 352-358, 2017.
- Hu, W. and Tan, Y. Black-box attacks against RNN based malware detection algorithms. CoRR , abs/1705.08131, 2017. · arXiv
- Kurakin, A., Goodfellow, I., and Bengio, S. Adversarial examples in the physical world. 2016. URL https://arxiv.org/ abs/1607.02533 . · arXiv
- Liu, Y., Chen, X., Liu, C., and Song, D. Delving into transferable adversarial examples and black-box attacks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. 2017. URL https://arxiv.org/abs/1706.06083 . · arXiv
- Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. Deepfool: a simple and accurate method to fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.
- Narodytska, N. and Kasiviswanathan, S. P. Simple black-box adversarial perturbations for deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.
- Nesterov, Y. and Spokoiny, V. Random gradient-free minimization of convex functions. Found. Comput. Math. , 17(2):527566, April 2017. ISSN 1615-3375. doi: 10.1007/s10208-0159296-2. URL https://doi.org/10.1007/s10208015-9296-2 . · DOI
- Nguyen, A. M., Yosinski, J., and Clune, J. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CoRR , abs/1412.1897, 2014. · arXiv
- Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., and Swami, A. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security , ASIA CCS '17, pp. 506-519, New York, NY, USA, 2017. ACM. ISBN 978-14503-4944-4. doi: 10.1145/3052973.3053009. URL http: //doi.acm.org/10.1145/3052973.3053009 . · DOI · Paper page
- Salimans, T., Ho, J., Chen, X., and Sutskever, I. Evolution strategies as a scalable alternative to reinforcement learning. CoRR , abs/1703.03864, 2017. URL http://arxiv.org/abs/ 1703.03864 . · arXiv
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. 2013. URL https://arxiv.org/abs/ 1312.6199 .
- Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. 2015. URL https://arxiv.org/abs/1512.00567 . · arXiv
- Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters, J., and Schmidhuber, J. Natural evolution strategies. J. Mach. Learn. Res. , 15(1):949-980, January 2014. ISSN 15324435. URL http://dl.acm.org/citation.cfm? id=2627435.2638566 .
- Xu, W., Yi, Y., and Evans, D. Automatically evading classifiers: A case study on pdf malware classifiers. Network and Distributed System Security Symposium (NDSS) , 2016.
- Authors
- Ilyas, A., Engstrom, L., Athalye, A., & Lin, J.
- Affiliations
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1804.08598
- OpenAlex
- W2798801120
- Cited by
- 806
- References
- 30
- Citations by year
- 2026: 1 · 2025: 55 · 2024: 99 · 2023: 112 · 2022: 115 · 2021: 165 · 2020: 159 · 2019: 81 · 2018: 18 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- black-box adversarial attacks, threat model, adversarial examples, targeted attacks, Google Cloud Vision API, neural network classifier
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
