Intriguing properties of neural networks
Open accessAbstract
Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn uninterpretable solutions that could have counter-intuitive properties. In this paper we report two such properties. First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains of the semantic information in the high layers of neural networks. Second, we find that deep neural networks learn input-output mappings that are fairly discontinuous to a significant extend. We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error. In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
DP can degrade perception model precision, increasing false negatives in cluttered environments.
“DP can degrade perception model precision -as noise in general can reduce detection or bounding-box accuracy [53, 133], increasing false negatives in cluttered environments.”
From Privacy-preserving Machine Learning in Internet of Vehicle Applications: Fundamentals, Recent Advances, and Future Direction (2025) · cites this paper as [133]
The Introduction (p. 2) reports that an imperceptible, deliberately optimized image perturbation can change an object-recognition network’s prediction. That supports the general possibility of perturbation-induced recognition errors, but the paper does not establish that DP noise degrades detection or bounding-box accuracy, or that it increases false negatives in cluttered environments. (LLM, conf=0.84)
Experimental — accuracy not yet measuredDivergence between epistemic standards seems entirely legitimate.
“Divergence between epistemic standards seems entirely legitimate, for instance, given widespread debates about the soundness and robustness of machine learning models”
From Algorithmic Accountability and Public Reason (2018) · cites this paper as Szegedy et al., 2013
Section 4, p. 4, shows that small, imperceptible changes can cause deep neural networks to misclassify images, supporting the sentence’s reference to concerns about model robustness. The paper does not discuss divergence between epistemic standards or establish that debates about model soundness and robustness are widespread; the legitimacy of such divergence is the citing authors’ own judgment. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThere are cases in which the lack of a proper understanding of the model might drive the user toward incorrect assumptions and fatal consequences.
“There are cases in which the lack of a proper understanding of the model might drive the user toward incorrect assumptions and fatal consequences [44, 164].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [164]
The Introduction (pp. 1–2) says neural networks can be difficult to interpret and have counter-intuitive properties. Section 4 (p. 4) shows that an expectation of robustness to tiny image changes can be wrong. The paper does not discuss users making incorrect assumptions or those assumptions leading to fatal consequences. (LLM, conf=0.85)
Experimental — accuracy not yet measuredAdvanced detection methods mainly rely on DNN-based binary classification models, which are known to be vulnerable to adversarial examples.
“the advanced detection methods mainly rely on DNN-based binary classiication models, which are known to be vulnerable to adversarial examples”
From DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models (2024) · cites this paper as [66]
The Introduction (p. 2) reports that imperceptible perturbations can change a deep neural network’s prediction, and the Experimental results (p. 6) include adversarial examples for a binary car classifier. These passages support the vulnerability claim, but do not establish that advanced detection methods mainly rely on DNN-based binary classifiers. (LLM, conf=0.86)
Experimental — accuracy not yet measuredAI exhibits strong pattern recognition capabilities, relying on different features and mechanisms than humans, thereby likely failing for reasons distinct from those affecting human performance.
“Moreover, AI exhibits strong pattern recognition capabilities, relying on different features and mechanisms than humans, thereby likely failing for reasons distinct from those affecting human performance [40] .”
From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [40]
The Abstract (p. 1) reports strong recognition performance and shows that hardly perceptible changes can make a neural network misclassify an image. The Introduction (p. 2) describes this as an intrinsic blind spot. These passages support the suggestion that network failures can differ from human perception, but the paper does not establish that AI generally relies on different features and mechanisms than humans. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Deep neural networks are vulnerable to adversarial examples, which are carefully chosen inputs that cause the network to change output without a visible change to a human.
“On the other hand, it is worth noting that researchers have recently that noticed deep neural networks are vulnerable to adversarial examples, which are carefully chosen inputs that cause the network to change output without a visible change to a human (Fig. 15) [60–63].”
From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [60]
Adversarial attacks have been widely studied against ML models under both white-box and black-box assumptions.
“Adversarial attacks have been widely studied against ML models [47,50] under both white-box [9, 21, 31, 55] and black-box assumptions [33, 43].”
From Adversarial Detection Avoidance Attacks: Evaluating the robustness of perceptual hashing-based client-side scanning (2021) · cites this paper as [55]
These studies indicate that the way artificial networks see and predict is different from the way we do.
“Although the impact of adversarial examples in the medical domain is unknown, these studies indicate that the way artificial networks see and predict is different from the way we do.”
From Convolutional neural networks: an overview and application in radiology (2018) · cites this paper as [60]
Potential solutions include federated distillation and optimizing workers' scheduling policy.
“Potential solutions to address this challenge include federated distillation (FD), in which workers exchange their model output parameters (logits) as opposed to the model parameters (gradients and/weights), and optimizing workers' scheduling policy with appropriate communication and computing resources [248, 368, 402].”
From Advances and Open Problems in Federated Learning (2021) · cites this paper as [402]
The passages in §4.2 (pp. 5–8) concern adversarial examples and their generalization across neural networks, not federated learning. Searches for federated distillation, workers exchanging logits, and scheduling with communication or computing resources found no passage addressing the attributed solutions. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 6 · Cited by 8,143
Research works printed but not linked: 10
- [1][1] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and KlausRobert M¨ uller. How to explain individual classification decisions. The Journal of Machine Learning Research , 99:1803-1831, 2010. -
- [2][2] Yoshua Bengio. Learning deep architectures for ai. Foundations and trends® in Machine Learning , 2(1):1-127, 2009. -
- [3][3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on , pages 248-255. IEEE, 2009. -
- [4][4] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal, June 2009. Also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montr´ eal, Canada. -
- [5][5] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively trained, multiscale, deformable part model. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on , pages 1-8. IEEE, 2008. -
- [6][6] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv preprint arXiv:1311.2524 , 2013. - · arXiv
- [7][7] Ian Goodfellow, Quoc Le, Andrew Saxe, Honglak Lee, and Andrew Y Ng. Measuring invariances in deep networks. Advances in neural information processing systems , 22:646-654, 2009. -
- [9][9] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 , pages 1106-1114, 2012. -
- [10][10] Quoc V Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S Corrado, Jeff Dean, and Andrew Y Ng. Building high-level features using large scale unsupervised learning. arXiv preprint arXiv:1112.6209 , 2011. - · arXiv
- [13][13] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional neural networks. arXiv preprint arXiv:1311.2901 , 2013. · arXiv
- Authors
- Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R.
- Affiliations
- Google (United States) (US)
- New York University (US)
- Université de Montréal (CA)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2013
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1312.6199
- OpenAlex
- W1673923490
- Cited by
- 8,143
- References
- 6
- Citations by year
- 2026: 31 · 2025: 501 · 2024: 1192 · 2023: 1372 · 2022: 1270 · 2021: 1321 · 2020: 1079 · 2019: 725 · 2018: 383 · 2017: 145 · 2016: 70 · 2015: 37 · 2014: 5 · 2012: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- deep neural network, interpretability, unit analysis, adversarial perturbations, imperceptible perturbations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
