The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
DLFuzz: differential fuzzing testing of deep learning systems
Open accessAbstract
Deep learning (DL) systems are increasingly applied to safety-critical domains such as autonomous driving cars. It is of significant importance to ensure the reliability and robustness of DL systems. Existing testing methodologies always fail to include rare inputs in the testing dataset and exhibit low neuron coverage. In this paper, we propose DLFuzz, the first differential fuzzing testing framework to guide DL systems exposing incorrect behaviors. DLFuzz keeps minutely mutating the input to maximize the neuron coverage and the prediction difference between the original input and the mutated input, without manual labeling effort or cross-referencing oracles from other DL systems with the same functionality. We present empirical evaluations on two well-known datasets to demonstrate its efficiency. Compared with DeepXplore, the state-of-the-art DL whitebox testing framework, DLFuzz does not require extra efforts to find similar functional DL systems for cross-referencing check, but could generate 338.59% more adversarial inputs with 89.82% smaller perturbations, averagely obtain 2.86% higher neuron coverage, and save 20.11% time consumption.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Self-citationVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“g CNNs used by DeepXplore for image classification tasks to evaluate DLFuzz. MNIST [8] is a large database of handwritten digits consisting of 60000 training images and 10000 testing images. ImageNet [2] is a large visual database containing over 14 million images for object recognition. The same as DeepXplore, DLFuzz…”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 4×
“Although DL systems exhibit impressive performance on image classification tasks, the classifiers can also be easily led to incorrect classifications by applying imperceptible perturbations [12], as shown in Figure 1.”
BackgroundMethodInfluentialEnd to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
DeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217
GradientBased Learning Applied to Document Recognition
(2009) · Intelligent Signal Processing · doi:10.1109/9780470544976.ch9 · Cited by 1,135
Connectomic reconstruction of the inner plexiform layer in the mouse retina
Helmstaedter, M., Briggman, K. L., Turaga, S. C., Jain, V., Seung, H. S., & Denk, W. (2013) · Nature · doi:10.1038/nature12346 · Cited by 997
Safety Verification of Deep Neural Networks
Huang, X., Kwiatkowska, M., Wang, S., & Wu, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-63387-9_1 · Cited by 818
Towards Deep Neural Network Architectures Robust to Adversarial Examples
Gu, S., & Rigazio, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.5068 · Cited by 622
Droid-Sec: deep learning in android malware detection
Yuan, Z., Lu, Y., Wang, Z., & Xue, Y. (2014) · Proceedings of the 2014 ACM conference on SIGCOMM · doi:10.1145/2619239.2631434 · Cited by 371
Feature-Guided Black-Box Safety Testing of Deep Neural Networks
Wicker, M., Huang, X., & Kwiatkowska, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-89960-2_22 · Cited by 212
DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.08559 · Cited by 208
Droid-Sec: deep learning in android malware detection
Yuan, Z., Lu, Y., Wang, Z., & Xue, Y. (2015) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2740070.2631434 · Cited by 167
Fuzz testing in practice: Obstacles and solutions
Liang, J., Wang, M., Chen, Y., Jiang, Y., & Zhang, R. (2018) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2018.8330260 · Cited by 55
Self-citation
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
(2021) · doi:10.1109/tse.2019.2946563 · Cited by 574
Huang, X., Kroening, D., Ruan, W., Sharp, J., Sun, Y., Thamo, E., Wu, M., & Yi, X. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100270 · Cited by 532
Qayyum, A., Usama, M., Qadir, J., & Al-Fuqaha, A. (2020) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.2975048 · Cited by 257
“In [222], the first approach utilizing differential fuzzing testing is presented for exploiting incorrect behavior of DL systems.”
MethodTesting machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
“Interestingly, all of the GPU models reported in the papers are NVIDIA products, with the GeForce series being mentioned five times [7, 9, 16, 19, 42] and the Tesla series four times [12, 26, 27, 63].”
BackgroundMethodInfluential
Metadata
- Authors
- Guo, J., Jiang, Y., Zhao, Y., Chen, Q., & Sun, J.
- Venue
- Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2018-10-26
- Pages
- 739-743
- OpenAlex
- W2888824816
- Cited by
- 281
- References
- 21
- FWCI
- 16.31
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.