CoChat

Fuzzing deep-learning libraries via automated relational API inference

Open access
Deng, Yinlin, Yang, Chenyuan, Wei, Anjiang, Zhang, Lingming · 2022 · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.

Abstract

Deep Learning (DL) has gained wide attention in recent years. Meanwhile, bugs in DL systems can lead to serious consequences, and may even threaten human lives. As a result, a growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, e.g., PyTorch and TensorFlow, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily “borrow” test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic/semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).

Overview

Integrity signals

Registry

Crossref · 10.1145/3540250.3549085

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

  • Deep Residual Learning for Image Recognition

    He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×

    “Next, we use SentenceBERT [58] to encode these informative description sentences into semantically meaningful sentence embeddings.”
    Method
  • Speech recognition with deep recurrent neural networks

    Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks

    Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
  • The Limitations of Deep Learning in Adversarial Settings

    Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993

  • Algorithms for the Assignment and Transportation Problems

    Munkres, J. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 3,986

  • Deep Learning for Health Informatics

    Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Deep learning for detecting robotic grasps

    Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DeepTest: automated testing of deep-neural-network-driven autonomous cars

    Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [15, 20, 28, 42, 46, 50], metrics for model testing [32, 35, 41, 52, 75], applicationspecific model testing [61, 83, 86], and verifying certain properties of models [16, 39].”
    Method

Show all 58 references

Research works printed but not linked: 12

  1. Martín Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , and Michael Isard . 2016 . Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283.
  2. Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. · arXiv
  3. Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. · arXiv
  4. Felix A Gers , Jürgen Schmidhuber , and Fred Cummins . 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 ( 2000 ), 2451–2471. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 (2000), 2451–2471.
  5. Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. · arXiv
  6. Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. · arXiv
  7. Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. · arXiv
  8. Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. · arXiv
  9. Seonwoo Min , Byunghan Lee , and Sungroh Yoon . 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 ( 2017 ), 851–869. Seonwoo Min, Byunghan Lee, and Sungroh Yoon. 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 (2017), 851–869.
  10. Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , and Luca Antiga . 2019 . Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037.
  11. Martin F Porter . 2001 . Snowball: A language for stemming algorithms. Martin F Porter. 2001. Snowball: A language for stemming algorithms.
  12. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv

Cited by

Items 101–106 of 106

Metadata

Authors
Deng, Y., Yang, C., Wei, A., & Zhang, L.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Stanford University (US)
Venue
Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Publisher
ACM
Published
2022-11-07
Pages
44-56
OpenAlex
W4308643012
Cited by
86
References
57
Citations by year
2026: 17 · 2025: 34 · 2024: 20 · 2023: 15
FWCI
13.01
Type
conference paper
Language
en
Open access
Locations
  • ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (CCF-2131943,CCF-2141474)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
fuzzing, deep learning libraries, API testing, PyTorch, TensorFlow, bug detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.