CoChat

Fuzzing deep-learning libraries via automated relational API inference

Open access
Deng, Yinlin, Yang, Chenyuan, Wei, Anjiang, Zhang, Lingming · 2022 · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.

Abstract

Deep Learning (DL) has gained wide attention in recent years. Meanwhile, bugs in DL systems can lead to serious consequences, and may even threaten human lives. As a result, a growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, e.g., PyTorch and TensorFlow, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily “borrow” test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic/semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).

Overview

Integrity signals

Registry

Crossref · 10.1145/3540250.3549085

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

  • Deep Residual Learning for Image Recognition

    He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1×

    “Next, we use SentenceBERT [58] to encode these informative description sentences into semantically meaningful sentence embeddings.”
    Method
  • Speech recognition with deep recurrent neural networks

    Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks

    Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
  • The Limitations of Deep Learning in Adversarial Settings

    Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993

  • Algorithms for the Assignment and Transportation Problems

    Munkres, J. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 3,986

  • Deep Learning for Health Informatics

    Ravì, D., Wong, C., Deligianni, F., Berthelot, M., Andreu-Perez, J., Lo, B., & Yang, G.-Z. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2636665 · Cited by 2,000 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Deep learning for detecting robotic grasps

    Lenz, I., Lee, H., & Saxena, A. (2015) · The International Journal of Robotics Research · doi:10.1177/0278364914549607 · Cited by 1,634 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DeepTest: automated testing of deep-neural-network-driven autonomous cars

    Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [15, 20, 28, 42, 46, 50], metrics for model testing [32, 35, 41, 52, 75], applicationspecific model testing [61, 83, 86], and verifying certain properties of models [16, 39].”
    Method
  • Coverage-based Greybox Fuzzing as Markov Chain

    Böhme, M., Pham, V.-T., & Roychoudhury, A. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978428 · Cited by 649 · cited 1×

    “, various fuzzing techniques have been applied for 24h or even more [18, 40, 58, 66, 78] (including the recent LEMON work [64]).”
    Method
  • Multimodal deep learning for robust RGB-D object recognition

    Eitel, A., Springenberg, J. T., Spinello, L., Riedmiller, M., & Burgard, W. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353446 · Cited by 631 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems

    Zhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 1×

    “specific model testing [62, 85, 88], and verifying certain properties of models [15, 38].”
    BackgroundSelf-citation
  • DeepGauge: multi-granularity testing criteria for deep learning systems

    Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
  • Deep learning code fragments for code clone detection

    White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Deep code search

    Gu, X., Zhang, H., & Kim, S. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180167 · Cited by 547 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Untitled work

    (1998) · Cited by 469

  • Guiding Deep Learning System Testing Using Surprise Adequacy

    Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
  • DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization

    Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    BackgroundSelf-citation
  • A survey on deep learning for software engineering

    Yang, Y., Xia, X., Lo, D., & Grundy, J. (2022) · ACM Computing Surveys · doi:10.1145/3505243 · Cited by 223 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DLFix: context-based code transformation learning for automated program repair

    Li, Y., Wang, S., & Nguyen, T. N. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380345 · Cited by 209 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries

    Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190 · cited 11×

    “To reduce such threats, we adopt the number of covered APIs and detected bugs, following prior work on DL library testing [54, 65, 66].”
    BackgroundMethodInfluential
  • Deep learning library testing via effective model generation

    Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167 · cited 7×

    “To reduce such threats, we adopt the number of covered APIs and detected bugs, following prior work on DL library testing [54, 65, 66].”
    BackgroundMethodInfluential
  • Automated testing of graphics shader compilers

    Donaldson, A. F., Evrard, H., Lascu, A., & Thomson, P. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133917 · Cited by 160 · cited 1×

    “It is commonly used in prior work on fuzzing or testing [22, 61, 80].”
    Method
  • MemLock: memory usage guided fuzzing

    Wen, C., Wang, H., Li, Y., Qin, S., Liu, Y., Xu, Z., Chen, H., Xie, X., Pu, G., & Liu, T. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380396 · Cited by 160 · cited 1×

    “, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
    Method
  • Exploring API Embedding for API Usages and Applications

    Nguyen, T. D., Nguyen, A. T., Phan, H. D., & Nguyen, T. N. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.47 · Cited by 159 · cited 1×

    “Prior work [49, 50] proposed to train neural networks to mine API mappings by learning transformations between the vector spaces of APIs from different languages (e.”
    Background
  • DeepBillboard: systematic physical-world testing of autonomous driving systems

    Zhou, H., Li, W., Kong, Z., Guo, J., Zhang, Y., Yu, B., Zhang, L., & Liu, C. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380422 · Cited by 153 · cited 1×

    “specific model testing [62, 85, 88], and verifying certain properties of models [15, 38].”
    BackgroundSelf-citation
  • An extensive study on pre-trained models for program understanding and generation

    Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    BackgroundSelf-citation
  • DeepXplore: automated whitebox testing of deep learning systems

    Pei, K., Cao, Y., Yang, J., & Jana, S. (2019) · Communications of the ACM · doi:10.1145/3361566 · Cited by 150

  • Is neuron coverage a meaningful measure for testing deep neural networks?

    Harel-Canada, F., Wang, L., Gulzar, M. A., Gu, Q., & Kim, M. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409754 · Cited by 145 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
  • ProFuzzer: On-the-fly Input Type Probing for Better Zero-Day Vulnerability Discovery

    You, W., Wang, X., Ma, S., Huang, J., Zhang, X., Wang, X., & Liang, B. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00057 · Cited by 131 · cited 2×

    “, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
    Method
  • Practical, Low-Effort Equivalence Verification of Real Code

    Ramos, D. A., & Engler, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22110-1_55 · Cited by 126 · cited 2×

    “UC-KLEE [56] leverages symbolic execution to check two different implemen-”
    Method
  • Search-based inference of polynomial metamorphic relations

    Zhang, J., Chen, J., Hao, D., Xiong, Y., Xie, B., Zhang, L., & Mei, H. (2014) · Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/2642937.2642994 · Cited by 115

    Self-citation
  • Automatic generation of oracles for exceptional behaviors

    Goffi, A., Gorla, A., Ernst, M. D., & Pezzè, M. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931061 · Cited by 111 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • DeepDelta: learning to repair compilation errors

    Mesbah, A., Rice, A., Johnston, E., Glorioso, N., & Aftandilian, E. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3340455 · Cited by 107 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Free lunch for testing: fuzzing deep-learning libraries from open source

    Wei, A., Deng, Y., Yang, C., & Zhang, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510041 · Cited by 106 · cited 10×

    “We obtain the valid inputs traced from various input sources used by state-of-the-art FreeFuzz [65], which include documentations, developer tests, and 202 DL models in the wild,”
    BackgroundMethodInfluentialSelf-citation
  • Deep just-in-time defect prediction: how far are we?

    Zeng, Z., Zhang, Y., Zhang, H., & Zhang, L. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464819 · Cited by 102 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    BackgroundSelf-citation
  • MTFuzz: fuzzing with a multi-task neural network

    She, D., Krishna, R., Yan, L., Jana, S., & Ray, B. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409723 · Cited by 93 · cited 1×

    “, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
    Method
  • Audee: automated testing for deep learning frameworks

    Guo, Q., Xie, X., Li, Y., Zhang, X., Liu, Y., Li, X., & Shen, C. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416571 · Cited by 92 · cited 9×

    “Among those 106 confirmed bugs, only 9 can be found by FreeFuzz, and none of them can be detected by CRADLE, AUDEE, or LEMON.”
    BackgroundMethodInfluential
  • An SMT-Based Approach for Verifying Binarized Neural Networks

    Amir, G., Wu, H., Barrett, C., & Katz, G. (2021) · Lecture notes in computer science · doi:10.26226/morressier.604907f41a80aac83ca25cda · Cited by 84

  • DocTer: documentation-guided fuzzing for testing deep learning API functions

    Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., & Godfrey, M. W. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534220 · Cited by 81 · cited 1×

    “A very recent work DocTer [76] constructs rules to extract DL-specific input constraints from API documentation and uses them to generate valid/invalid inputs for testing DL libraries; meanwhile, it requires manual annotation for 30% of API parameters.”
    Method
  • Automated conformance testing for JavaScript engines via deep compiler fuzzing

    Ye, G., Tang, Z., Tan, S. H., Huang, S., Fang, D., Sun, X., Bian, L., Wang, H., & Wang, Z. (2021) · Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation · doi:10.1145/3453483.3454054 · Cited by 77 · cited 1×

    “Recent years have witnessed the surge of deep learning (DL) in a variety of applications, including computer vision [32, 60], natural language processing [24, 28], robotics [23, 35], bioinformatics [46, 57], and software engineering [25, 29, 36, 37, 45, 68, 78, 79, 81, 82].”
    Background
  • Correlations between deep neural network model coverage criteria and model quality

    Yan, S., Tao, G., Liu, X., Zhai, J., Ma, S., Xu, L., & Zhang, X. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409671 · Cited by 74 · cited 1×

    “To date, most prior work on DL testing focused on testing/verifying DL models, with an emphasis on adversarial attacks [14, 19, 27, 42, 47, 51], metrics for model testing [31, 34, 41, 53, 77], application-”
  • MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation

    Blasi, A., Gorla, A., Ernst, M. D., Pezzè, M., & Carzaniga, A. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111041 · Cited by 69 · cited 10×

    “We substituted the first three phases of DeepREL with MeMo, and found that it can infer only 17/28 API relations for TensorFlow/PyTorch (compared to 8808/4290 relations found by DeepREL), and can at most detect 8 (including 6 found by MeMo and 2 found by FreeFuzz on the APIs involved in the MeMo inferred relations) of…”
    BackgroundInfluential
  • Predoo: precision testing of deep learning operators

    Zhang, X., Sun, N., Fang, C., Liu, J., Liu, J., Chai, D., Jiang, W., & Chen, Z. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464843 · Cited by 68 · cited 1×

    “, Predoo [86] can only test 7 APIs for TensorFlow.”
    Background
  • Path-based function embedding and its application to error-handling specification mining

    DeFreez, D., Thakur, A. V., & Rubio-González, C. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236059 · Cited by 61 · cited 2×

    “Func2vec [21] is a technique for finding function synonyms in Linux file systems and drivers via learning function embeddings to improve the quality of error handling specifications for Linux code.”
    BackgroundMethod
  • Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs

    Su, T., Yan, Y., Wang, J., Sun, J., Xiong, Y., Pu, G., Wang, K., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485533 · Cited by 60 · cited 1×

    “It is commonly used in prior work on fuzzing or testing [22, 61, 80].”
    Method
  • Coverage-guided tensor compiler fuzzing with joint IR-pass mutation

    Liu, J., Wei, Y., Yang, S., Deng, Y., & Zhang, L. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3527317 · Cited by 51 · cited 1×

    “, various fuzzing techniques have been applied for 24h or even more [17, 40, 59, 67, 80] (including the recent LEMON work [65]).”
    MethodSelf-citation
  • Automated inference of likely metamorphic relations for model transformations

    Troya, J., Segura, S., & Ruiz–Cortés, A. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2017.05.043 · Cited by 49 · cited 1×

    “, TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the traditional metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [16, 26, 33, 43, 63, 75, 83, 84].”
    Background
  • Mapping API elements for code migration with vector representations

    Nguyen, T. D., Nguyen, A. T., & Nguyen, T. N. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2892661 · Cited by 46 · cited 1×

    “Prior work [49, 50] proposed to train neural networks to mine API mappings by learning transformations between the vector spaces of APIs from different languages (e.”
    Background
  • Automatic Discovery and Cleansing of Numerical Metamorphic Relations

    Zhang, B., Zhang, H., Chen, J., Hao, D., & Moscato, P. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00035 · Cited by 42 · cited 1×

    “, TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the traditional metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [16, 26, 33, 43, 63, 75, 83, 84].”
    Background
  • Search-based synthesis of equivalent method sequences

    Goffi, A., Gorla, A., Mattavelli, A., Pezzè, M., & Tonella, P. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635888 · Cited by 36 · cited 6×

    “For example, SBES [27, 43] synthesizes sequences of method invocations that are equivalent to a target method based on the observed dynamic program behaviors.”
    BackgroundMethodInfluential
  • Techniques for Automatic Detection of Metamorphic Relations

    Kanewala, U. (2014) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw.2014.62 · Cited by 31 · cited 1×

    “…if we treat the entire systems under test (e.g., TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [17, 27, 34, 43, 62, 73, 81, 82].”
  • Discovering discrepancies in numerical libraries

    Vanover, J., Deng, X., & Rubio-González, C. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397380 · Cited by 18 · cited 8×

    “Also, FPDiff can only target numerical library APIs taking double or int variables as input, whereas the input of APIs in DL libraries can be much more complex (e.g., tensors of different dimensions/types, objects, strings, etc.).”
    BackgroundInfluential
  • NeoFlow: A Flexible Framework for Enabling Efficient Compilation for High Performance DNN Training

    Zheng, S., Chen, R., Jin, Y., Wei, A., Wu, B., Li, X., Yan, S., & Liang, Y. E. (2021) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3138862 · Cited by 16 · cited 1×

    “Actually, the backward part needs to construct a very complex graph [87], but we omit it in Figure 1 for simplicity.”
    BackgroundSelf-citation
  • A Genetic Algorithm-Based Approach for Composite Metamorphic Relations Construction

    Xiang, Z., Wu, H., & Yu, F. (2019) · Information · doi:10.3390/info10120392 · Cited by 12 · cited 1×

    “, TensorFlow or PyTorch) as test objects, this work can also be viewed as an instance/extension of the traditional metamorphic oracle generation work, which aims to automatically infer metamorphic relations as test oracle [16, 26, 33, 43, 63, 75, 83, 84].”
    Background
  • Towards Zero Knowledge Learning for Cross Language API Mappings

    Bui, N. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00054 · Cited by 9 · cited 1×

    “Another work [18] leveraged an unsupervised domain adaptation approach to automatically construct and align vector spaces for identifying API mappings with much less human effort.”
    Background
  • Synthesis of Equivalent Method Calls in Guava

    Mattavelli, A., Goffi, A., & Gorla, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22183-0_19 · Cited by 6 · cited 6×

    “For example, SBES [27, 43] synthesizes sequences of method invocations that are equivalent to a target method based on the observed dynamic program behaviors.”
    BackgroundMethodInfluential

Research works printed but not linked: 12

  1. Martín Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , and Michael Isard . 2016 . Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. Tensorflow: A system for large-scale machine learning. In 12th $USENIX$ symposium on operating systems design and implementation ($OSDI$ 16). 265–283.
  2. Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. Nicholas Carlini Anish Athalye Nicolas Papernot Wieland Brendel Jonas Rauber Dimitris Tsipras Ian Goodfellow Aleksander Madry and Alexey Kurakin. 2019. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705. · arXiv
  3. Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. · arXiv
  4. Felix A Gers , Jürgen Schmidhuber , and Fred Cummins . 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 ( 2000 ), 2451–2471. Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. Neural computation, 12, 10 (2000), 2451–2471.
  5. Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. · arXiv
  6. Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. Changliu Liu Tomer Arnon Christopher Lazarus Christopher Strong Clark Barrett and Mykel J Kochenderfer. 2019. Algorithms for verifying deep neural networks. arXiv preprint arXiv:1903.06758. · arXiv
  7. Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. Jiawei Liu Jinkun Lin Fabian Ruffy Cheng Tan Jinyang Li Aurojit Panda and Lingming Zhang. 2022. Finding Deep-Learning Compilation Bugs with NNSmith. arXiv preprint arXiv:2207.13066. · arXiv
  8. Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. · arXiv
  9. Seonwoo Min , Byunghan Lee , and Sungroh Yoon . 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 ( 2017 ), 851–869. Seonwoo Min, Byunghan Lee, and Sungroh Yoon. 2017. Deep learning in bioinformatics. Briefings in bioinformatics, 18, 5 (2017), 851–869.
  10. Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , and Luca Antiga . 2019 . Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32 (2019), 8026–8037.
  11. Martin F Porter . 2001 . Snowball: A language for stemming algorithms. Martin F Porter. 2001. Snowball: A language for stemming algorithms.
  12. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. · arXiv

Cited by

See all 106

Metadata

Authors
Deng, Y., Yang, C., Wei, A., & Zhang, L.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Stanford University (US)
Venue
Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Publisher
ACM
Published
2022-11-07
Pages
44-56
OpenAlex
W4308643012
Cited by
86
References
57
Citations by year
2026: 17 · 2025: 34 · 2024: 20 · 2023: 15
FWCI
13.01
Type
conference paper
Language
en
Open access
Locations
  • ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (CCF-2131943,CCF-2141474)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
fuzzing, deep learning libraries, API testing, PyTorch, TensorFlow, bug detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.