CoChat

Deep learning library testing via effective model generation

Wang, Zan, Yan, Ming, Chen, Junjie, Liu, Shuang, Zhang, Dongdi · 2020 · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.

Abstract

Deep learning (DL) techniques are rapidly developed and have been widely adopted in practice. However, similar to traditional software systems, DL systems also contain bugs, which could cause serious impacts especially in safety-critical domains. Recently, many research approaches have focused on testing DL models, while little attention has been paid for testing DL libraries, which is the basis of building DL models and directly affects the behavior of DL systems. In this work, we propose a novel approach, LEMON, to testing DL libraries. In particular, we (1) design a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors; and (2) propose a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs, so as to mitigate the impact of potential noise introduced by uncertain factors in DL libraries. We conducted an empirical study to evaluate the effectiveness of LEMON with 20 release versions of 4 widely-used DL libraries, i.e., TensorFlow, Theano, CNTK, MXNet. The results demonstrate that LEMON detected 24 new bugs in the latest release versions of these libraries, where 7 bugs have been confirmed and one bug has been fixed by developers. Besides, the results confirm that the heuristic strategy for model generation indeed effectively guides LEMON in amplifying the inconsistent degrees for bugs.

Overview

Integrity signals

Registry

Crossref · 10.1145/3368089.3409761

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 61 references

Research works printed but not linked: 11

  1. Chen Junjie, 2019, Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 223-234
  2. Chen Junjie, 2020, Characterizing and Prioritizing Incidents for LargeScale Online Service Systems. In The 35th IEEE/ACM International Conference on Automated Software Engineering.
  3. Guo Qianyu, 2019, Proceedings of the 34th International Conference on Automated Software Engineering.
  4. Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ). Fitash Ul Haq Donghwan Shin Shiva Nejati and Lionel Briand. 2019. Comparing Ofline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study. arXiv preprint arXiv:1912. 00805 ( 2019 ).
  5. Islam Md Johirul, 1906, A Comprehensive Study on Deep Learning Bug Characteristics. arXiv preprint arXiv
  6. 10.5555/3155562.3155636 · doi:10.5555/3155562.3155636
  7. Kass Robert E., 1998, Neal
  8. McKeeman William M, 1998, Diferential testing for software, Digital Technical Journal, 10
  9. Odena Augustus, 2018, Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv
  10. Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996. Yi Sun Yuheng Chen Xiaogang Wang and Xiaoou Tang. 2014. Deep learning face representation by joint identification-verification. In Advances in neural information processing systems. 1988-1996.
  11. Zhang Jie M, 1906, Machine Learning Testing: Survey, Landscapes and Horizons. arXiv preprint arXiv

Cited by

Items 201–221 of 221

Metadata

Authors
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D.
Affiliations
Tianjin University (CN)
Venue
Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Publisher
ACM
Published
2020-11-08
Pages
788-799
OpenAlex
W3104663419
Cited by
167
References
61
Citations by year
2026: 26 · 2025: 38 · 2024: 28 · 2023: 36 · 2022: 25 · 2021: 12 · 2020: 2
FWCI
12.44
Type
conference paper
Language
en
Open access
closed
Locations
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
Funders
  • National Natural Science Foundation of China (61872263,U1836214,61802275)
  • Intelligent Manufacturing Special Fund of Tianjin (20191012,20193155)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
mutation testing, TensorFlow, Theano, CNTK, MXNet, lemon

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.