The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
An empirical study on TensorFlow program bugs
Abstract
Deep learning applications become increasingly popular in important domains such as self-driving systems and facial identity systems. Defective deep learning applications may lead to catastrophic consequences. Although recent research efforts were made on testing and debugging deep learning applications, the characteristics of deep learning defects have never been studied. To fill this gap, we studied deep learning applications built on top of TensorFlow and collected program bugs related to TensorFlow from StackOverflow QA pages and Github projects. We extracted information from QA pages, commit messages, pull request messages, and issue discussions to examine the root causes and symptoms of these bugs. We also studied the strategies deployed by TensorFlow users for bug detection and localization. These findings help researchers and TensorFlow users to gain a better understanding of coding defects in TensorFlow programs and point out a new direction for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2016) · Cited by 11,679
(2017) · Cited by 11,625
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819
Static Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
Self-citation(2000) · Cited by 1,467
DeepXplore: Automated Whitebox Testing of Deep Learning Systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2017) · Proceedings of the 26th Symposium on Operating Systems Principles · doi:10.1145/3132747.3132785 · Cited by 1,217
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Hand, D. J. (2007) · Drug Safety · doi:10.2165/00002018-200730070-00010 · Cited by 1,044
Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
Hernández, M. A., & Stolfo, S. J. (1998) · Data Mining and Knowledge Discovery · doi:10.1023/a:1009761603038 · Cited by 812
Testing and validating machine learning classifiers by metamorphic testing
Xie, X., Ho, J. W. K., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2010.11.920 · Cited by 389
A brief survey of program slicing
Xu, B., Qian, J., Zhang, X., Wu, Z., & Chen, L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1050849.1050865 · Cited by 313
Understanding and detecting real-world performance bugs
Jin, G., Song, L., Shi, X., Scherpelz, J., & Lu, S. (2012) · Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2254064.2254075 · Cited by 304
What makes a good code example?: A study of programming Q&A in StackOverflow
Nasehi, S. M., Sillito, J., Maurer, F., & Burns, C. (2012) · 2012 28th IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2012.6405249 · Cited by 287
DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.08559 · Cited by 208
What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems
Gunawi, H. S., Hao, M., Leesatapornwongsa, T., Patana-anake, T., Do, T., Adityatama, J., Eliazar, K. J., Laksono, A., Lukman, J. F., Martin, V., et al. (2014) · Proceedings of the ACM Symposium on Cloud Computing · doi:10.1145/2670979.2670986 · Cited by 205
The Impact of API Change- and Fault-Proneness on the User Ratings of Android Apps
Bavota, G., Linares-Vasquez, M., Bernal-Cardenas, C. E., Penta, M. D., Oliveto, R., & Poshyvanyk, D. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2367027 · Cited by 186
Data Cleansing: Beyond Integrity Analysis 1
Maletic, J. I., & Marcus, A. (2000) · Cited by 182
Daran, M., & Thévenod-Fosse, P. (1996) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/226295.226313 · Cited by 154
An Empirical Study of Bugs in Machine Learning Systems
Thung, F., Wang, S., Lo, D., & Jiang, L. (2012) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2012.22 · Cited by 149
An empirical comparison of compiler testing techniques
Chen, J., Hu, W., Hao, D., Xiong, Y., Zhang, H., Zhang, L., & Xie, B. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884878 · Cited by 138
Self-citationTitian: data provenance support in Spark
Interlandi, M., Shah, K., Tetali, S. D., Gulzar, M. A., Yoo, S., Kim, M., Millstein, T. D., & Condie, T. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2850583.2850595 · Cited by 130
Titian: Data Provenance Support in Spark.
Interlandi, M., Shah, K., Tetali, S. D., Gulzar, M. A., Yoo, S., Kim, M., Millstein, T. D., & Condie, T. (2015) · PubMed · Cited by 107
Learning to Prioritize Test Programs for Compiler Testing
Chen, J., Bai, Y., Hao, D., Xiong, Y., Zhang, H., & Xie, B. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.70 · Cited by 90
Self-citationAn analysis of test data selection criteria using the RELAY model of fault detection
Richardson, D. J., & Thompson, M. C. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.232020 · Cited by 87
An empirical study of dormant bugs
Chen, T.-H. P., Nagappan, M., Shihab, E., & Hassan, A. E. (2014) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2597073.2597108 · Cited by 84
Seaman, C. B., Shull, F. J., Regardie, M., Elbert, D., Feldmann, R. L., Guo, Y., & Godfrey, S. (2008) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1414004.1414030 · Cited by 77
Avoiding coincidental correctness in boundary value analysis
Hierons, R. M. (2006) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/1151695.1151696 · Cited by 48
An empirical study on the impact of refactoring activities on evolving client-used APIs
Kula, R. G., Ouni, A., Germán, D. M., & Inoue, K. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.09.007 · Cited by 48
Multiple-Implementation Testing of Supervised Learning Software
Srisakaokul, S., Wu, Z., Astorga, A., Alebiosu, O., & Xie, T. (2016) · National Conference on Artificial Intelligence · Cited by 44
Marick, B. (1991) · doi:10.1145/120807.120825 · Cited by 41
LAMP: data provenance for graph based machine learning algorithms through derivative computation
Ma, S., Aafer, Y., Xu, Z., Lee, W., Zhai, J., Liu, Y., & Zhang, X. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106291 · Cited by 20
Model-based testing of probabilistic systems
Gerhold, M., & Stoelinga, M. I. A. (2018) · Formal Aspects of Computing · doi:10.1007/s00165-017-0440-4 · Cited by 19
(1990) · doi:10.1007/978-3-7091-8990-0_3 · Cited by 16
Cited by
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Software Engineering for AI-Based Systems: A Survey
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 335
A comprehensive study on deep learning bug characteristics
Islam, M. J., Nguyen, G., Pan, R., & Rajan, H. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338955 · Cited by 305
Taxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380395 · Cited by 291
Testing machine learning based systems: a systematic mapping
Riccio, V., Jahangirova, G., Stocco, A., Humbatova, N., Weiss, M., & Tonella, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09881-0 · Cited by 251
Metadata
- Authors
- Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L.
- Venue
- Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2018-07-12
- Pages
- 129-140
- OpenAlex
- W2850992922
- Cited by
- 304
- References
- 37
- FWCI
- 17.00
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.