DocTer: documentation-guided fuzzing for testing deep learning API functions
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
Input constraints are useful for many software development tasks. For example, input constraints of a function enable the generation of valid inputs, i.e., inputs that follow these constraints, to test the function deeper. API functions of deep learning (DL) libraries have DL-specific input constraints, which are described informally in the free-form API documentation. Existing constraint-extraction techniques are ineffective for extracting DL-specific input constraints.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We include both PyTorch (v1.12) [57] and TensorFlow (v2.10) [66], since they are two of the most popular DL libraries and are widely studied in prior DL library testing work [70,72,77].”
From Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models (2023) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 3×
“, TensorFlow [14] and PyTorch [39]) is crucial because these libraries are widely used and contain software bugs [26, 27, 42, 58, 59], which hurt not only the development but also the accuracy and speed of the DL models.”
BackgroundMethodTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037 · cited 2×
“Dependency tree parsing: Once the sentences are normalized, they are fed to the dependency tree parser [33], which conducts POSTagging and builds tree structure relationships (i.”
BackgroundMethodNatural language processing with Python
Bird, S., Klein, E., & Loper, E. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,955
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819 · cited 1×
“• A document-analysis technique that extracts 16,035 constraints automatically from API documentation with the focus on four categories of input properties in DL APIs: structure, dtype, shape, and valid values for 2,415 API functions across the three widelyusedDL libraries, TensorFlow [14], PyTorch [39], andMXNet [17].”
MethodSelf-citationStatic Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
Fraser, G., & Arcuri, A. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.14 · Cited by 586 · cited 1×
“Existing test-generation tools [3, 5, 9, 19, 37, 44] such as AFL [3] and libFuzzer [5] have no knowledge of such input constraints, thus are very limited in testing DL API functions.”
MethodRandoop: feedback-directed random testing for Java
Pacheco, C., & Ernst, M. D. (2007) · Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion · doi:10.1145/1297846.1297902 · Cited by 558
FairFuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage
Lemieux, C., & Sen, K. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238176 · Cited by 477 · cited 1×
“Fuzzers [3, 5, 9] have been adopted to test non-DL libraries [29, 41].”
MethodDeepHunter: a coverage-guided fuzz testing framework for deep neural networks
Xie, X., Ma, L., Juefei-Xu, F., Xue, M., Chen, H., Liu, Y., Zhao, J., Li, B., Yin, J., & See, S. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330579 · Cited by 426 · cited 1×
“Testing DL models: Many fuzzing techniques test the robustness of DLmodels instead of DL libraries [20, 23, 25, 36, 48, 50, 54].”
MethodEfficiently mining frequent trees in a forest: algorithms and applications
Zaki, M. J. (2005) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2005.125 · Cited by 306 · cited 2×
“Specifically, TreeMiner [56] collects 873, 426, and 321 frequent parse subtrees from the three libraries respectively with the corresponding min_support.”
BackgroundMethodA comprehensive study on deep learning bug characteristics
Islam, M. J., Nguyen, G., Pan, R., & Rajan, H. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338955 · Cited by 305 · cited 2×
“, TensorFlow [14] and PyTorch [38]) is crucial because these libraries are widely used and contain software bugs [26, 27, 41, 57, 58], which hurt not only the development but also the accuracy and speed of the DL models.”
BackgroundMethodAn empirical study on TensorFlow program bugs
Zhang, Y., Chen, Y., Cheung, S.-C., Xiong, Y., & Zhang, L. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213866 · Cited by 304 · cited 1×
“, TensorFlow [14] and PyTorch [38]) is crucial because these libraries are widely used and contain software bugs [26, 27, 41, 57, 58], which hurt not only the development but also the accuracy and speed of the DL models.”
BackgroundT-Fuzz: Fuzzing by Program Transformation
Peng, H., Shoshitaishvili, Y., & Payer, M. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00056 · Cited by 293 · cited 1×
“Fuzzers [3, 5, 9] have been adopted to test non-DL libraries [29, 41].”
MethodTaxonomy of real faults in deep learning systems
Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V., Stocco, A., & Tonella, P. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380395 · Cited by 291 · cited 2×
“, TensorFlow [14] and PyTorch [38]) is crucial because these libraries are widely used and contain software bugs [26, 27, 41, 57, 58], which hurt not only the development but also the accuracy and speed of the DL models.”
BackgroundMethodDLFuzz: differential fuzzing testing of deep learning systems
Guo, J., Jiang, Y., Zhao, Y., Chen, Q., & Sun, J. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264835 · Cited by 281 · cited 1×
“Testing DL models: Many fuzzing techniques test the robustness of DLmodels instead of DL libraries [20, 23, 25, 36, 48, 50, 54].”
Method/*icomment: bugs or bad comments?*/
Tan, L., Yuan, D., Krishna, G., & Zhou, Y. (2007) · Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles · doi:10.1145/1294261.1294276 · Cited by 268
Self-citation@tComment: Testing Javadoc Comments to Detect Comment-Code Inconsistencies
Tan, S. H., Marinov, D., Tan, L., & Leavens, G. T. (2012) · 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2012.106 · Cited by 197 · cited 5×
“Constraint extraction: Existing constraint-extraction techniques are insufficient for extracting DL-specific constraints [7, 16, 31, 47, 53, 57, 62], because they miss most of the DL-specific constraints, cannot analyze across Python and C++ code, or requires formal specifications.”
BackgroundMethodResultInfluentialSelf-citationInferring Resource Specifications from Natural Language API Documentation
Zhong, H., Zhang, L., Xie, T., & Mei, H. (2009) · 2009 IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2009.94 · Cited by 190
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190 · cited 3×
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 42, 43, 50, 52] or oracle approximation [35, 60].”
BackgroundMethodSelf-citationInferring method specifications from natural language API descriptions
Pandita, R., Xiao, X., Zhong, H., Xie, T., Oney, S., & Paradkar, A. M. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227137 · Cited by 186
Deep learning library testing via effective model generation
Wang, Z., Yan, M., Chen, J., Liu, S., & Zhang, D. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3409761 · Cited by 167 · cited 1×
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 42, 43, 50, 52] or oracle approximation [35, 60].”
MethodGrammar-based whitebox fuzzing
Godefroid, P., Kieżun, A., & Levin, M. Y. (2008) · ACM SIGPLAN Notices · doi:10.1145/1379022.1375607 · Cited by 167
An empirical study on program failures of deep learning jobs
Zhang, R., Xiao, W., Zhang, H., Liu, Y., Lin, H., & Yang, M. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380362 · Cited by 143 · cited 1×
“, TensorFlow [14] and PyTorch [39]) is crucial because these libraries are widely used and contain software bugs [26, 27, 42, 58, 59], which hurt not only the development but also the accuracy and speed of the DL models.”
BackgroundDeepMutation++: A Mutation Testing Framework for Deep Learning Systems
Hu, Q., Ma, L., Xie, X., Yu, B., Liu, Y., & Zhao, J. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00126 · Cited by 136 · cited 1×
“Testing DL models: Many fuzzing techniques test the robustness of DLmodels instead of DL libraries [20, 23, 25, 36, 49, 51, 55].”
MethodTensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
Odena, A., & Goodfellow, I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.10875 · Cited by 127 · cited 1×
“Testing DL models: Many fuzzing techniques test the robustness of DLmodels instead of DL libraries [20, 23, 25, 36, 49, 51, 55].”
MethodTranslating code comments to procedure specifications
Blasi, A., Goffi, A., Kuznetsov, K., Gorla, A., Ernst, M. D., Pezzè, M., & Castellanos, S. D. (2018) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3213846.3213872 · Cited by 122 · cited 8×
“Existing techniques that derive constraints from software text extract different types of constraints that are not DLspecific, such as exceptions [16, 56], command-line options and file formats [52], locking [44], call-relations [31, 44], interrupts [45], nullness [46, 60] and inheritance relations [60].”
BackgroundMethodInfluentialInferring method specifications from natural language API descriptions
Pandita, R., Xiao, X., Zhong, H., et al. (2012) · doi:10.5555/2337223.2337319 · Cited by 116
Fuzz testing based data augmentation to improve robustness of deep neural networks
Gao, X., Saha, R. K., Prasad, M. R., & Roychoudhury, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380415 · Cited by 115 · cited 1×
“Testing DL models: Many fuzzing techniques test the robustness of DLmodels instead of DL libraries [20, 23, 25, 36, 49, 51, 55].”
MethodAutomatic generation of oracles for exceptional behaviors
Goffi, A., Gorla, A., Ernst, M. D., & Pezzè, M. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931061 · Cited by 111 · cited 2×
“Some prior work translates software specifications into assertions [30] and oracles [22, 34].”
MethodaComment: mining annotations from comments and code to detect interrupt related concurrency bugs
Tan, L., Zhou, Y., & Padioleau, Y. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985796 · Cited by 93
Self-citationAudee: automated testing for deep learning frameworks
Guo, Q., Xie, X., Li, Y., Zhang, X., Liu, Y., Li, X., & Shen, C. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416571 · Cited by 92 · cited 2×
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 42, 43, 50, 52] or oracle approximation [35, 60].”
MethodSelf-citationFuzzFactory: domain-specific fuzzing with waypoints
Padhye, R., Lemieux, C., Sen, K., Simon, L. J., & Vijayakumar, H. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360600 · Cited by 76
A Study of Oracle Approximations in Testing Deep Learning Libraries
Nejadgholi, M., & Yang, J. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00078 · Cited by 68 · cited 1×
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 42, 43, 50, 52] or oracle approximation [35, 60].”
MethodC2S: translating natural language comments to formal program specifications
Zhai, J., Shi, Y., Pan, M.-X., Zhou, G., Liu, Y., Fang, C., Ma, S., Tan, L., & Zhang, X. (2020) · ESEC/SIGSOFT FSE · doi:10.1145/3368089.3409716 · Cited by 68 · cited 6×
“Constraint extraction: Existing constraint-extraction techniques are insufficient for extracting DL-specific constraints [7, 16, 31, 47, 53, 57, 62], because they miss most of the DL-specific constraints, cannot analyze across Python and C++ code, or requires formal specifications.”
BackgroundMethodInfluentialSelf-citationAutomatic Detection and Repair Recommendation of Directive Defects in Java API Documentation
Zhou, Y., Wang, C., Yan, X., Chen, T., Panichella, S., & Gall, H. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2872971 · Cited by 51
Testing probabilistic programming systems
Dutta, S., Legunsen, O., Huang, Z., & Misailovic, S. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236057 · Cited by 49 · cited 2×
“…constraint-extraction techniques Analyzing software text to detect bugs: Prior work leverages numerical [18, 24] (e.g., returns NaN/Inf ), decreased accuracy [18], and Testing DL models: Many fuzzing techniques test the robustness of DL models instead of DL libraries [20, 23, 25, 36, 49, 51, 55].”
MethodDirected test generation using symbolic grammars
Majumdar, R., & Xu, R.-G. (2007) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1287624.1287708 · Cited by 48
Multiple-Implementation Testing of Supervised Learning Software
Srisakaokul, S., Wu, Z., Astorga, A., Alebiosu, O., & Xie, T. (2016) · National Conference on Artificial Intelligence · Cited by 44
Automatically Generating Precise Oracles from Structured Natural Language Specifications
Motwani, M., & Brun, Y. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00035 · Cited by 44 · cited 1×
“Some prior work translates software specifications into assertions [30] and oracles [22, 34].”
MethodLv, T., Li, R., Yang, Y., Chen, K., Liao, X., Wang, X., Hu, P., & Xing, L. (2020) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3372297.3423360 · Cited by 43 · cited 5×
“Constraint extraction: Existing constraint-extraction techniques are insufficient for extracting DL-specific constraints [7, 16, 31, 47, 53, 57, 62], because they miss most of the DL-specific constraints, cannot analyze across Python and C++ code, or requires formal specifications.”
BackgroundMethodInfluentialWang, H., Sun, D. L., & Xing, E. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017136 · Cited by 41
DASE: Document-Assisted Symbolic Execution for Improving Automated Software Testing
Wong, E., Zhang, L., Wang, S., Liu, T., & Tan, L. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.78 · Cited by 37 · cited 8×
“This confirms that to detect real-world bugs effectively, one does not need to have complete constraints [53].”
BackgroundMethodInfluentialSelf-citationAutomatic early defects detection in use case documents
Liu, S., Sun, J., Liu, Y., Zhang, Y., Wadhwa, B., Dong, J. S., & Wang, X. (2014) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2642937.2642969 · Cited by 30 · cited 1×
“Some prior work translates software specifications into assertions [30] and oracles [22, 34].”
MethodInferring dependency constraints on parameters for web services
Wu, Q., Wu, L., Liang, G., Wang, Q., Xie, T., & Mei, H. (2013) · International World Wide Web Conference (WWW) · doi:10.1145/2488388.2488512 · Cited by 28 · cited 1×
“Many existing techniques [16, 22, 47, 53, 54] use a handful of manually-designed rules to extract constraints.”
MethodDetecting and understanding real-world differential performance bugs in machine learning libraries
Tizpaz-Niari, S., Černý, P., & Trivedi, A. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3404540 · Cited by 26 · cited 1×
“, returns NaN/Inf), decreased accuracy [18], and performance [47].”
BackgroundTesting Untestable Neural Machine Translation: An Industrial Case
Zheng, W., Wang, W., Liu, D., Zhang, C., Zeng, Q., Deng, Y., Yang, W., He, P., & Xie, T. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion.2019.00131 · Cited by 25 · cited 1×
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 42, 43, 50, 52] or oracle approximation [35, 60].”
MethodGrammar Based Directed Testing of Machine Learning Systems
Udeshi, S., & Chattopadhyay, S. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2953066 · Cited by 18
Discovering discrepancies in numerical libraries
Vanover, J., Deng, X., & Rubio-González, C. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397380 · Cited by 18 · cited 1×
“The DL library testing techniques focus on addressing the test oracle challenge, by using differential testing [18, 24, 41, 42, 49, 51] or oracle approximation [35, 59].”
Method
Cited by
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
“Traditional techniques with API-level fuzzing [139], [140] or model-level fuzzing [141], [142] suffer from the following limitations: 1) lack of diverse API sequence thus cannot reveal bugs caused by chained API sequences; 2) cannot generate arbitrary code thus cannot explore the huge search space…”
Deng, Y., Xia, C. S., Peng, H., Yang, C., & Zhang, L. (2023) · Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3597926.3598067 · Cited by 326
“While it is not explicitly stated in the documentation, the developers commented on the issue report that because the CPU and GPU use different lower-level shifting operators, the output result when shifting with negative values will be dependent on the implementation.”
MethodInfluentialDeng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., & Wang, R. (2024) · Transactions on machine learning research · doi:10.13140/rg.2.2.32710.69440/1 · Cited by 194
WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models
Yang, C., Deng, Y., Lu, R., Yao, J., Liu, J., Jabbarvand, R., & Zhang, L. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689736 · Cited by 82
Metadata
- Authors
- Xie, D., Li, Y., Kim, M., Pham, H. V., Tan, L., Zhang, X., & Godfrey, M. W.
- Affiliations
- Purdue University West Lafayette (US)
- University of Waterloo (CA)
- Ulsan National Institute of Science and Technology (KR)
- Venue
- Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2022-07-18
- Pages
- 176-188
- OpenAlex
- W4285490440
- Cited by
- 81
- References
- 54
- Citations by year
- 2026: 14 · 2025: 27 · 2024: 23 · 2023: 15 · 2022: 2
- FWCI
- 12.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Scholarworks@UNIST (Ulsan National Institute of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1901242)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- deep learning libraries, API testing, input constraints, constraint extraction, API documentation, software testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.