Problems and opportunities in training deep learning software systems: an analysis of variance
The DOI matches its Crossref record; 116 of the 116 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) training algorithms utilize nondeterminism to improve models' accuracy and training efficiency. Hence, multiple identical training runs (e.g., identical training data, algorithm, and network) produce different models with different accuracies and training times. In addition to these algorithmic factors, DL libraries (e.g., TensorFlow and cuDNN) introduce additional variance (referred to as implementation-level variance) due to parallelism, optimization, and floating-point computation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“For example, the time to convergence of the 16 fixed-seed identical training runs of another popular network, ResNet56 [51], ranges”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“Following common practice [14, 24, 45, 51, 62, 97, 102], we use the validation set to select the best model.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Following common practice [14, 24, 45, 51, 62, 97, 102], we use the validation set to select the best model.”
MethodLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
“For example, our experiments show that for 16 identical training runs of a popular DL network, LeNet5 [65], the accuracy of the resulting 16 models ranges from 8.6% to 99.0% —a large accuracy difference of 90.4%.”
BackgroundMethodInfluential(2014) · Cited by 33,419
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 4×
“DL implementation consists of 1 https://github.com/lin-tan/dl-variance high-level DL libraries (e.g., Keras), core DL libraries (e.g., Tensor-Flow and PyTorch), low-level computation libraries (e.g., cuDNN and CUDA), and hardware (e.g., GPU, CPU, and TPU).”
MethodInfluentialLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 1×
“Following common practice [14, 24, 45, 51, 62, 97, 102], we use the validation set to select the best model.”
MethodDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“Weight Initialization: Weight initialization [42, 50, 66, 90] is an important step in DL training [101, 104].”
BackgroundZhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · cited 1×
“In addition, we focus on studying the variance of models’ overall accuracy,per-classaccuracy,andtrainingtime,asthesearecommon metricsthatDLresearchersandpractitionersuse,andtherehave beenmanytechniques[14,22,38,45,103,118]proposedtoimprove onthesemetrics.”
Background(2010) · Cited by 14,134
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203 · cited 1×
“Data augmentation [96] is an inexpensive method that randomly transforms the input to increase the input’s diversity.”
MethodTensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Self-citation(2012) · Cited by 8,009
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
(2013) · Cited by 3,740
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · cited 2×
“In addition, we focus on studying the variance of models’ overall accuracy, per-class accuracy, and training time, as these are common metrics that DL researchers and practitioners use, and there have been many techniques [14, 22, 38, 45, 103, 118] proposed to improve on these metrics.”
BackgroundDeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundWhat every computer scientist should know about floating-point arithmetic
Goldberg, D. (1991) · ACM Computing Surveys · doi:10.1145/103162.103163 · Cited by 1,750 · cited 1×
“tive due to rounding errors [43], varying orders of additions may produce nondeterministic output (A + B +C , B +C +A).”
BackgroundDeep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
LeCun, Y. A., Bottou, L., Orr, G. B., & Müller, K.-R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_3 · Cited by 1,264 · cited 2×
“It breaks up the order of the training data to prevent the model from quickly overfitting to a particular label [66].”
BackgroundDeepTest: automated testing of deep-neural-network-driven autonomous cars
Tian, Y., Pei, K., Jana, S., & Ray, B. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180220 · Cited by 1,217 · cited 2×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodGradientBased Learning Applied to Document Recognition
(2009) · Intelligent Signal Processing · doi:10.1109/9780470544976.ch9 · Cited by 1,135 · cited 6×
“We choose to experiment on LeNet [65], ResNet [51], andWideResNet [112] architectures as they are popular networks for image classification.”
MethodInfluentialDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationPreprint
He, K., Zhang, X., Ren, S., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.01852 · Cited by 1,000 · cited 1×
“Weight Initialization: Weight initialization [42, 50, 66, 90] is an important step in DL training [101, 104].”
BackgroundArcuri, A., & Briand, L. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985795 · Cited by 968 · cited 1×
“, more than 30 runs[15]) the improvement is not statistically significant, then the variance might be large enough such that a statistically significant conclusion about the difference between the two techniques might not be possible.”
BackgroundcuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943 · cited 1×
“, TensorFlow [13] and cuDNN [23]) introduce additional variance.”
BackgroundThe Cambridge Dictionary of Statistics
SRJ, & Everitt, B. S. (1999) · Journal of the American Statistical Association · doi:10.2307/2670205 · Cited by 940
Visualizing the Loss Landscape of Neural Nets
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000461393 · Cited by 926
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Salimans, T., & Kingma, D. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07868 · Cited by 883 · cited 2×
“Impact of weight initialization: Prior work [57, 88, 101, 104] measure the impact of different initial weights on models’ training time.”
BackgroundMethodIdentifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827 · cited 4×
“Nondeterminism in stochastic gradient descent (SGD): Much work investigates variance caused by SGD [26, 33, 35, 58, 68].”
BackgroundInfluentialExact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J. L., & Ganguli, S. (2014) · International Conference on Learning Representations · Cited by 761
The Loss Surfaces of Multilayer Networks
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.0233 · Cited by 733 · cited 3×
“There are existing theoretical analyses of DL training [26, 27, 35, 58, 68, 90] that study how well optimizers find good local optima given algorithmic NI-factors.”
BackgroundDSOD: Learning Deeply Supervised Object Detectors from Scratch
Shen, Z., Liu, Z., Li, J., Jiang, Y.-G., Chen, Y., & Xue, X. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.212 · Cited by 646 · cited 1×
“We pick training from-scratch as our scenario for the training phase because it is a common and fundamental training scenario [14, 51, 94, 98, 103, 112, 122], i.”
BackgroundZhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238187 · Cited by 611 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodDeepGauge: multi-granularity testing criteria for deep learning systems
Ma, L., Juefei-Xu, F., Zhang, F., Sun, J., Xue, M., Li, B., Chen, C., Su, T., Li, L., Liu, Y., et al. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238202 · Cited by 588 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodGuiding Deep Learning System Testing Using Surprise Adequacy
Kim, J., Feldt, R., & Yoo, S. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00108 · Cited by 430 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodOptimizing deep learning hyper-parameters through an evolutionary algorithm
Young, S. R., Rose, D. C., Karnowski, T. P., Lim, S.-H., & Patton, R. M. (2015) · Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments · doi:10.1145/2834892.2834896 · Cited by 424 · cited 1×
“It is expected that models with different inputs perform differently, and there is a flurry of work on how to select the best input (e.g., hyper-parameter tuning) [17, 54, 111].”
BackgroundNeural Networks and Deep Learning: A Textbook
Aggarwal, C. C. (2018) · doi:10.1007/978-3-319-94463-0 · Cited by 417
Benchmarking deep learning models on large healthcare datasets
Purushotham, S., Meng, C., Che, Z., & Liu, Y. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.04.007 · Cited by 406 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundComposing Text and Image for Image Retrieval - an Empirical Odyssey
Vo, N., Jiang, L., Sun, C., Murphy, K., Li, L.-J., Fei-Fei, L., & Hays, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00660 · Cited by 399 · cited 1×
“Anecdotal evidence of NI-factors: Some studies [59, 76, 77, 108] quantify the variance in their results caused by NI-factors.”
BackgroundCNTK: Microsoft's Open-Source Deep-Learning Toolkit
Seide, F., & Agarwal, A. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945397 · Cited by 382 · cited 1×
“DL libraries: We use Keras version 2.2.2 [25] as our high-level library since it provides us with the ability to transparently switch between three DL core libraries (TensorFlow [13], CNTK [16], and Theano [91]).”
MethodInfluentialAn Alternative View: When Does SGD Escape Local Minima?Preprint
Kleinberg, R. D., Li, Y.-Z., & Yuan, Y. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1802.06175 · Cited by 349 · cited 6×
“Nondeterminism in stochastic gradient descent (SGD):Much work investigates variance caused by SGD [26, 33, 35, 58, 68].”
BackgroundInfluentialSearching for Efficient Multi-Scale Architectures for Dense Image Prediction
Chen, L.-C., Collins, M. D., Zhu, Y., Papandreou, G., Zoph, B., Schroff, F., Adam, H., & Shlens, J. (2018) · arXiv · doi:10.48550/arxiv.1809.04184 · Cited by 347 · cited 1×
“In addition, we focus on studying the variance of models’ overall accuracy, per-class accuracy, and training time, as these are common metrics that DL researchers and practitioners use, and there have been many techniques [14, 22, 38, 45, 103, 118] proposed to improve on these metrics.”
BackgroundConcolic testing for deep neural networks
Sun, Y., Wu, M., Ruan, W., Huang, X., Kwiatkowska, M., & Kroening, D. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238172 · Cited by 332 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodBenchmarking State-of-the-Art Deep Learning Software Tools
Shi, S., Wang, Q., Xu, P., & Chu, X. (2016) · 2016 7th International Conference on Cloud Computing and Big Data (CCBD) · doi:10.1109/ccbd.2016.029 · Cited by 329 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundDeep Reinforcement Learning that Matters
Henderson, P. B., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06560 · Cited by 324 · cited 1×
“Papers [29, 53] that investigates the impact of random seeds on RL are different from ours, since they only consider the impact of random seeds (i.”
BackgroundProducing wrong data without doing anything obviously wrong!
Mytkowicz, T., Diwan, A., Hauswirth, M., & Sweeney, P. F. (2009) · Proceedings of the 14th international conference on Architectural support for programming languages and operating systems · doi:10.1145/1508244.1508275 · Cited by 303 · cited 2×
“Another paper [79] states that small changes in the experimental setup can generate measurement bias.”
BackgroundMethodRetrieval-based neural source code summarization
Zhang, J., Wang, X., Zhang, H., Sun, H., & Liu, X. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380383 · Cited by 275 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundDeepSim: deep learning code functional similarity
Zhao, G., & Huang, J. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236068 · Cited by 261 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundBenchmarking and Analyzing Deep Neural Network Training
Zhu, H., Akrout, M., Zheng, B., Pelegris, A., Jayarajan, A., Phanishayee, A., Schroeder, B., & Pekhimenko, G. G. (2018) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc.2018.8573476 · Cited by 237 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundShow Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233 · cited 1×
“Recently, the DL research community promoted sharing artifacts and results transparently[12, 34].”
BackgroundTheano: Deep Learning on GPUs with Python
Bergstra, J., Bastien, F., Breuleux, O., Lamblin, P., Pascanu, R., Delalleau, O., Desjardins, G., Warde-Farley, D., Goodfellow, I., Bergeron, A., et al. (2012) · Cited by 223
MODE: automated neural network model debugging via state differential analysis and input selection
Ma, S., Liu, Y., Lee, W.-C., Zhang, X., & Grama, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236082 · Cited by 211 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodDLFix: context-based code transformation learning for automated program repair
Li, Y., Wang, S., & Nguyen, T. N. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380345 · Cited by 209 · cited 1×
“Producing better models: When a DL model is the contribution (e.g., defect prediction [70] or program repair [69]), practitioners could leverage variance to obtain a more accurate model.”
MethodGradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local MinimaPreprint
Du, S. S., Lee, J. D., Tian, Y., Póczos, B., & Singh, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00779 · Cited by 195 · cited 1×
“However, different initial weights may result in convergence to different local minima [36, 40, 67].”
BackgroundHow Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
Colas, C., Sigaud, O., & Oudeyer, P.-Y. (2018) · arXiv · Cited by 194 · cited 1×
“Papers [29, 53] that investigates the impact of random seeds on RL are different from ours, since they only consider the impact of random seeds (i.e., algorithmic NI-factors), while we also study the variance caused by implementation-level NI-factors.”
BackgroundNeural Networks and Deep Learning: A Textbook
Aggarwal, C. C. (2023) · Springer eBooks · doi:10.1007/978-3-031-29642-0 · Cited by 191
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries
Pham, H. V., Lutellier, T., Qi, W., & Tan, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00107 · Cited by 190 · cited 2×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodSelf-citationHellendoorn, V. J., Bird, C., Barr, E. T., & Allamanis, M. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236051 · Cited by 181 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundWuji: Automatic Online Combat Game Testing Using Evolutionary Deep Reinforcement Learning
Zheng, Y., Xie, X., Su, T., Ma, L., Hao, J., Meng, Z., Liu, Y., Shen, R., Chen, Y., & Fan, C. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00077 · Cited by 174 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundEscaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
Ge, R., Huang, F., Jin, C., & Yang, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.02101 · Cited by 173 · cited 1×
“However, different initial weights may result in convergence to different local minima [36, 40, 67].”
BackgroundEssentially No Barriers in Neural Network Energy Landscape
Draxler, F., Veschgini, K., Manfred Salmhofer, B., & Hamprecht, F. A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00885 · Cited by 171 · cited 6×
“Nondeterminism in stochastic gradient descent (SGD):Much work investigates variance caused by SGD [26, 33, 35, 58, 68].”
BackgroundInfluentialDIRE: A Neural Approach to Decompiled Identifier Naming
Lacomis, J., Yin, P., Schwartz, E. J., Allamanis, M., Goues, C. L., Neubig, G., & Vasilescu, B. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00064 · Cited by 168 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundIlievski, I., Akhtar, T., Feng, J., & Shoemaker, C. A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10647 · Cited by 166 · cited 1×
“It is expected that models with different inputs perform differently, and there is a flurry of work on how to select the best input (e.g., hyper-parameter tuning) [17, 54, 111].”
BackgroundLi, Y., Wang, S., Nguyen, T. N., & Van Nguyen, S. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360588 · Cited by 162
Understanding and correcting pathologies in the training of learned optimizersPreprint
Metz, L., Maheswaranathan, N., Nixon, J., Freeman, C. D., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10180 · Cited by 160 · cited 1×
“Anecdotal evidence of NI-factors: Some studies [59, 76, 77, 108] quantify the variance in their results caused by NI-factors.”
BackgroundDeepDriving: Learning Affordance for Direct Perception in Autonomous DrivingPreprint
Chen, C., Seff, A., Kornhauser, A. L., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00256 · Cited by 154 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundDeepBillboard: systematic physical-world testing of autonomous driving systems
Zhou, H., Li, W., Kong, Z., Guo, J., Zhang, Y., Yu, B., Zhang, L., & Liu, C. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380422 · Cited by 153
Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing
Wang, J., Dong, G., Sun, J., Wang, X., & Zhang, P. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00126 · Cited by 151
Self-citationDeepXplore: automated whitebox testing of deep learning systems
Pei, K., Cao, Y., Yang, J., & Jana, S. (2019) · Communications of the ACM · doi:10.1145/3361566 · Cited by 150
ScratchDet: Training Single-Shot Object Detectors From Scratch
Zhu, R., Zhang, S., Wang, X., Wen, L., Shi, H., Bo, L., & Mei, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00237 · Cited by 141
Open Problem: The landscape of the loss surfaces of multilayer networks
Choromanska, A., LeCun, Y., & Arous, G. B. (2015) · Conference on Learning Theory · Cited by 138
Software visualization and deep transfer learning for effective software defect prediction
Chen, J., Hu, K., Yu, Y., Chen, Z., Xuan, Q., Liu, Y., & Filkov, V. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380389 · Cited by 126 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundSelf-citationGuo, Q., Chen, S., Xie, X., Ma, L., Hu, Q., Liu, H., Liu, Y., Zhao, J., & Li, X. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00080 · Cited by 125 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundWeakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer
Fang, H.-S., Lu, G., Fang, X., Xie, J., Tai, Y.-W., & Lu, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00015 · Cited by 122
Fuzz testing based data augmentation to improve robustness of deep neural networks
Gao, X., Saha, R. K., Prasad, M. R., & Roychoudhury, A. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380415 · Cited by 115 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodA Deep Learning Approach to Diabetic Blood Glucose Prediction
Mhaskar, H. N., Pereverzyev, S. V., & van der Walt, M. D. (2017) · Frontiers in Applied Mathematics and Statistics · doi:10.3389/fams.2017.00014 · Cited by 115 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundApricot: A Weight-Adaptation Approach to Fixing Deep Learning Models
Zhang, H., & Chan, W. K. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00043 · Cited by 112
Importance-driven deep learning system testing
Gerasimou, S., Enişer, H. F., Şen, A., & Çakan, A. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380391 · Cited by 110 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodCui, H., Šimša, J., Lin, Y., Li, H., Blum, B., Xu, X., Yang, J., Gibson, G. A., & Bryant, R. E. (2013) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/2517349.2522735 · Cited by 109 · cited 1×
“As future work, deterministic multithreading [32] may be promising for more realistic deterministic DL systems.”
BackgroundUnblind your apps: predicting natural-language labels for mobile GUI components by deep learning
Chen, J., Chen, C., Xing, Z., Xu, X., Zhu, L., Li, G., & Wang, J. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380327 · Cited by 109 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundSelf-citationTowards characterizing adversarial defects of deep learning software from the lens of uncertainty
Zhang, X., Xie, X., Ma, L., Du, X., Hu, Q., Liu, Y., Zhao, J., & Sun, M. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380368 · Cited by 106
Gradient Descent Converges to Minimizers
Lee, J. D., Simchowitz, M., Jordan, M. I., & Recht, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.04915 · Cited by 103 · cited 1×
“However, different initial weights may result in convergence to different local minima [36, 40, 67].”
BackgroundGonzález-Barahona, J. M., & Robles, G. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9181-9 · Cited by 102 · cited 1×
“Approaches to improve reproducibility suggested by the SE community [44, 75, 107] need to also consider training variance.”
BackgroundProperty Inference for Deep Neural Networks
Gopinath, D., Converse, H., Păsăreanu, C. S., & Taly, A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00079 · Cited by 102 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodAdversarial sample detection for deep neural network through model mutation testingPreprint
Wang, J., DONG, G., Sun, J., Wang, X., & PEIXIN, Z. (2019) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 101
Self-citationDSOD: Learning Deeply Supervised Object Detectors from Scratch
Shen, Z., Liu, Z., Li, J., et al. (2017) · arXiv · doi:10.48550/arxiv.1708.01241 · Cited by 94 · cited 1×
“We pick training from-scratch as our scenario for the training phase because it is a common and fundamental training scenario [14, 51, 94, 98, 103, 112, 122], i.”
BackgroundDissector: input validation for deep learning applications by crossing-layer dissection
Wang, H., Xu, J., Xu, C., Ma, X., & Lu, J. J. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380379 · Cited by 88
Teo, K. K., Wang, L., & Lin, Z. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45718-6_35 · Cited by 87 · cited 4×
“This large accuracy difference is caused by the random initialization of the weights [101, 104].”
BackgroundInfluentialReluDiff: differential verification of deep neural networks
Paulsen, B., Wang, J., & Wang, C. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380337 · Cited by 86 · cited 1×
“We choose image classification architectures as they are often used as test subjects in recent SE papers that test [41, 71, 73, 85, 100, 105, 106, 109, 115, 120], verify [46, 83], and improve [39, 56, 74, 110, 113, 116] DL models and libraries.”
MethodSelf-citationBoosting operational DNN testing efficiency through conditioning
Li, Z., Ma, X., Xu, C., Cao, C., Xu, J., & Lü, J. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338930 · Cited by 85
Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF SurrogatesPreprint
Ilievski, I., Akhtar, T., Feng, J., & Shoemaker, C. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.08316 · Cited by 81 · cited 1×
“It is expected that models with different inputs perform differently, and there is a flurry of work on how to select the best input (e.g., hyper-parameter tuning) [17, 54, 111].”
BackgroundDepth CNNs for RGB-D Scene Recognition: Learning from Scratch Better than Transferring from RGB-CNNs
Song, X., Herranz, L., & Jiang, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11226 · Cited by 75 · cited 1×
“We pick training from-scratch as our scenario for the training phase because it is a common and fundamental training scenario [14, 51, 94, 98, 103, 112, 122], i.”
BackgroundBug synthesis: challenging bug-finding tools with deep faults
Roy, S., Pandey, A., Dolan-Gavitt, B., & Hu, Y. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236084 · Cited by 75 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundEvaluation of Deep Learning Frameworks Over Different HPC Architectures
Shams, S., Platania, R., Lee, K., & Park, S. (2017) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2017.259 · Cited by 74 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundEliminating Exposure Bias and Metric Mismatch in Multiple Object Tracking
Maksai, A., & Fua, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00477 · Cited by 67 · cited 1×
“Anecdotal evidence of NI-factors: Some studies [59, 76, 77, 108] quantify the variance in their results caused by NI-factors.”
BackgroundTrautsch, F., Herbold, S., Makedonski, P., & Grabowski, J. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9537-x · Cited by 58 · cited 1×
“Approaches to improve reproducibility suggested by the SE community [44, 75, 107] need to also consider training variance.”
BackgroundA comparative study of open source deep learning frameworks
Shatnawi, A., Al-Bdour, G., Al-Qurran, R., & Al-Ayyoub, M. (2018) · 2018 9th International Conference on Information and Communication Systems (ICICS) · doi:10.1109/iacs.2018.8355444 · Cited by 58 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundWeight value initialization for improving training speed in the backpropagation network
Kim, Y., & Ra, J. B. (1991) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1991.170747 · Cited by 55 · cited 2×
“Impact of weight initialization: Prior work [57, 88, 101, 104] measure the impact of different initial weights on models’ training time.”
BackgroundTesting DNN image classifiers for confusion & bias errors
Tian, Y., Zhong, Z., Ordóñez, V., Kaiser, G. E., & Ray, B. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380400 · Cited by 55
Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond
Liu, L., Wu, Y., Wei, W., Cao, W., Sahin, S., & Zhang, Q. (2018) · 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2018.00125 · Cited by 54 · cited 1×
“However, their target is finding the best networks [30, 86, 121], hardware [92, 121], hyper-parameters [72], and framework [48, 60, 72, 93, 95, 121].”
BackgroundKovalev, V. A., Kalinovsky, A., & КОВАЛЕВ, С. А. (2016) · BSU Digital Library (Belarusian State University) · Cited by 45
AutoFocus: Interpreting Attention-Based Neural Networks by Code Perturbation
Bui, N. D. Q., Yu, Y., & Jiang, L. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00014 · Cited by 44 · cited 1×
“Deep learning is widely used in many fields including autonomous driving cars [19], diabetic blood glucose prediction [78], and software engineering [18, 20, 21, 52, 63, 87, 114, 117, 119].”
BackgroundSelf-citationDepth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNsPreprint
Song, X., Herranz, L., & Jiang, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.06797 · Cited by 43 · cited 1×
“We pick training from-scratch as our scenario for the training phase because it is a common and fundamental training scenario [14, 51, 94, 98, 103, 112, 122], i.”
BackgroundReproducibility and replicability of software defect prediction studies
Mahmood, Z., Bowes, D., Hall, T., Lane, P. C. R., & Petrić, J. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.02.003 · Cited by 35 · cited 1×
“Approaches to improve reproducibility suggested by the SE community [44, 75, 107] need to also consider training variance.”
BackgroundDeterministic Implementations for Reproducibility in Deep Reinforcement Learning
Nagarajan, P., Warnell, G., & Stone, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.05676 · Cited by 33 · cited 1×
“Variance study of reinforcement learning (RL): Closest to our work is the study [80] that measures the impact of some implementation-level NI-factors on RL experiments.”
BackgroundKochura, Y., Stirenko, S., Alienin, O., Novotarskiy, M., & Gordienko, Y. G. (2017) · IEEE International Conference on Computer Sciences and Information Technologies (CSIT) · doi:10.1109/stc-csit.2017.8098808 · Cited by 31
Jooybar, H., Fung, W. W. L., O’Connor, M., Devietti, J. L., & Aamodt, T. M. (2013) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2451116.2451118 · Cited by 29 · cited 12×
“[55] propose a hardware mechanism to introduce determinism in GPU-based algorithms.”
BackgroundInfluentialKochura, Y., Stirenko, S., Alienin, O., Novotarskiy, M., & Gordienko, Y. G. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-70581-1_17 · Cited by 27 · cited 1×
“Anecdotal evidence of NI-factors: Some studies [59, 76, 77, 108] quantify the variance in their results caused by NI-factors.”
BackgroundReproducibility in Machine Learning-Based Studies: An Example of Text Mining
Olorisade, B. K., Pearl Brereton, A., & András, P. (2017) · Research Output (Edinburgh Napier University) · Cited by 24
Wootz: a compiler-based framework for fast CNN pruning via composability
Guan, H., Shen, X., & Lim, S. (2019) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3314221.3314652 · Cited by 13 · cited 1×
“Since DL training is expensive, an important research direction could be less-expensive variance estimation and training approaches such as software support for incremental training [47].”
BackgroundLamb, A., Binas, J., Goyal, A., Subramanian, S., Mitliagkas, I., Kazakov, D. V., Bengio, Y., & Mozer, M. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11382 · Cited by 4 · cited 3×
“For example, a recent research paper [64] proposed a new approach with a reported 0.8% accuracy improvement over the standard WRN-28-10.”
BackgroundInfluential
Research works printed but not linked: 19
- 2019. ASE '19: Proceedings of the 34th ACM/IEEE International Conference on Automated Software Engineering.
- 2019. ASPLOS '19: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems.
- 2019. CVPR'19: Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition.
- 2019. ESEC/FSE '19: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering.
- 2019. ICCV'19: Proceedings of The IEEE International Conference on Computer Vision. IEEE Press.
- 2019. ICML'19: Proceedings of The International Conference on Machine Learning.
- 2019. ICSE '19: Proceedings of the 41st International Conference on Software Engineering. IEEE Press.
- 2019. MLSys'19: Proceedings of Machine Learning and Systems.
- 2019. NIPS'19: Proceedings of the 33rd Conference on Neural Information Processing Systems.
- 2019. SOSP '19: Proceedings of the 27th ACM Symposium on Operating Systems Principles.
- 2020. ICLR'20: Proceedings of The International Conference on Learning Representations.
- 2020. Reproducibility Challenge @ NeurIPS 2019. https://reproducibilitychallenge.github.io/neurips2019/
- Charu C Aggarwal. 2018. Neural networks and deep learning. In Springer.
- Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. 2017. Dawnbench: An end-to-end deep learning benchmark and competition. In Training.
- Standard Performance Evaluation Corporation. 2006. SPEC CPU2006 Benchmarks. http://www.spec.org/cpu2006/.
- Heming Cui, Jiri Simsa, Yi-Hong Lin, Hao Li, Ben Blum, Xinan Xu, Junfeng Yang, Garth A. Gibson, and Randal E. Bryant. 2013. Parrot: a Practical Runtime for Deterministic, Stable, and Reliable Threads. In SOSP.
- Jeff Haochen and Suvrit Sra. 2019. Random Shuffling Beats SGD after Finite Epochs. In ICML.
- Hadi Jooybar, Wilson WL Fung, Mike O'Connor, Joseph Devietti, and Tor M Aamodt. 2013. GPUDet: a deterministic GPU architecture. In ASPLOS.
- Shlomo Sawilowsky, Jack Sawilowsky, and Robert J. Grissom. 2011. Effect Size.
Cited by
Machine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
An extensive study on pre-trained models for program understanding and generation
Zeng, Z., Tan, H., Zhang, H., Li, J., Zhang, Y., & Zhang, L. (2022) · Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3533767.3534390 · Cited by 153
“Note that considering the model randomness, we run all our experiments for 5 runs as recommended by prior work [52] and present the average results, while all our studied pre-trained models do not clearly indicate their number of runs in their original papers.”
MethodYan, J., Cai, J., Zhang, B., Wang, Y., Wong, D. F., & Siu, S. W. I. (2022) · Antibiotics · doi:10.3390/antibiotics11101451 · Cited by 146
Metadata
- Authors
- Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., & Nagappan, N.
- Affiliations
- University of Waterloo (CA)
- Purdue University West Lafayette (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering
- Publisher
- ACM
- Published
- 2020-12-21
- Pages
- 771-783
- OpenAlex
- W3122184684
- Cited by
- 100
- References
- 114
- Citations by year
- 2026: 13 · 2025: 11 · 2024: 16 · 2023: 29 · 2022: 19 · 2021: 11
- FWCI
- 9.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/ACM International Conference on Automated Software Engineering (ASE) (publishedVersion)
- Funders
- National science foundation
- Facebook Research Award
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- deep learning training, non-determinism, model reproducibility, parallelism, floating-point arithmetic, training efficiency, TensorFlow, cuDNN
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.