TASO: optimizing deep learning computation with automatic generation of graph substitutions
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Existing deep neural network (DNN) frameworks optimize the computation graph of a DNN by applying graph transformations manually designed by human experts. This approach misses possible graph optimizations and is difficult to scale, as new DNN operators are introduced on a regular basis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
(2010) · International Conference on Machine Learning · Cited by 13,995
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 7,831
de Moura, L., & Bjørner, N. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-78800-3_24 · Cited by 6,595
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
A guide to convolution arithmetic for deep learning
Dumoulin, V., & Visin, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07285 · Cited by 1,172
cuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943
Pnueli, A., Siegel, M., & Singerman, E. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0054170 · Cited by 499
Translation validation for an optimizing compiler
Necula, G. C. (2000) · Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation · doi:10.1145/349299.349314 · Cited by 488
Compiler validation via equivalence modulo inputs
Le, V., Afshari, M., & Su, Z. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594334 · Cited by 388 · cited 1×
“In the compiler literature, significant effort has been devoted to formally verifying optimizations [7, 11, 13, 23, 29, 30, 33, 35].”
BackgroundTraining and Inference with Integers in Deep Neural Networks
Wu, S., Li, G., Chen, F., & Shi, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04680 · Cited by 306
Device Placement Optimization with Reinforcement Learning
Mirhoseini, A., Pham, H., Le, Q. V., Steiner, B., Larsen, R., Zhou, Y., Kumar, N., Norouzi, M., Bengio, S., & Dean, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.04972 · Cited by 299
Superoptimizer: a look at the smallest program
Massalin, H. (1987) · ACM SIGPLAN Notices · doi:10.1145/36205.36194 · Cited by 221
Learning to Optimize Tensor Programs
Chen, T., Zheng, L., Yan, E., Jiang, Z., Moreau, T., Ceze, L., Guestrin, C., & Krishnamurthy, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08166 · Cited by 177
TVM: End-to-End Optimization Stack for Deep LearningPreprint
Chen, T., Moreau, T., Jiang, Z., Shen, H., Yan, E., Wang, L., Hu, Y., Ceze, L., Guestrin, C., & Krishnamurthy, A. (2018) · arXiv (Cornell University) · Cited by 173
Automatic generation of peephole superoptimizers
Bansal, S., & Aiken, A. (2006) · Proceedings of the 12th international conference on Architectural support for programming languages and operating systems · doi:10.1145/1168857.1168906 · Cited by 161
Self-citationBeyond Data and Model Parallelism for Deep Neural Networks
Jia, Z., Zaharia, M., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05358 · Cited by 140
Self-citationOptimizing DNN Computation with Relaxed Graph Substitutions
Jia, Z., Thomas, J. J., Warszawski, T., Gao, M., Zaharia, M., & Aiken, A. (2019) · Proceedings of Machine Learning and Systems · Cited by 112
Self-citationSemantic program alignment for equivalence checking
Churchill, B., Padon, O., Sharma, R., & Aiken, A. (2019) · Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/3314221.3314596 · Cited by 97
Self-citationData-driven equivalence checking
Sharma, R., Schkufza, E., Churchill, B., & Aiken, A. (2013) · Conference on Object-Oriented Programming Systems, Languages, and Applications · doi:10.1145/2509136.2509509 · Cited by 86
Self-citationAstra: Exploiting Predictability to Optimize Deep Learning
Sivathanu, M., Chugh, T., Singapuram, S. S. V., & Zhou, L. (2019) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3297858.3304072 · Cited by 61
Equality Saturation: A New Approach to Optimization
Tate, R., Stepp, M., Tatlock, Z., & Lerner, S. (2011) · Logical Methods in Computer Science · doi:10.2168/lmcs-7(1:10)2011 · Cited by 59
Discovering affine equalities using random interpretation
Gulwani, S., & Necula, G. C. (2003) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/604131.604138 · Cited by 54
Black-Box Equivalence Checking Across Compiler Optimizations
Dahiya, M., & Bansal, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-71237-6_7 · Cited by 37
Exploring the Hidden Dimension in Accelerating Convolutional Neural Networks
Jia, Z., Lin, S., Qi, C. R., & Aiken, A. (2018) · International Conference on Machine Learning · Cited by 13
Self-citation
Cited by
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. (2024) · Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 · doi:10.1145/3620666.3651335 · Cited by 114
Self-citationData Movement Is All You Need: A Case Study on Optimizing Transformers
Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., & Hoefler, T. (2020) · arXiv · doi:10.48550/arxiv.2007.00072 · Cited by 78
DeepPayload: Black-box Backdoor Attack on Deep Learning Models through Neural Payload Injection
Li, Y., Hua, J., Wang, H., Chen, C., & Liu, Y. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00035 · Cited by 75
“Such unified intermediate representation is the theoretic basis of model conversion tools [44], DNN performance optimization [45], and our payload injection technique.”
MethodDemystifying TensorRT: Characterizing Neural Network Inference Engine on Nvidia Edge Devices
Shafi, O., Rai, C., Sen, R., & Ananthanarayanan, G. (2021) · 2021 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc53511.2021.00030 · Cited by 51
Metadata
- Authors
- Jia, Z., Padon, O., Thomas, J., Warszawski, T., Zaharia, M., & Aiken, A.
- Venue
- Proceedings of the 27th ACM Symposium on Operating Systems Principles
- Publisher
- ACM
- Published
- 2019-10-27
- Pages
- 47-62
- OpenAlex
- W2981758446
- Cited by
- 261
- References
- 30
- FWCI
- 19.53
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.