The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Neo: a learned query optimizer
Open accessAbstract
Query optimization is one of the most challenging problems in database systems. Despite the progress made over the past decades, query optimizers remain extremely complex components that require a great deal of hand-tuning for specific workloads and datasets. Motivated by this shortcoming and inspired by recent advances in applying machine learning to data management challenges, we introduce Neo ( Neural Optimizer ), a novel learning-based query optimizer that relies on deep neural networks to generate query executions plans. Neo bootstraps its query optimization model from existing optimizers and continues to learn from incoming queries, building upon its successes and learning from its failures. Furthermore, Neo naturally adapts to underlying data patterns and is robust to estimation errors. Experimental results demonstrate that Neo, even when bootstrapped from a simple optimizer like PostgreSQL, can learn a model that offers similar performance to state-of-the-art commercial optimizers, and in some cases even surpass them.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2008) · Journal of Machine Learning Research · Cited by 33,051
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
(2011) · Cited by 6,993
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 6,961
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Software Framework for Topic Modelling with Large Corpora
Rek, R. R. U., & Sojka, P. (2010) · doi:10.13140/2.1.2393.1847 · Cited by 3,647
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239 · cited 2×
“phaGo can trivially represent the board as an image and use image convolution, possibly the most well-studied and highly-optimized neural network primitive [24, 28], in order to predict the desirability of a board state.”
BackgroundLinguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Bellman, R. (1957) · Indiana University Mathematics Journal · doi:10.1512/iumj.1957.6.56038 · Cited by 2,408
Access path selection in a relational database management system
Selinger, P. G., Astrahan, M. M., Chamberlin, D. D., Lorie, R. A., & Price, T. G. (1979) · Proceedings of the 1979 ACM SIGMOD international conference on Management of data - SIGMOD '79 · doi:10.1145/582095.582099 · Cited by 2,188
Deep Learning of Representations for Unsupervised and Transfer Learning.
Bengio, Y. (2011) · Cited by 1,702
Generalized best-first search strategies and the optimality of A*
Dechter, R., & Pearl, J. (1985) · Journal of the ACM · doi:10.1145/3828.3830 · Cited by 1,071
How good are query optimizers, really?
Leis, V., Gubichev, A., Mirchev, A., Boncz, P. A., Kemper, A., & Neumann, T. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2850583.2850594 · Cited by 1,052
The Case for Learned Index Structures
Kraska, T., Beutel, A., Chi, E. H., Dean, J., & Polyzotis, N. (2018) · Proceedings of the 2018 International Conference on Management of Data · doi:10.1145/3183713.3196909 · Cited by 956
Self-citationToward Multimodal Image-to-Image Translation
Zhu, J.-Y., Zhang, R., Pathak, D., Darrell, T. J., Efros, A. A., Wang, O., & Shechtman, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11586 · Cited by 906
Deep Q-learning From Demonstrations
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11757 · Cited by 889
Learning scheduling algorithms for data processing clusters
Mao, H., Schwarzkopf, M., Venkatakrishnan, S. B., Meng, Z., & Alizadeh, M. (2019) · Proceedings of the ACM Special Interest Group on Data Communication · doi:10.1145/3341302.3342080 · Cited by 723
Self-citationRetrofitting Word Vectors to Semantic Lexicons
Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., & Smith, N. A. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1184 · Cited by 721
Using anytime algorithms in intelligent systems
Shlomo Zilberstein, B. (1996) · doi:10.1609/aimag.v17i3.1232 · Cited by 680
Convolutional Neural Networks over Tree Structures for Programming Language Processing
Mou, L., Li, G., Zhang, L., Wang, T., & Jin, Z. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10139 · Cited by 675
Deep Learning for Entity Matching: A Design Space Exploration
Mudgal, S., Li, H., Rekatsinas, T., Doan, A., Park, Y., Krishnan, G., Deep, R., Arcaute, E., & Raghavendra, V. (2018) · Proceedings of the 2018 International Conference on Management of Data · doi:10.1145/3183713.3196926 · Cited by 550
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_447 · Cited by 400
The Volcano optimizer generator: extensibility and efficient search
Graefe, G., & McKenna, W. J. (2002) · Proceedings of IEEE 9th International Conference on Data Engineering · doi:10.1109/icde.1993.344061 · Cited by 393
CORDS: automatic discovery of correlations and soft functional dependencies
Ilyas, I. F., Markl, V., Haas, P., Brown, P., & Aboulnaga, A. (2004) · Proceedings of the 2004 ACM SIGMOD international conference on Management of data · doi:10.1145/1007568.1007641 · Cited by 315
Deep Q-learning from Demonstrations
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Dulac-Arnold, G., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03732 · Cited by 270
Learning to Optimize Join Queries With Deep Reinforcement LearningPreprint
Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J. M., & Stoica, I. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.03196 · Cited by 240
Deep Reinforcement Learning for Join Order Enumeration
Marcus, R., & Papaemmanouil, O. (2018) · Proceedings of the First International Workshop on Exploiting Artificial Intelligence Techniques for Data Management · doi:10.1145/3211954.3211957 · Cited by 227
Self-citationThinking Fast and Slow with Deep Learning and Tree Search
Anthony, T., Tian, Z., & Barber, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08439 · Cited by 198
HoloDetect: Few-Shot Learning for Error Detection
Heidari, A., McGrath, J., Ilyas, I. F., & Rekatsinas, T. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3319888 · Cited by 197
Why Deep Learning Works: A Manifold Disentanglement Perspective
Brahma, P. P., Wu, D., & She, Y. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2496947 · Cited by 190
Refining Word Embeddings Using Intensity Scores for Sentiment Analysis
Yu, L.-C., Wang, J., Lai, K. R., & Zhang, X. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2788182 · Cited by 175
Database Meets Deep Learning: Challenges and Opportunities
Wang, W., Zhang, M., Chen, G., Jagadish, H. V., Ooi, B. C., & Tan, K.-L. (2016) · ACM SIGMOD Record · doi:10.1145/3003665.3003669 · Cited by 158
Learned Cardinalities: Estimating Correlated Joins with Deep Learning
Kipf, A., Kipf, T., Radke, B., Leis, V., Boncz, P. A., & Kemper, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00677 · Cited by 158
4th Biennial Conference on Innovative Data Systems Research (CIDR)
Diao, Y., Li, B., Anna, L. S., et al. (2009) · Cited by 151
Innateness, AlphaZero, and Artificial IntelligencePreprint
Marcus, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05667 · Cited by 142
Towards a robust query optimizer
Babcock, B., & Chaudhuri, S. (2005) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1066157.1066172 · Cited by 135
Learned Cardinalities: Estimating Correlated Joins with Deep Learning
Kipf, A., Kipf, T., Radke, B., Leis, V., Boncz, P. A., & Kemper, A. (2018) · Data Archiving and Networked Services (DANS) · Cited by 116
Towards a Hands-Free Query Optimizer through Deep LearningPreprint
Marcus, R., & Papaemmanouil, O. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10212 · Cited by 67
Self-citationLIFT: Reinforcement Learning in Computer Systems by Learning From DemonstrationsPreprint
Schaarschmidt, M., Kuhnle, A., Ellis, B., Fricke, K., Gessert, F., & Yoneki, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.07903 · Cited by 52
Termite: a system for tunneling through heterogeneous data
Fernandez, R. C., & Madden, S. R. (2019) · International Workshop on Exploiting Artificial Intelligence Techniques for Data Management (aiDM) · doi:10.1145/3329859.3329877 · Cited by 27
Cuttlefish: A Lightweight Primitive for Adaptive Query ProcessingPreprint
Kaftan, T., Bałazińska, M., Cheung, A., & Gehrke, J. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09180 · Cited by 22
Cited by
Federated Learning on Non-IID Data Silos: An Experimental Study
Li, Q., Diao, Y., Chen, Q., & He, B. (2022) · 2022 IEEE 38th International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00077 · Cited by 1,132
An end-to-end learning-based cost estimator
Sun, J., & Li, G. (2019) · Proceedings of the VLDB Endowment · doi:10.14778/3368289.3368296 · Cited by 256
CodexDB: synthesizing code for query processing from natural language instructions using GPT-3 codex
Trummer, I. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3551793.3551841 · Cited by 65
Metadata
- Authors
- Marcus, R., Negi, P., Mao, H., Zhang, C., Alizadeh, M., Kraska, T., Papaemmanouil, O., & Tatbul, N.
- Venue
- Proceedings of the VLDB Endowment
- Published
- 2019-07
- Volume
- 12
- Issue
- 11
- Pages
- 1705-1718
- ISSN
- 2150-8097 (ISSN-L)
- OpenAlex
- W2970148517
- Cited by
- 389
- References
- 49
- FWCI
- 21.07
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.