The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Bayesian Optimization in a Billion Dimensions via Random Embeddings
Open accessAbstract
Bayesian optimization techniques have been successfully applied to robotics, planning, sensor placement, recommendation, advertising, intelligent user interfaces and automatic algorithm configuration. Despite these successes, the approach is restricted to problems of moderate dimension, and several workshops on Bayesian optimization have identified its scaling to high-dimensions as one of the holy grails of the field. In this paper, we introduce a novel random embedding idea to attack this problem. The resulting Random EMbedding Bayesian Optimization (REMBO) algorithm is very simple, has important invariance properties, and applies to domains with both categorical and continuous variables. We present a thorough theoretical analysis of REMBO. Empirical results confirm that REMBO can effectively solve problems with billions of dimensions, provided the intrinsic dimensionality is low. They also show that REMBO achieves state-of-the-art performance in optimizing the 47 discrete parameters of a popular mixed integer linear programming solver.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
(2012) · Cited by 8,009
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Completely Derandomized Self-Adaptation in Evolution Strategies
Hansen, N., & Ostermeier, A. (2001) · Evolutionary Computation · doi:10.1162/106365601750190398 · Cited by 4,397
(2011) · Cited by 3,939
Real-time human pose recognition in parts from single depth images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995316 · Cited by 3,491
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Self-citationA Taxonomy of Global Optimization Methods Based on Response Surfaces
Jones, D. R. (2001) · Journal of Global Optimization · doi:10.1023/a:1012771025575 · Cited by 2,159
Real-time human pose recognition in parts from single depth images
Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., & Moore, R. (2013) · Communications of the ACM · doi:10.1145/2398356.2398381 · Cited by 2,044
Brochu, E., Cora, V. M., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2599 · Cited by 2,011
Lipschitzian optimization without the Lipschitz constant
Jones, D. R., Perttunen, C. D., & Stuckman, B. E. (1993) · Journal of Optimization Theory and Applications · doi:10.1007/bf00941892 · Cited by 1,976
Bergstra, J., Yamins, D. L. K., & Cox, D. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 1,852
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. W. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2182033 · Cited by 1,850
(2020) · Data Mining and Machine Learning · doi:10.1017/9781108564175.026 · Cited by 1,401
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
Self-citationParamILS: An Automatic Algorithm Configuration Framework
Hutter, F., Hoos, H. H., Leyton-Brown, K., & Stuetzle, T. (2009) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2861 · Cited by 882
Self-citationMulti-Task Bayesian Optimization
Swersky, K., Snoek, J., & Adams, R. P. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 502
Convergence rates of efficient global optimization algorithms
Bull, A. D. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1101.3501 · Cited by 446
Entropy search for information-efficient global optimization
Hennig, P., & Schuler, C. J. (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2343701 · Cited by 443
Real-Time Human Pose Recognition in Parts from Single Depth Images
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., & Blake, A. (2013) · Studies in Computational Intelligence · doi:10.1007/978-3-642-28661-2_5 · Cited by 411
The Knowledge-Gradient Policy for Correlated Normal Beliefs
Frazier, P., Powell, W., & Dayanik, S. (2009) · INFORMS Journal on Computing · doi:10.1287/ijoc.1080.0314 · Cited by 408
Bayesian Approach to Global Optimization: Theory and Applications
Mockus, J. (1989) · Mathematics and Its Applications · doi:10.1007/978-94-009-0909-0 · Cited by 406
Application of Bayesian approach to numerical methods of global and stochastic optimization
Mockus, J. (1994) · Journal of Global Optimization · doi:10.1007/bf01099263 · Cited by 399
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification AlgorithmsPreprint
Thornton, C., Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1208.3719 · Cited by 374
Self-citationAn Efficient Approach for Assessing Hyperparameter Importance
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2014) · Cited by 350
Self-citationNon-asymptotic Theory of Random Matrices: Extreme Singular Values
Rudelson, M., & Vershynin, R. (2011) · Proceedings of the International Congress of Mathematicians 2010 (ICM 2010) · doi:10.1142/9789814324359_0111 · Cited by 296
The Bayesian approach to global optimization
Mockus, J. (2005) · Lecture Notes in Control and Information Sciences · doi:10.1007/bfb0006170 · Cited by 283
Bayesian optimization in high dimensions via random embeddings
Wang, Z., Zoghi, M., Hutter, F., Matheson, D. S., & de Freitas, N. (2013) · UvA-DARE (University of Amsterdam) · Cited by 252
Self-citationAutomatic gait optimization with Gaussian process regression
Lizotte, D. J., Wang, T., Bowling, M., & Schuurmans, D. (2007) · Cited by 237
Hoos, H. H. (2012) · Communications of the ACM · doi:10.1145/2076450.2076469 · Cited by 225
Practical Bayesian optimization
Lizotte, D. J. (2008) · ERA: Education and Research Archive (University of Alberta) · doi:10.7939/r3-379f-gw15 · Cited by 215
Bubeck, S., Munos, R., Stoltz, G., & Szepesvári, C. (2011) · Journal of Machine Learning Research · Cited by 214
Smoothed Analysis of the Condition Numbers and Growth Factors of Matrices
Sankar, A., Spielman, D. A., & Teng, S.-H. (2006) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/s0895479803436202 · Cited by 207
Vazquez, E., & Bect, J. (2010) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2010.04.018 · Cited by 183
Learning Where to Attend with Deep Architectures for Image Tracking
Denil, M., Bazzani, L., Larochelle, H., & de Freitas, N. (2012) · Neural Computation · doi:10.1162/neco_a_00312 · Cited by 180
Bayesian optimisation for Intelligent Environmental Monitoring
Marchant, R., & Ramos, F. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6385653 · Cited by 171
Gaussian Processes to Speed up Hybrid Monte Carlo for Expensive Bayesian Integrals
Rasmussen, C. E. (2003) · doi:10.1093/oso/9780198526155.003.0045 · Cited by 167
Automated Configuration of Mixed Integer Programming Solvers
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13520-0_23 · Cited by 166
Self-citationPortfolio allocation for Bayesian optimization
Hoffman, M. D., Brochu, E., & de Freitas, N. (2011) · Cited by 159
Active Policy Learning for Robot Planning and Exploration under Uncertainty
Martínez-Cantín, R., de Freitas, N., Arnaud Doucet, B., & Castellanos, J. A. (2007) · Robotics: Science and Systems · doi:10.15607/rss.2007.iii.041 · Cited by 140
A Bayesian interactive optimization approach to procedural animation design
Brochu, E., Brochu, T., & de Freitas, N. (2010) · doi:10.5555/1921427.1921443 · Cited by 132
Active Preference Learning with Discrete Choice Data
Brochu, E., de Freitas, N., & Ghosh, A. (2007) · Cited by 125
High-Dimensional Gaussian Process Bandits
Djolonga, J., Krause, A., & Cevher, V. (2013) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 116
Particle Learning of Gaussian Process Models for Sequential Design and Optimization
Gramacy, R. B., & Polson, N. G. (2011) · Journal of Computational and Graphical Statistics · doi:10.1198/jcgs.2010.09171 · Cited by 104
Portfolio Allocation for Bayesian OptimizationPreprint
Brochu, E., Hoffman, M. W., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1009.5419 · Cited by 89
Hoffman, M. D., Shahriari, B., & de Freitas, N. (2014) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 88
Batch Bayesian Optimization via Simulation Matching
Azimi, J., Fern, A., & Fern, X. Z. (2010) · Cited by 85
Automated configuration of algorithms for solving hard computational problems
Hutter, F. (2009) · cIRcle (University of British Columbia) · doi:10.14288/1.0051652 · Cited by 84
Self-citationParallel Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34413-8_5 · Cited by 82
Self-citationAn evaluation of sequential model-based optimization for expensive blackbox functions
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2013) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/2464576.2501592 · Cited by 78
Self-citationBayesian Multi-Scale Optimistic OptimizationPreprint
Wang, Z., Shakibi, B., Lin, J., & de Freitas, N. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.7005 · Cited by 78
Self-citationExponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations
de Freitas, N., Smola, A. J., & Zoghi, M. (2012) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1206.6457 · Cited by 76
Self-citationMaking a Science of Model SearchPreprint
Bergstra, J., Yamins, D. L. K., & Cox, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1209.5111 · Cited by 67
Adaptive MCMC with Bayesian Optimization
Mahendran, N., Wang, Z., Hamze, F., & de Freitas, N. (2012) · Cited by 54
Self-citationAn experimental methodology for response surface optimization methods
Lizotte, D. J., Greiner, R., & Schuurmans, D. (2011) · Journal of Global Optimization · doi:10.1007/s10898-011-9732-z · Cited by 52
SMC Samplers for Bayesian Optimal Nonlinear Design
Kück, H., de Freitas, N., & Doucet, A. (2006) · doi:10.1109/nsspw.2006.4378829 · Cited by 30
Cited by
Automated Machine Learning: Methods, Systems, Challenges
(2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5 · Cited by 1,485
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Self-citationHyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Frazier, P. I. (2018) · Recent Advances in Optimization and Modeling of Contemporary Problems · doi:10.1287/educ.2018.0188 · Cited by 826
An Evolutionary Algorithm for Large-Scale Sparse Multiobjective Optimization Problems
Tian, Y., Zhang, X., Wang, C., & Jin, Y. (2020) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2918140 · Cited by 440
Metadata
- Authors
- Wang, Z., Hutter, F., Zoghi, M., Matheson, D., & De Feitas, N.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2016-02-19
- Volume
- 55
- Pages
- 361-387
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W1480330138
- Cited by
- 390
- References
- 58
- FWCI
- 27.37
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.