The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Dealing with categorical and integer-valued variables in Bayesian Optimization with Gaussian processes
Open accessAbstract
Some optimization problems are characterized by an objective that is very expensive, that lacks an analytical expression, and whose evaluations can be contaminated by noise. Bayesian Optimization (BO) methods can be used to solve these problems efficiently. BO relies on a probabilistic model of the objective, which is typically a Gaussian process (GP). This model is used to compute an acquisition function that estimates the expected utility (for solving the optimization problem) of evaluating the objective at each potential new point. A problem with GPs is, however, that they assume real-valued input variables and cannot easily deal with categorical or integer-valued values. Common methods to account for these variables, before evaluating the objective, include assuming they are real and then using a one-hot encoding, for categorical variables, or rounding to the closest integer, for integer-valued variables. We show that this leads to suboptimal results and introduce a novel approach to tackle categorical or integer-valued input variables within the context of BO with GPs. Several synthetic and real-world experiments support our hypotheses and show that our approach outperforms the results of standard BO using GPs on problems with categorical or integer-valued input variables.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Kernel Methods for Pattern Analysis
Shawe-Taylor, J., & Cristianini, N. (2004) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809682 · Cited by 6,587
(2019) · Handbook of Machine Learning · doi:10.1142/9789811205675_0004 · Cited by 6,270
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
(2011) · Cited by 3,939
Encyclopedia of Machine Learning
(2010) · Springer eBooks · doi:10.1007/978-0-387-30164-8 · Cited by 3,465
Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization
Zhu, C., Byrd, R. H., Lu, P., & Nocedal, J. (1997) · ACM Transactions on Mathematical Software · doi:10.1145/279232.279236 · Cited by 3,462
(2003) · International Series in Operations Research & Management Science · doi:10.1007/b101874 · Cited by 3,430
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Dowsland, K. A., & Thompson, J. M. (2012) · Handbook of Natural Computing · doi:10.1007/978-3-540-92910-9_49 · Cited by 2,432
Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)
Rasmussen, C. E., & Williams, C. K. I. (2005) · The MIT Press eBooks · Cited by 1,829
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
Artificial intelligence and statistics
Wiggins, L. L. (1989) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(89)90027-6 · Cited by 1,117
Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms
Bergstra, J., Yamins, D., & Cox, D. (2013) · Proceedings of the Python in Science Conference · doi:10.25080/majora-8b375195-003 · Cited by 781
(2019) · International Series in Operations Research & Management Science · doi:10.1007/978-3-319-91086-4 · Cited by 677
Scalable Bayesian Optimization Using Deep Neural Networks
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M. A., Prabhat, & Adams, R. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05700 · Cited by 613
Simulated annealing: Theory and applications
Rota, G. (1988) · Advances in Mathematics · doi:10.1016/0001-8708(88)90072-2 · Cited by 607
Multi-Task Bayesian Optimization
Swersky, K., Snoek, J., & Adams, R. P. (2013) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 502
On bayesian methods for seeking the extremum
Močkus, J. (1975) · Lecture Notes in Computer Science · doi:10.1007/3-540-07165-2_55 · Cited by 499
Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
Hernández-Lobato, J. M., Hoffman, M. W., & Ghahramani, Z. (2014) · arXiv · doi:10.48550/arxiv.1406.2541 · Cited by 407
Proceedings of the International Joint Conference on Neural Networks (IJCNN) 2005
(2005) · Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. · doi:10.1109/ijcnn.2005.1555790 · Cited by 382
Razmjooy, N., Khalilpour, M., & Ramezani, M. (2016) · Journal of Control Automation and Electrical Systems · doi:10.1007/s40313-016-0242-6 · Cited by 281
Automatic gait optimization with Gaussian process regression
Lizotte, D. J., Wang, T., Bowling, M., & Schuurmans, D. (2007) · Cited by 237
Wiley Encyclopedia of Computer Science and Engineering
(2008) · Wiley eBooks · doi:10.1002/9780470050118 · Cited by 210
Bayesian optimization for sensor set selection
Garnett, R., Osborne, M. A., & Roberts, S. (2010) · ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1145/1791212.1791238 · Cited by 146
Automated configuration of algorithms for solving hard computational problems
Hutter, F. (2009) · cIRcle (University of British Columbia) · doi:10.14288/1.0051652 · Cited by 141
Student-t Processes as Alternatives to Gaussian ProcessesPreprint
Shah, A., Wilson, A. G., & Zoubin Ghahramani, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.4306 · Cited by 126
Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints
Garrido‐Merchán, E. C., & Hernández-Lobato, D. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.025 · Cited by 105
Self-citationBayesian optimization of a hybrid system for robust ocean wave features prediction
Cornejo-Bueno, L. M., Garrido‐Merchán, E. C., Hernández-Lobato, D., & Salcedo‐Sanz, S. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.09.025 · Cited by 98
Self-citationBayesian optimization for conditional hyperparameter spaces
Lévesque, J.-C., Durand, A., Gagné, C., & Sabourin, R. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965867 · Cited by 44
Cited by
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., et al. (2023) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1484 · Cited by 930
Bayesian Optimization for Adaptive Experimental Design: A Review
Greenhill, S., Rana, S., Gupta, S., Vellanki, P., & Venkatesh, S. (2020) · IEEE Access · doi:10.1109/access.2020.2966228 · Cited by 520
Sun, D., Wen, H., Wang, D., & Xu, J. (2020) · Geomorphology · doi:10.1016/j.geomorph.2020.107201 · Cited by 480
Recent Advances in Bayesian Optimization
Wang, X., Jin, Y., Schmitt, S., & Olhofer, M. (2023) · ACM Computing Surveys · doi:10.1145/3582078 · Cited by 477
Benchmark and Survey of Automated Machine Learning Frameworks
Zöller, M.-A., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11854 · Cited by 448
Metadata
- Authors
- Garrido-Merchán, E. C., & Hernández-Lobato, D.
- Venue
- Neurocomputing
- Publisher
- Elsevier BV
- Published
- 2020-03
- Volume
- 380
- Pages
- 20-35
- ISSN
- 0925-2312 (ISSN-L)
- OpenAlex
- W2802235618
- Cited by
- 291
- References
- 34
- FWCI
- 19.50
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.