The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Open accessAbstract
Big Data applications are typically associated with systems involving large numbers of users, massive complex software systems, and large-scale heterogeneous computing and storage architectures. The construction of such systems involves many distributed design choices. The end products (e.g., recommendation systems, medical analysis tools, real-time game engines, speech recognizers) thus involve many tunable configuration parameters. These parameters are often specified and hard-coded into the software by various developers or teams. If optimized jointly, these parameters can result in significant improvements. Bayesian optimization is a powerful tool for the joint optimization of design choices that is gaining great popularity in recent years. It promises greater automation so as to increase both product quality and human productivity. This review paper introduces Bayesian optimization, highlights some of its methodological aspects, and showcases a wide range of applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Self-citationCompletely Derandomized Self-Adaptation in Evolution Strategies
Hansen, N., & Ostermeier, A. (2001) · Evolutionary Computation · doi:10.1162/106365601750190398 · Cited by 4,397
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
An introduction to MCMC for machine learning.
Andrieu, C., de Freitas, N., Doucet, A., & Jordan, M. I. (2003) · Machine Learning · doi:10.1023/a:1020281327116 · Cited by 2,451
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems
Bubeck, S., & Cesa-Bianchi, N. (2012) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000024 · Cited by 1,295
Martinez-Cantin, R., de Freitas, N., Brochu, E., Castellanos, J., & Doucet, A. (2009) · Autonomous Robots · doi:10.1007/s10514-009-9130-2 · Cited by 217
Cited by
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. (2017) · Nature · doi:10.1038/nature24270 · Cited by 9,084
Larian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618
“This way, the framework can identify the underlying concurrence relations after some number of independent samplings, and use the inferred concurrence relation to conduct user-selected relational sampling algorithms like CMA-ES [17] and GP-BO [18].”
MethodDeep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
Analyzing Learned Molecular Representations for Property Prediction
Yang, K., Swanson, K., Jin, W., Coley, C., Eiden, P., Gao, H., Guzman-Perez, A., Hopper, T., Kelley, B., Mathea, M., et al. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00237 · Cited by 1,923
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Burger, B., Maffettone, P. M., Gusev, V. V., Aitchison, C. M., Bai, Y., Wang, X., Li, X., Alston, B. M., Li, B., Clowes, R., et al. (2020) · Nature · doi:10.1038/s41586-020-2442-2 · Cited by 1,516
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Closed-loop optimization of fast-charging protocols for batteries with machine learning
Attia, P. M., Grover, A., Jin, N., Severson, K. A., Markov, T. M., Liao, Y.-H., Chen, M. H., Cheong, B., Perkins, N., Yang, Z., et al. (2020) · Nature · doi:10.1038/s41586-020-1994-5 · Cited by 1,180
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Bayesian reaction optimization as a tool for chemical synthesis
Shields, B. J., Stevens, J., Li, J., Parasram, M., Damani, F., Alvarado, J. I. M., Janey, J. M., Adams, R. P., & Doyle, A. G. (2021) · Nature · doi:10.1038/s41586-021-03213-y · Cited by 1,063
Self-citationMeta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
Automated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Elsken, T., Metzen, J. H., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_3 · Cited by 785
What do we need to build explainable AI systems for the medical domain?
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017) · arXiv · doi:10.48550/arxiv.1712.09923 · Cited by 636
Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety
Huckvale, K., Venkatesh, S., & Christensen, H. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0166-1 · Cited by 433
Emerging materials intelligence ecosystems propelled by machine learning
Batra, R., Song, L., & Ramprasad, R. (2020) · Nature Reviews Materials · doi:10.1038/s41578-020-00255-y · Cited by 446
AI for life: Trends in artificial intelligence for biotechnology
Holzinger, A., Keiblinger, K., Holub, P., Zatloukal, K., & Müller, H. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.02.001 · Cited by 438
A review of automatic selection methods for machine learning algorithms and hyper-parameter values
Luo, G. (2016) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-016-0125-6 · Cited by 357
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342
From Machine Learning to Explainable AI
Holzinger, A. (2018) · 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA) · doi:10.1109/disa.2018.8490530 · Cited by 306
The prospects of quantum computing in computational molecular biology
Outeiral, C., Strahm, M., Shi, J., Morris, G. M., Benjamin, S. C., & Deane, C. M. (2021) · WIREs Computational Molecular Science · doi:10.1002/wcms.1481 · Cited by 232
Kim, M., Chen, C., Wang, P., Mulvey, J. J., Yang, Y., Wun, C., Antman-Passig, M., Luo, H.-B., Cho, S., Long-Roche, K., et al. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00860-y · Cited by 214
Janet, J. P., Ramesh, S., Duan, C., & Kulik, H. J. (2020) · ACS Central Science · doi:10.1021/acscentsci.0c00026 · Cited by 200
SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
Lindauer, M., Eggensperger, K., Feurer, M., Biedenkapp, A., Deng, D., Benjamins, C., Ruhopf, T., Sass, R., & Hutter, F. (2021) · arXiv · doi:10.48550/arxiv.2109.09831 · Cited by 195
Machine-learned interatomic potentials by active learning: amorphous and liquid hafnium dioxide
Sivaraman, G., Krishnamoorthy, A. N., Baur, M., Holm, C., Stan, M., Csányi, G., Benmore, C., & Vázquez-Mayagoitia, Á. (2020) · npj Computational Materials · doi:10.1038/s41524-020-00367-7 · Cited by 194
Tackling climate change with machine learning
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K., Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont, N., Jaques, N., Waldman-Brown, A., et al. (2023) · ACM Computing Surveys · doi:10.1145/3485128 · Cited by 336
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171
A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials
Chatzilygeroudis, K., Vassiliades, V., Stulp, F., Calinon, S., & Mouret, J.-B. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2958211 · Cited by 160
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671573 · Cited by 142
Herbol, H. C., Hu, W., Frazier, P., Clancy, P., & Poloczek, M. (2018) · npj Computational Materials · doi:10.1038/s41524-018-0106-7 · Cited by 121
Acerbi, L., Dokka, K., Angelaki, D. E., & Ma, W. J. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006110 · Cited by 111
Artificial intelligence-driven approaches for materials design and discovery
Cheng, M., Fu, C.-L., Okabe, R., Chotrattanapituk, A., Boonkird, A., Hung, N. T., & Li, M. (2026) · Nature Materials · doi:10.1038/s41563-025-02403-7 · Cited by 109
Bradshaw, J., Matthews, A. G. D. G., & Ghahramani, Z. (2017) · arXiv · doi:10.48550/arxiv.1707.02476 · Cited by 109
Autonomous Molecular Design: Then and Now
Dimitrov, T., Kreisbeck, C., Becker, J. S., Aspuru-Guzik, A., & Saikin, S. K. (2019) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.9b01226 · Cited by 103
Bayesian optimization with known experimental and design constraints for chemistry applications
Hickman, R. J., Aldeghi, M., Häse, F., & Aspuru-Guzik, A. (2022) · Digital Discovery · doi:10.1039/d2dd00028h · Cited by 98
HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
Hong, S., Xu, Y., Khare, A., Priambada, S., Maher, K., Aljiffry, A., Sun, J., & Tumanov, A. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403212 · Cited by 89
A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
Ros, R., Bjarnason, E., & Runeson, P. (2017) · Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3084226.3084243 · Cited by 80
FederatedScope: A Flexible Federated Learning Platform for Heterogeneity
Xie, Y., Wang, Z., Gao, D., Chen, D., Yao, L., Kuang, W., Li, Y., Ding, B., & Zhou, J. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3579075.3579081 · Cited by 84
Self-citationHossain, S. M. Z., Sultana, N., Razzak, S. A., & Hossain, M. M. (2022) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2021.112016 · Cited by 75
Optimizing quantum heuristics with meta-learning
Wilson, M., Stromswold, R., Wudarski, F., Hadfield, S., Tubman, N. M., & Rieffel, E. G. (2021) · Quantum Machine Intelligence · doi:10.1007/s42484-020-00022-w · Cited by 79
HyperPower: Power- and memory-constrained hyper-parameter optimization for neural networks
Stamoulis, D., Cai, E., Juan, D.-C., & Marculescu, D. (2018) · 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE) · doi:10.23919/date.2018.8341973 · Cited by 67
Perrone, V., Donini, M., Zafar, M. B., Schmucker, R., Kenthapadi, K., & Archambeau, C. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462629 · Cited by 66
Epistemic Communities under Active Inference
Albarracin, M., Demekas, D., Ramstead, M. J. D., & Heins, C. (2022) · Entropy · doi:10.3390/e24040476 · Cited by 60
Wallert, J., Boberg, J., Kaldo, V., Mataix-Cols, D., Flygare, O., Crowley, J. J., Halvorsen, M., Ben Abdesslem, F., Boman, M., Andersson, E., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02133-3 · Cited by 48
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
Daxberger, E., & Hernández-Lobato, J. M. (2019) · arXiv · doi:10.48550/arxiv.1912.05651 · Cited by 45
Genetic Neural Architecture Search for automatic assessment of human sperm images
Miahi, E., Mirroshandel, S. A., & Nasr, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115937 · Cited by 39
Bayesian optimization for ternary complex prediction (BOTCP)
Rao, A., Tunjic, T. M., Brunsteiner, M., Müller, M., Fooladi, H., Gasbarri, C., & Weber, N. (2023) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2023.100072 · Cited by 25
Chemical design with GPU-based Ising machines
Mao, Z., Matsuda, Y., Tamura, R., & Tsuda, K. (2023) · Digital Discovery · doi:10.1039/d3dd00047h · Cited by 23
A review on machine learning-guided design of energy materials
Kim, S., Xu, J., Shang, W., Xu, Z., Lee, E., & Luo, T. (2024) · Progress in Energy · doi:10.1088/2516-1083/ad7220 · Cited by 19
Designing optimal behavioral experiments using machine learning
Valentin, S., Kleinegesse, S., Bramley, N. R., Seriès, P., Gutmann, M. U., & Lucas, C. G. (2024) · eLife · doi:10.7554/elife.86224 · Cited by 8
Ou, C., Buschek, D., Mayer, S., & Butz, A. (2022) · Mensch und Computer 2022 · doi:10.1145/3543758.3543761 · Cited by 7
Qorich, M., & El Ouazzani, R. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128405 · Cited by 6
On queue length estimation in urban traffic intersections via inductive loops
Sassella, A., Abbracciavento, F., Formentin, S., Bianchessi, A. G., & Savaresi, S. M. (2023) · 2023 American Control Conference (ACC) · doi:10.23919/acc55779.2023.10156258 · Cited by 6
Efficient Prompt Optimization Through the Lens of Best Arm Identification
Shi, C., Yang, K., Chen, Z., Li, J., Yang, J., & Shen, C. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3161 · Cited by 3
Uncertainty Quantification for Bayesian Optimization
Tuo, R., & Wang, W. (2020) · arXiv · doi:10.48550/arxiv.2002.01569 · Cited by 2
Metadata
- Authors
- Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N.
- Venue
- Proceedings of the IEEE
- Published
- 2016-01
- Volume
- 104
- Issue
- 1
- Pages
- 148-175
- ISSN
- 0018-9219 (ISSN-L)
- 1558-2256
- OpenAlex
- W2192203593
- Cited by
- 6,049
- References
- 10
- FWCI
- 145.48
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
