The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Constrained Bayesian Optimization with Noisy Experiments
Open accessAbstract
Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multiple continuous parameters, but existing approaches degrade in performance when the noise level is high, limiting its applicability to many randomized experiments. We derive an expression for expected improvement under greedy batch optimization with noisy observations and noisy constraints, and develop a quasi-Monte Carlo approximation that allows it to be efficiently optimized. Simulations with synthetic functions show that optimization performance on noisy, constrained problems outperforms existing methods. We further demonstrate the effectiveness of the method with two real-world experiments conducted at Facebook: optimizing a ranking system, and optimizing server compiler flags.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Global Optimization of Expensive Black-Box Functions
Jones, D. R., Schonlau, M., & Welch, W. J. (1998) · Journal of Global Optimization · doi:10.1023/a:1008306431147 · Cited by 8,070
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Deep Neural Networks for YouTube Recommendations
Covington, P., Adams, J., & Sargin, E. (2016) · Proceedings of the 10th ACM Conference on Recommender Systems · doi:10.1145/2959100.2959190 · Cited by 3,539
The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte CarloPreprint
Homan, M. D., & Gelman, A. (2014) · arXiv (Cornell University) · Cited by 3,405
THOMPSON, W. R. (1933) · Biometrika · doi:10.1093/biomet/25.3-4.285 · Cited by 2,712
Monte Carlo and quasi-Monte Carlo methods
Caflisch, R. E. (1998) · Acta Numerica · doi:10.1017/s0962492900002804 · Cited by 1,754
The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
Hoffman, M. D., & Gelman, A. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.4246 · Cited by 1,713
Bayesian Optimization with Inequality Constraints
Gardner, J. R., Kusner, M. J., Zhixiang, Weinberger, K. Q., & Cunningham, J. P. (2014) · PolyPublie (École Polytechnique de Montréal) · Cited by 764
Estimating Individualized Treatment Rules Using Outcome Weighted Learning
Zhao, Y., Zeng, D., Rush, A. J., & Kosorok, M. R. (2012) · Journal of the American Statistical Association · doi:10.1080/01621459.2012.695674 · Cited by 739
Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models
Huang, D., Allen, T. T., Notz, W. I., & Zeng, N. (2006) · Journal of Global Optimization · doi:10.1007/s10898-005-2454-3 · Cited by 625
High-dimensional integration: The quasi-Monte Carlo way
Dick, J., Kuo, F. Y., & Sloan, I. H. (2013) · Acta Numerica · doi:10.1017/s0962492913000044 · Cited by 615
Convergence rates of efficient global optimization algorithms
Bull, A. D. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1101.3501 · Cited by 446
Entropy search for information-efficient global optimization
Hennig, P., & Schuler, C. J. (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2343701 · Cited by 443
Global versus local search in constrained optimization of computer models
Schonlau, M., Welch, W. J., & Jones, D. R. (1998) · Institute of Mathematical Statistics Lecture Notes - Monograph Series · doi:10.1214/lnms/1215456182 · Cited by 435
Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
Hernández-Lobato, J. M., Hoffman, M. W., & Ghahramani, Z. (2014) · arXiv · doi:10.48550/arxiv.1406.2541 · Cited by 407
Doubly Robust Policy Evaluation and Optimization
Dudík, M., Erhan, D., Langford, J., & Li, L. (2014) · Statistical Science · doi:10.1214/14-sts500 · Cited by 387
A benchmark of kriging-based infill criteria for noisy optimization
Picheny, V., Wagner, T., & Ginsbourger, D. (2013) · Structural and Multidisciplinary Optimization · doi:10.1007/s00158-013-0919-4 · Cited by 325
An informational approach to the global optimization of expensive-to-evaluate functions
Villemonteix, J., Vazquez, E., & Walter, E. (2009) · Journal of Global Optimization · doi:10.1007/s10898-008-9354-2 · Cited by 308
Bayesian Optimization with Unknown Constraints
Gelbart, M. A., Snoek, J., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.5607 · Cited by 299
Input Warping for Bayesian Optimization of Non-stationary FunctionsPreprint
Snoek, J., Swersky, K., Zemel, R., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.0929 · Cited by 240
Seven rules of thumb for web site experimenters
Kohavi, R., Deng, A., Longbotham, R., & Xu, Y. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623341 · Cited by 226
Gentle, J. E. (2009) · Statistics and Computing · doi:10.1007/978-0-387-98144-4 · Cited by 216
Scott, W., Frazier, P., & Powell, W. (2011) · SIAM Journal on Optimization · doi:10.1137/100801275 · Cited by 209
Scrambling Sobol' and Niederreiter–Xing Points
Owen, A. B. (1998) · Journal of Complexity · doi:10.1006/jcom.1998.0487 · Cited by 207
The effective dimension and quasi-Monte Carlo integration
Wang, X., & Fang, K. (2003) · Journal of Complexity · doi:10.1016/s0885-064x(03)00003-7 · Cited by 200
Quantile-Based Optimization of Noisy Computer Experiments With Tunable Precision
Picheny, V., Ginsbourger, D., Richet, Y., & Caplin, G. (2013) · Technometrics · doi:10.1080/00401706.2012.707580 · Cited by 184
Replication or Exploration? Sequential Design for Stochastic Simulation Experiments
Binois, M., Huang, J., Gramacy, R. B., & Ludkovski, M. (2018) · Technometrics · doi:10.1080/00401706.2018.1469433 · Cited by 164
Gramacy, R. B., & Taddy, M. A. (2010) · Journal of Statistical Software · doi:10.18637/jss.v033.i06 · Cited by 162
Parallelised Bayesian Optimisation via Thompson Sampling
Kandasamy, K., Krishnamurthy, A., Schneider, J., & Póczos, B. (2018) · International Conference on Artificial Intelligence and Statistics · Cited by 154
The Parallel Knowledge Gradient Method for Batch Bayesian OptimizationPreprint
Wu, J., & Frazier, P. I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04414 · Cited by 148
Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection
Chevalier, C., & Ginsbourger, D. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-44973-4_7 · Cited by 147
Optimization Under Unknown Constraints*
Gramacy, R. B., & Lee, H. K. H. (2011) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199694587.003.0008 · Cited by 143
Modeling an Augmented Lagrangian for Blackbox Constrained Optimization
Gramacy, R. B., Gray, G. A., Digabel, S. L., Lee, H. K. H., Ranjan, P., Wells, G. N., & Wild, S. M. (2015) · Technometrics · doi:10.1080/00401706.2015.1014065 · Cited by 127
Marco, A., Berkenkamp, F., Hennig, P., Schoellig, A. P., Krause, A., Schaal, S., & Trimpe, S. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989186 · Cited by 120
Multiple objective optimization in recommender systems
Rodriguez, M., Posse, C., & Zhang, E. (2012) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2365952.2365961 · Cited by 111
Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective FunctionsPreprint
Shah, A., & Zoubin Ghahramani, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07130 · Cited by 108
Predictive Entropy Search for Bayesian Optimization with Unknown ConstraintsPreprint
Hernández-Lobato, J. M., Gelbart, M. A., Hoffman, M. D., Adams, R. P., & Zoubin Ghahramani, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.05312 · Cited by 102
Comparison of Kriging-based algorithms for simulation optimization with heterogeneous noise
Jalali, H., Nieuwenhuyse, I. V., & Picheny, V. (2017) · European Journal of Operational Research · doi:10.1016/j.ejor.2017.01.035 · Cited by 100
Parallel and Distributed Thompson Sampling for Large-scale Accelerated Exploration of Chemical SpacePreprint
Hernández-Lobato, J. M., Requeima, J., Pyzer‐Knapp, E. O., & Aspuru‐Guzik, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.01825 · Cited by 98
Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned
Deng, A., & Shi, X. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939700 · Cited by 86
Bayesian Guided Pattern Search for Robust Local Optimization
Taddy, M. A., Lee, H. K. H., Gray, G. A., & Griffin, J. D. (2009) · Technometrics · doi:10.1198/tech.2009.08007 · Cited by 80
Dealing with asynchronicity in parallel Gaussian Process based global optimization
Ginsbourger, D., Januševskis, J., & Riche, R. L. (2011) · HAL · Cited by 70
Using trajectory data to improve bayesian optimization for reinforcement learning
Wilson, A., Fern, A., & Tadepalli, P. V. (2014) · doi:10.5555/2627435.2627443 · Cited by 63
Parallel Bayesian Global Optimization of Expensive FunctionsPreprint
Wang, J., Clark, S., Liu, E., & Frazier, P. I. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.05149 · Cited by 61
Bayesian optimization under mixed constraints with a slack-variable augmented LagrangianPreprint
Picheny, V., Gramacy, R. B., Wild, S. M., & Digabel, S. L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09466 · Cited by 53
Design and Analysis of Benchmarking Experiments for Distributed Internet Services
Bakshy, E., & Frachtenberg, E. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741082 · Cited by 51
Self-citationThe anatomy of an ad: structured indexing and retrieval for sponsored search
Bendersky, M., Gabrilovich, E., Josifovski, V., & Metzler, D. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772702 · Cited by 14
Efficient batch-sequential Bayesian optimization with moments of truncated Gaussian vectorsPreprint
Marmin, S., Chevalier, C., & Ginsbourger, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02700 · Cited by 13
Cited by
Lookman, T., Balachandran, P. V., Xue, D., & Yuan, R. (2019) · npj Computational Materials · doi:10.1038/s41524-019-0153-8 · Cited by 832
Prescriptive analytics: Literature review and research challenges
Lepenioti, K., Bousdekis, A., Apostolou, D., & Mentzas, G. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.04.003 · Cited by 501
“Typically, this process begins by evaluating a small number of randomly selected function values, and fitting a Gaussian process regression model to the results (Letham, Karrer, Ottoni, & Bakshy, 2019).”
Recent Advances in Bayesian Optimization
Wang, X., Jin, Y., Schmitt, S., & Olhofer, M. (2023) · ACM Computing Surveys · doi:10.1145/3582078 · Cited by 477
Application of Improved LightGBM Model in Blood Glucose Prediction
Wang, Y., & Wang, T. (2020) · Applied Sciences · doi:10.3390/app10093227 · Cited by 129
Algorithmic transparency and assessing effects of algorithmic ranking
Eckles, D. (2022) · doi:10.31235/osf.io/c8za6 · Cited by 12
Metadata
- Authors
- Letham, B., Karrer, B., Ottoni, G., & Bakshy, E.
- Venue
- Bayesian Analysis
- Publisher
- Institute of Mathematical Statistics
- Published
- 2019-06-01
- Volume
- 14
- Issue
- 2
- Pages
- 495-519
- ISSN
- 1931-6690 (ISSN-L)
- 1936-0975
- OpenAlex
- W2646476955
- Cited by
- 307
- References
- 48
- FWCI
- 24.19
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.