The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Google Vizier: A Service for Black-Box Optimization
Open accessAbstract
Any sufficiently complex system acts as a black box when it becomes easier to experiment with than to understand. Hence, black-box optimization has become increasingly important as systems have become more complex. In this paper we describe Google Vizier, a Google-internal service for performing black-box optimization that has become the de facto parameter tuning engine at Google. Google Vizier is used to optimize many of our machine learning models and other systems, and also provides core capabilities to Google's Cloud Machine Learning HyperTune subsystem. We discuss our requirements, infrastructure design, underlying algorithms, and advanced features such as transfer learning and automated early stopping that the service provides.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
A Simplex Method for Function Minimization
Nelder, J. A., & Mead, R. (1965) · The Computer Journal · doi:10.1093/comjnl/7.4.308 · Cited by 28,905 · cited 1×
“Another class of algorithms performs a local search by selecting points thatmaintain a search pattern, such as a simplex in the case of the classic Nelder–Mead algorithm [22].”
BackgroundPractical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
Completely Derandomized Self-Adaptation in Evolution Strategies
Hansen, N., & Ostermeier, A. (2001) · Evolutionary Computation · doi:10.1162/106365601750190398 · Cited by 4,397
(2011) · Cited by 3,939
Bostock, M., Ogievetsky, V., & Heer, J. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.185 · Cited by 3,165
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting
Srinivas, N., Krause, A., Kakade, S. M., & Seeger, M. W. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2182033 · Cited by 1,850
Cited by
Larian Studios Dev Team (2024) · doi:10.1145/3292500.3330701 · Cited by 8,618
“or cost-eective optimization method. The strategy for the termination of unpromising trials is often referred to as pruning in many literatures, and it is also well known as automated early stopping [5] [7]. We, however, refer to this functionality as pruning in order to distinguish it from the early stopping regulari…”
BackgroundMethodInfluentialRandaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223
RandAugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1909.13719 · Cited by 3,223
Machine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842
“In other words, any sufficiently complex system acts as a black box when it becomes easier to experiment with than to understand [8].”
BackgroundMethodUniversal Sentence Encoder for English
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2029 · Cited by 1,277
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · arXiv · doi:10.48550/arxiv.1803.11175 · Cited by 1,267
“Transfer task model hyperparamaters are tuned using a combination of Vizier (Golovin et al., 2017) and light manual tuning.”
MethodCan You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Self-citationAutomated machine learning: Review of the state-of-the-art and opportunities for healthcare
Waring, J., Lindvall, C., & Umeton, R. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101822 · Cited by 834
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., & Goodfellow, I. J. (2018) · arXiv · doi:10.48550/arxiv.1804.09170 · Cited by 772
In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images
Christiansen, E. M., Yang, S. J., Ando, D. M., Javaherian, A., Skibinski, G., Lipnick, S., Mount, E., O’Neil, A., Shah, K., Lee, A. K., et al. (2018) · Cell · doi:10.1016/j.cell.2018.03.040 · Cited by 635
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
Menghani, G. (2023) · ACM Computing Surveys · doi:10.1145/3578938 · Cited by 605
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.034 · Cited by 564
Tune: A Research Platform for Distributed Model Selection and Training
Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., & Stoica, I. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05118 · Cited by 567
Deep Neural Networks as Gaussian Processes
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., & Sohl‐Dickstein, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00165 · Cited by 529
Recent Advances in Bayesian Optimization
Wang, X., Jin, Y., Schmitt, S., & Olhofer, M. (2023) · ACM Computing Surveys · doi:10.1145/3582078 · Cited by 477
Benchmark and Survey of Automated Machine Learning Frameworks
Zöller, M.-A., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11854 · Cited by 448
Hyper-Parameter Optimization: A Review of Algorithms and Applications
Yu, T., & Zhu, H. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05689 · Cited by 437
Sayres, R., Taly, A., Rahimy, E., Blumer, K., Coz, D., Hammel, N., Krause, J., Narayanaswamy, A., Rastegar, Z., Wu, D., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2018.11.016 · Cited by 418
Derivative-free optimization methods
Larson, J., Menickelly, M., & Wild, S. M. (2019) · Acta Numerica · doi:10.1017/s0962492919000060 · Cited by 381
Shallue, C. J., & Vanderburg, A. (2018) · The Astronomical Journal · doi:10.3847/1538-3881/aa9e09 · Cited by 368
Using deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332
Self-citationVanschoren, J. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_2 · Cited by 328
Oboe: auto-tuning video ABR algorithms to network conditions
Akhtar, Z., Nam, Y. S., Govindan, R., Rao, S., Chen, J., Katz-Bassett, E., Ribeiro, B., Zhan, J., & Zhang, H. (2018) · Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication · doi:10.1145/3230543.3230558 · Cited by 327
The ML test score: A rubric for ML production readiness and technical debt reduction
Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017) · 2017 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2017.8258038 · Cited by 310
Self-citationAutoML to Date and Beyond: Challenges and Opportunities
Karmaker (“Santu”), S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., & Veeramachaneni, K. (2022) · ACM Computing Surveys · doi:10.1145/3470918 · Cited by 292
Autonomous navigation of stratospheric balloons using reinforcement learning
Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., & Wang, Z. (2020) · Nature · doi:10.1038/s41586-020-2939-8 · Cited by 286
Self-citationCho, H., Kim, Y., Lee, E., Choi, D., Lee, Y., & Rhee, W. (2020) · IEEE Access · doi:10.1109/access.2020.2981072 · Cited by 280
Sensitivity and Generalization in Neural Networks: an Empirical Study
Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08760 · Cited by 275
Accurate, scalable cohort variant calls using DeepVariant and GLnexus
Yun, T., Li, H., Chang, P.-C., Lin, M. F., Carroll, A., & McLean, C. Y. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btaa1081 · Cited by 269
BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
White, C., Neiswanger, W., & Savani, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17233 · Cited by 261
Tiwary, S., Levy, R., Gutenbrunner, P., Salinas Soto, F., Palaniappan, K. K., Deming, L., Berndl, M., Brant, A., Cimermancic, P., & Cox, J. (2019) · Nature Methods · doi:10.1038/s41592-019-0427-6 · Cited by 256
Machine Learning for Microcontroller-Class Hardware: A Review
Saha, S. S., Sandha, S. S., & Srivastava, M. (2022) · IEEE Sensors Journal · doi:10.1109/jsen.2022.3210773 · Cited by 253
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187
Floridi, L., Holweg, M., Taddeo, M., Amaya, J., Mökander, J., & Wen, Y. (2022) · doi:10.2139/ssrn.4064091 · Cited by 157
Machine-guided design of cell-type-targeting cis-regulatory elements
Gosai, S. J., Castro, R. I., Fuentes, N., Butts, J. C., Mouri, K., Alasoadura, M., Kales, S., Nguyen, T. T. L., Noche, R. R., Rao, A. S., et al. (2024) · Nature · doi:10.1038/s41586-024-08070-z · Cited by 142
Hyperparameter ensembles for robustness and uncertainty quantification
Wenzel, F., Snoek, J., Tran, D., & Jenatton, R. (2020) · arXiv · doi:10.48550/arxiv.2006.13570 · Cited by 122
Herbol, H. C., Hu, W., Frazier, P., Clancy, P., & Poloczek, M. (2018) · npj Computational Materials · doi:10.1038/s41524-018-0106-7 · Cited by 121
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118
Drozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119
Task-agnostic exoskeleton control via biological joint moment estimation
Molinaro, D. D., Scherpereel, K. L., Schonhaut, E. B., Evangelopoulos, G., Shepherd, M. K., & Young, A. J. (2024) · Nature · doi:10.1038/s41586-024-08157-7 · Cited by 107
Perrone, V., Donini, M., Zafar, M. B., Schmucker, R., Kenthapadi, K., & Archambeau, C. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462629 · Cited by 66
Measuring the Reliability of Reinforcement Learning Algorithms
Chan, S. C. Y., Fishman, S., Canny, J., Korattikara, A., & Guadarrama, S. (2019) · arXiv · doi:10.48550/arxiv.1912.05663 · Cited by 57
The Use of Machine Learning Models with Optuna in Disease Prediction
Lai, L.-H., Lin, Y.-L., Liu, Y.-H., Lai, J.-P., Yang, W.-C., Hou, H.-P., & Pai, P.-F. (2024) · Electronics · doi:10.3390/electronics13234775 · Cited by 30
Automatic Discovery of Privacy–Utility Pareto Fronts
Avent, B., González, J., Diethe, T., Paleyes, A., & Balle, B. (2020) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2020-0060 · Cited by 3
On Accelerating Edge AI: Optimizing Resource-Constrained Environments
Sander, J., Cohen, A., Dasari, V. R., Venable, B., & Jalaian, B. (2025) · arXiv · doi:10.48550/arxiv.2501.15014 · Cited by 2
Metadata
- Authors
- Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D.
- Venue
- Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2017-08-13
- Pages
- 1487-1495
- OpenAlex
- W2732547613
- Cited by
- 571
- References
- 27
- FWCI
- 33.03
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.