The DOI matches its Crossref record; 170 of the 170 references it cites were found in a registry or an index.
Benchmark and Survey of Automated Machine Learning Frameworks
Open accessAbstract
Machine learning (ML) has become a vital part in many aspects of our daily life. However, building well performing machine learning applications requires highly specialized data scientists and domain experts. Automated machine learning (AutoML) aims to reduce the demand for data scientists by enabling domain experts to build machine learning applications automatically without extensive knowledge of statistics and machine learning. This paper is a combination of a survey on current AutoML methods and a benchmark of popular AutoML frameworks on real data sets. Driven by the selected frameworks for evaluation, we summarize and review important AutoML techniques and methods concerning every step in building an ML pipeline. The selected AutoML frameworks are evaluated on 137 data sets from established AutoML benchmark suites.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392
Particle swarm optimization: An overview
Poli, R., Kennedy, J., & Blackwell, T. (2007) · Swarm Intelligence · doi:10.1007/s11721-007-0002-0 · Cited by 21,101
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 14,807
Genetic Programming: On the Programming of Computers by Means of Natural Selection
Koza, J. R. (1992) · Medical Entomology and Zoology · Cited by 13,082
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Classification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
On Estimation of a Probability Density Function and Mode
Parzen, E. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704472 · Cited by 10,585
Gene selection for cancer classification using support vector machines
Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002) · Machine Learning · doi:10.1023/a:1012487302797 · Cited by 9,954
(2012) · Cited by 8,009
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · cited 1×
“An interesting approach for ensemble learning is stacking (Wolpert, 1992).”
BackgroundFlocks, herds and schools: A distributed behavioral model
Reynolds, C. W. (1987) · Proceedings of the 14th annual conference on Computer graphics and interactive techniques · doi:10.1145/37401.37406 · Cited by 7,714
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
An Introduction to Variable and Feature Selection.
Guyon, I., & Elisseeff, A. (2003) · Cited by 6,576
Taking the Human Out of the Loop: A Review of Bayesian Optimization
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & de Freitas, N. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2494218 · Cited by 6,049
Evolutionary Algorithms for Solving Multi-Objective Problems
(2007) · Genetic and Evolutionary Computation Series · doi:10.1007/978-0-387-36797-2 · Cited by 5,686
A General Coefficient of Similarity and Some of Its Properties
Gower, J. C. (1971) · Biometrics · doi:10.2307/2528823 · Cited by 5,303
A review of feature selection techniques in bioinformatics
Saeys, Y., Inza, I., & Larrañaga, P. (2007) · Bioinformatics · doi:10.1093/bioinformatics/btm344 · Cited by 5,244
A Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,046
A Comparative Study on Feature Selection in Text Categorization
Yang, Y., & Pedersen, J. (1997) · Cited by 4,790
(2011) · Cited by 3,939
Feature selection for classification
DASH, M., & LIU, H. (1997) · Intelligent Data Analysis · doi:10.1016/s1088-467x(97)00008-5 · Cited by 3,574
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Floating search methods in feature selection
Pudil, P., Novovičová, J., & Kittler, J. (1994) · Pattern Recognition Letters · doi:10.1016/0167-8655(94)90127-9 · Cited by 3,098
The estimation of the gradient of a density function, with applications in pattern recognition
Fukunaga, K., & Hostetler, L. (1975) · IEEE Transactions on Information Theory · doi:10.1109/tit.1975.1055330 · Cited by 3,043
A Survey of Monte Carlo Tree Search Methods
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2012.2186810 · Cited by 2,970 · cited 2×
“Monte-Carlo tree search (Kocsis & Szepesvári, 2006; Browne et al., 2012) is a heuristic best-first tree search algorithm.”
MethodEstimating attributes: Analysis and extensions of RELIEF
Kononenko, I. (1994) · Lecture Notes in Computer Science · doi:10.1007/3-540-57868-4_57 · Cited by 2,958
Popular Ensemble Methods: An Empirical Study
Opitz, D., & Maclin, R. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.614 · Cited by 2,953 · cited 1×
“A well-known concept in ML is ensemble learning (Opitz & Maclin, 1999; Polikar, 2006; Rokach, 2010).”
BackgroundEnsemble based systems in decision making
Polikar, R. (2006) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2006.1688199 · Cited by 2,938
Bandit Based Monte-Carlo Planning
Kocsis, L., & Szepesvári, C. (2006) · Lecture Notes in Computer Science · doi:10.1007/11871842_29 · Cited by 2,855
Rokach, L. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-009-9124-7 · Cited by 2,657
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Choosing Multiple Parameters for Support Vector Machines
Chapelle, O., Vapnik, V., Bousquet, O., & Mukherjee, S. (2002) · Machine Learning · doi:10.1023/a:1012450327387 · Cited by 2,206
Some aspects of the sequential design of experiments
Robbins, H. (1952) · Bulletin of the American Mathematical Society · doi:10.1090/s0002-9904-1952-09620-8 · Cited by 2,184
Petri, C. A. (1962) · Cited by 2,159
Brochu, E., Cora, V. M., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2599 · Cited by 2,011
Stochastic Gradient Descent Tricks
Bottou, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_25 · Cited by 1,929
AutoML: A survey of the state-of-the-art
He, X., Zhao, K., & Chu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,691
Minimization by Random Search Techniques
Solis, F. J., & Wets, R. J.-B. (1981) · Mathematics of Operations Research · doi:10.1287/moor.6.1.19 · Cited by 1,671
(2000) · Cited by 1,467
Data preparation for data mining
Pyle, D. (1999) · Cited by 1,368
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
(2015) · neural information processing systems · Cited by 1,305
OpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2641190.2641198 · Cited by 1,141
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 3×
“Self-play (Lake et al., 2017) is a reinforcement learning strategy that has received a lot of attention lately due to the recent successes of AlphaZero (Silver et al., 2017).”
BackgroundInfluentialGenetic Programming: An Introduction
Banzhaf, W., Keller, R. E., & Nordin, P. C. (1997) · Cited by 1,052
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06560 · Cited by 1,018
ParamILS: An Automatic Algorithm Configuration Framework
Hutter, F., Hoos, H. H., Leyton-Brown, K., & Stuetzle, T. (2009) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2861 · Cited by 882
Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms
Bergstra, J., Yamins, D., & Cox, D. (2013) · Proceedings of the Python in Science Conference · doi:10.25080/majora-8b375195-003 · Cited by 781
An Efficient Approach for Assessing Hyperparameter Importance
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2014) · International Conference on Machine Learning · Cited by 736
(2007) · Choice Reviews Online · doi:10.5860/choice.44-2754 · Cited by 726
Variable selection using svm based criteria
Rakotomamonjy, A. (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944977 · Cited by 712
Mixed-integer nonlinear optimization
Belotti, P., Kirches, C., Leyffer, S., Linderoth, J., Luedtke, J., & Mahajan, A. (2013) · Acta Numerica · doi:10.1017/s0962492913000032 · Cited by 712
Non-stochastic Best Arm Identification and Hyperparameter Optimization
Jamieson, K., & Talwalkar, A. (2016) · International Conference on Artificial Intelligence and Statistics · Cited by 681
Domhan, T., Springenberg, J. T., & Hutter, F. (2015) · FreiDok plus (Universitätsbibliothek Freiburg) · Cited by 670
Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA
Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F., & Leyton-Brown, K. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_4 · Cited by 650
Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science
Olson, R. S., Bartley, N., Urbanowicz, R. J., & Moore, J. H. (2016) · Proceedings of the Genetic and Evolutionary Computation Conference 2016 · doi:10.1145/2908812.2908918 · Cited by 595
Google Vizier: A Service for Black-Box Optimization
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098043 · Cited by 571
Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Probst, P., Bischl, B., & Boulesteix, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.09596 · Cited by 565
BOHB: Robust and Efficient Hyperparameter Optimization at Scale
Falkner, S., Klein, A., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1807.01774 · Cited by 551
TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning
Olson, R. S., & Moore, J. H. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_8 · Cited by 547
Samanta, B. (2004) · Mechanical Systems and Signal Processing · doi:10.1016/s0888-3270(03)00020-7 · Cited by 517
High Performance Cluster Computing: Architectures and Systems
Buyya, R. (1999) · Prentice Hall PTR eBooks · Cited by 511
Gradient-based Hyperparameter Optimization through Reversible Learning
Maclaurin, D., Duvenaud, D., & Adams, R. P. (2015) · arXiv · doi:10.48550/arxiv.1502.03492 · Cited by 509
Potter's Wheel: An Interactive Data Cleaning System
Raman, V., & Hellerstein, J. L. (2001) · Cited by 492
Data Cleaning: Overview and Emerging Challenges
Chu, X., Ilyas, I. F., Krishnan, S., & Wang, J. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2912574 · Cited by 488
HoloClean: holistic data repairs with probabilistic inference
Rekatsinas, T., Chu, X., Ilyas, I. F., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137631 · Cited by 480
Entropy search for information-efficient global optimization
Hennig, P., & Schuler, C. J. (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2343701 · Cited by 443
Fast Bayesian Optimization of Machine Learning Hyperparameters on Large DatasetsPreprint
Klein, A., Falkner, S., Bartels, S., Hennig, P., & Hutter, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07079 · Cited by 424
Deep feature synthesis: Towards automating data science endeavors
Kanter, J. M., & Veeramachaneni, K. (2015) · 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344858 · Cited by 421
Kriging Is Well-Suited to Parallelize Optimization
Ginsbourger, D., Le Riche, R., & Carraro, L. (2010) · Adaptation Learning and Optimization · doi:10.1007/978-3-642-10701-6_6 · Cited by 396
An Economics Approach to Hard Computational Problems
Huberman, B. A., Lukose, R. M., & Hogg, T. (1997) · Science · doi:10.1126/science.275.5296.51 · Cited by 390
Initializing Bayesian Hyperparameter Optimization via Meta-Learning
Feurer, M., Springenberg, J., & Hutter, F. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9354 · Cited by 390
Feature Selection with Ensembles, Artificial Variables, and Redundancy Elimination
Tuv, E., Borisov, A., Runger, G. C., & Torkkola, K. (2009) · Journal of Machine Learning Research · doi:10.5555/1577069.1755828 · Cited by 386
Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification AlgorithmsPreprint
Thornton, C., Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1208.3719 · Cited by 374
Automated Machine Learning: State-of-The-Art and Open ChallengesPreprint
Shawi, R. E., Maher, M., & Sakr, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02287 · Cited by 361
A review of automatic selection methods for machine learning algorithms and hyper-parameter values
Luo, G. (2016) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-016-0125-6 · Cited by 357
Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit OptimizationPreprint
Desautels, T., Krause, A., & Burdick, J. W. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6402 · Cited by 353
Taking Human out of Learning Applications: A Survey on Automated Machine LearningPreprint
Yao, Q., Wang, M., Escalante, H. J., Guyon, I., Hu, Y.-Q., Li, Y., Tu, W.-W., Yang, Q., & Yang, Y. (2018) · arXiv (Cornell University) · Cited by 353
A Tutorial on Bayesian Optimization
Frazier, P. I. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02811 · Cited by 342
On the Relationship between Classical Grid Search and Probabilistic Roadmaps
LaValle, S. M., & Branicky, M. S. (2004) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-540-45058-0_5 · Cited by 338
Automating Biomedical Data Science Through Tree-Based Pipeline Optimization
Olson, R. S., Urbanowicz, R. J., Andrews, P. C., Lavender, N. A., Kidd, L. C., & Moore, J. H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-31204-0_9 · Cited by 329
Freeze-Thaw Bayesian OptimizationPreprint
Swersky, K., Snoek, J., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3896 · Cited by 310
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn
Komer, B., Bergstra, J., & Eliasmith, C. (2014) · Proceedings of the Python in Science Conference · doi:10.25080/majora-14bd3278-006 · Cited by 295
Garrido-Merchán, E. C., & Hernández-Lobato, D. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.004 · Cited by 291
Learning Feature Engineering for Classification
Nargesian, F., Samulowitz, H., Khurana, U., Khalil, E. B., & Turaga, D. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/352 · Cited by 287
KATARA: A Data Cleaning System Powered by Knowledge Bases and Crowdsourcing
Chu, X., Morcos, J., Ilyas, I. F., Ouzzani, M., Papotti, P., Tang, N., & Ye, Y. (2015) · Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2723372.2749431 · Cited by 279
Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation
Maron, O., & Moore, A. (2018) · doi:10.1184/r1/6555050 · Cited by 265
ActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264
Declarative Support for Sensor Data Cleaning
Jeffery, S. R., Alonso, G., Franklin, M. J., Hong, W., & Widom, J. (2006) · Lecture Notes in Computer Science · doi:10.1007/11748625_6 · Cited by 249
Genetic algorithms as a tool for feature selection in machine learning
Vafaie, H., & Jong, K. A. D. (2003) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/tai.1992.246402 · Cited by 239
A System for Massively Parallel Hyperparameter Tuning
Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Ben-tzur, J., Hardt, M., Recht, B., & Talwalkar, A. (2020) · Proceedings of Machine Learning and Systems · Cited by 236
Munos, R. (2014) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000038 · Cited by 233
Non-stochastic Best Arm Identification and Hyperparameter Optimization
Jamieson, K., & Talwalkar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.07943 · Cited by 232
Hyperparameter optimization with approximate gradient
Pedregosa, F. (2016) · arXiv · doi:10.48550/arxiv.1602.02355 · Cited by 217
Tran, B., Xue, B., & Zhang, M. (2016) · Memetic Computing · doi:10.1007/s12293-015-0173-y · Cited by 217
Hyperparameter Importance Across Datasets
van Rijn, J. N., & Hutter, F. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220058 · Cited by 210
Automatic Parameter Selection by Minimizing Estimated Error
Kohavi, R., & John, G. H. (1995) · Machine Learning Proceedings 1995 · doi:10.1016/b978-1-55860-377-6.50045-1 · Cited by 202
ExploreKit: Automatic Feature Generation and Selection
Katz, G., Shin, E. C. R., & Song, D. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0123 · Cited by 202
Feature Engineering for Predictive Modeling Using Reinforcement Learning
Khurana, U., Samulowitz, H., & Turaga, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11678 · Cited by 202
Quantitative Data Cleaning for Large Databases
Hellerstein, J. M. (2008) · Cited by 201
Analysis of the AutoML Challenge Series 2015–2018
Guyon, I., Sun-Hosoya, L., Boullé, M., Escalante, H. J., Escalera, S., Liu, Z., Jajetic, D., Ray, B., Saeed, M., Sébag, M., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-05318-5_10 · Cited by 196
Meta-learning for evolutionary parameter optimization of classifiers
Reif, M., Faisal Shafait, B., & Dengel, A. R. (2012) · Machine Learning · doi:10.1007/s10994-012-5286-7 · Cited by 193
Automating model search for large scale machine learning
Sparks, E., Talwalkar, A., Haas, D., Franklin, M. J. M., Jordan, M. I., & Kraska, T. (2015) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2806777.2806945 · Cited by 182
Combining meta-learning and search techniques to select parameters for support vector machines
Gomes, T. A. F., Prudêncio, R. B. C., Soares, C., Rossi, A. L. D., & de Carvalho, A. C. P. L. F. (2011) · Neurocomputing · doi:10.1016/j.neucom.2011.07.005 · Cited by 178
Learning Curve Prediction with Bayesian Neural Networks
Klein, A., Falkner, S., Springenberg, J. T., & Hutter, F. (2017) · International Conference on Learning Representations · Cited by 176
AJAX: an extensible data cleaning tool
Galhardas, H., Florescu, D., Shasha, D., & Simon, E. (2000) · ACM SIGMOD Record · doi:10.1145/335191.336568 · Cited by 174
ML-Plan: Automated machine learning via hierarchical planning
Mohr, F., Wever, M., & Hüllermeier, E. (2018) · Machine Learning · doi:10.1007/s10994-018-5735-z · Cited by 174
BigDansing: A System for Big Data Cleansing
Khayyat, Z., Ilyas, I. F., Jindal, A., Madden, S. R., Ouzzani, M., Papotti, P., Quiané-Ruiz, J.-A., Tang, N., & Yin, S. (2015) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2723372.2747646 · Cited by 163
Cognito: Automated Feature Engineering for Supervised Learning
Khurana, U., Turaga, D. S., Samulowitz, H., & Srinivasan, P. (2016) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2016.0190 · Cited by 159
A Pattern Search Method for Model Selection of Support Vector Regression
Momma, M., & Bennett, K. P. (2002) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972726.16 · Cited by 155
AutoLearn — Automated Feature Generation and Selection
Kaul, A., Maheshwary, S., & Pudi, V. (2017) · Proceedings · doi:10.1109/icdm.2017.31 · Cited by 153
A meta-learning approach to automatic kernel selection for support vector machines
Ali, S., & Smith‐Miles, K. (2006) · Neurocomputing · doi:10.1016/j.neucom.2006.03.004 · Cited by 148
Genetic Programming with a Genetic Algorithm for Feature Construction and Selection
Smith, M. G., & Bull, L. (2005) · Genetic Programming and Evolvable Machines · doi:10.1007/s10710-005-2988-7 · Cited by 144
Feature Generation Using General Constructor Functions
Markovitch, S., & Rosenstein, D. (2002) · Machine Learning · doi:10.1023/a:1014046307775 · Cited by 143
Efficient Hyperparameter Optimization and Infinitely Many Armed BanditsPreprint
Rostamizadeh, A., Talwalkar, A., DeSalvo, G., Jamieson, K., & Li, L. (2017) · arXiv (Cornell University) · Cited by 143
Characterization of classification algorithms
da Gama, J. M. P., & Brazdil, P. (1995) · Lecture notes in computer science · doi:10.1007/3-540-60428-6_16 · Cited by 123
Tunability: Importance of Hyperparameters of Machine Learning AlgorithmsPreprint
Probst, P., Boulesteix, A., & Bischl, B. (2019) · arXiv (Cornell University) · Cited by 123
Meta analysis of classification algorithms for pattern recognition
Sohn, S. Y. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.809107 · Cited by 122
ATM: A distributed, collaborative, scalable system for automated machine learning
Swearingen, T., Drevo, W., Cyphers, B., Cuesta‐Infante, A., Ross, A. A., & Veeramachaneni, K. K. (2017) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2017.8257923 · Cited by 120
Alaa, A. M., & van der Schaar, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07207 · Cited by 118
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates
Eggensperger, K., Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9375 · Cited by 116
Easy Hyperparameter Search Using OptunityPreprint
Claesen, M., Simm, J., Popovic, D. D., Moreau, Y., & Moor, B. D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1114 · Cited by 113
RECIPE: A Grammar-Based Framework for Automatically Evolving Classification Pipelines
de Sá, A. G. C., Pinto, W. J. G. S., Oliveira, L. O. V. B., & Pappa, G. L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-55696-3_16 · Cited by 112
Automated Machine Learning in Practice: State of the Art and Recent Results
Tuggener, L., Amirian, M., Rombach, K., Lörwald, S., Varlet, A., Westermann, C. E., & Stadelmann, T. (2019) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds.2019.00-11 · Cited by 108
Learning hyperparameter optimization initializations
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344817 · Cited by 106
Koch, P. N., Golovidov, O., Gardner, S., Wujek, B. A., Griffin, J., & Xu, Y. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219837 · Cited by 95
Hoffman, M. D., Shahriari, B., & de Freitas, N. (2014) · Oxford University Research Archive (ORA) (University of Oxford) · Cited by 88
Feature Selection as a One-Player Game
Gaudel, R., & Sébag, M. (2010) · Cited by 86
Towards human-guided machine learning
Gil, Y., Honaker, J., Gupta, S., Ma, Y., D'Orazio, V. J., Garijo, D., Gadewar, S. P., Yang, Q., & Jahanshad, N. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302324 · Cited by 84
Parallel Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton‐Brown, K. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34413-8_5 · Cited by 82
Autostacker: a compositional evolutionary learning system
Chen, B., Wu, H., Mo, W., Chattopadhyay, I., & Lipson, H. (2018) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3205455.3205586 · Cited by 80
Recent Advances in Finding Best Operating Conditions
Anderson, R. L. (1953) · Journal of the American Statistical Association · doi:10.1080/01621459.1953.10501200 · Cited by 74
Fast Algorithm Selection Using Learning Curves
van Rijn, J. N., Abdulrahman, S. M., Brazdil, P., & Vanschoren, J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24465-5_26 · Cited by 71
Dealing with asynchronicity in parallel Gaussian Process based global optimization
Ginsbourger, D., Januševskis, J., & Riche, R. L. (2011) · HAL · Cited by 70
AlphaClean: Automatic Generation of Data Cleaning PipelinesPreprint
Krishnan, S., & Wu, E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11827 · Cited by 70
Efficient benchmarking of algorithm configurators via model-based surrogates
Eggensperger, K., Lindauer, M., Hoos, H. H., Hutter, F., & Leyton‐Brown, K. (2017) · Machine Learning · doi:10.1007/s10994-017-5683-z · Cited by 68
Guyon, I., Chaabane, I., Escalante, H. J., Escalera, S., Jajetic, D., Lloyd, J. R., Macià, N., Ray, B., Romaszko, Ł., Sébag, M., et al. (2016) · HAL · Cited by 60
Model selection of SVMs using GA approach
Chen, P.-W., Wang, J., & Lee, H.-M. (2005) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2004.1380929 · Cited by 58
One button machine for automating feature engineering in relational databasesPreprint
Lam, H. T., Thiebaut, J.-M., Sinn, M., Chen, B., Mai, T., & Alkan, Ö. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.00327 · Cited by 57
HoloClean: Holistic Data Repairs with Probabilistic InferencePreprint
Rekatsinas, T., Chu, X., Ilyas, I. F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.00820 · Cited by 54
On the predictive power of meta-features in OpenML
Bilalli, B., Abelló, A., & Aluja‐Banet, T. (2017) · International Journal of Applied Mathematics and Computer Science · doi:10.1515/amcs-2017-0048 · Cited by 53
OpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., et al. (2014) · doi:10.48550/arxiv.1407.7722 · Cited by 53
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-23525-7_7 · Cited by 52
SampleClean: Fast and Reliable Analytics on Dirty Data.
Krishnan, S., Wang, J., Franklin, M. J. M., Goldberg, K., Kraska, T., Milo, T., & Wu, E. (2015) · IEEE Data(base) Engineering Bulletin · Cited by 45
Sequential Model-Based Ensemble OptimizationPreprint
Lacoste, A., Larochelle, H., Laviolette, F., & Marchand, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.0796 · Cited by 45
Strengthening learning algorithms by feature discovery
Dor, O., & Reich, Y. (2011) · Information Sciences · doi:10.1016/j.ins.2011.11.039 · Cited by 44
Bayesian optimization for conditional hyperparameter spaces
Lévesque, J.-C., Durand, A., Gagné, C., & Sabourin, R. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965867 · Cited by 44
Automated Machine Learning with Monte-Carlo Tree Search
Rakotoarison, H., Schoenauer, M., & Sébag, M. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/457 · Cited by 44
FLASH: Fast Bayesian Optimization for Data Analytic Pipelines
Zhang, Y., Bahadori, M. T., Su, H., & Sun, J. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939829 · Cited by 38
Particle Swarm Model Selection for Authorship Verification
Escalante, H. J., Montes, M., & Villaseñor-Pineda, L. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-10268-4_66 · Cited by 37
OpenML Benchmarking Suites and the OpenML100.Preprint
Bischl, B., Casalicchio, G., Feurer, M., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., & Vanschoren, J. (2017) · arXiv (Cornell University) · Cited by 36
Drori, I., Krishnamurthy, Y., Lourenço, R., Rampin, R., Cho, K., Silva, C. T., & Freire, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10345 · Cited by 36
Automatic Frankensteining: Creating Complex Ensembles Autonomously
Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2017) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611974973.83 · Cited by 34
Autostacker: A Compositional Evolutionary Learning SystemPreprint
Chen, B., Wu, H., Mo, W., Chattopadhyay, I., & Lipson, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.00684 · Cited by 33
Analysing the Overfit of the Auto-sklearn Automated Machine Learning Tool
Fabris, F., & Freitas, A. A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-37599-7_42 · Cited by 26
Automated Machine Learning on Big Data using Stochastic Algorithm TuningPreprint
Nickson, T., Osborne, M. A., Reece, S., & Roberts, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.7969 · Cited by 25
FeatureHub: Towards Collaborative Data Science
Smith, M. J., Wedge, R., & Veeramachaneni, K. K. (2017) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2017.66 · Cited by 25
Warmstarting of Model-Based Algorithm Configuration
Lindauer, M., & Hutter, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11532 · Cited by 23
Towards Automatic Composition of Multicomponent Predictive Systems
Salvador, M. M., Budka, M., & Gabryś, B. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-32034-2_3 · Cited by 22
Toward Provably Correct Feature Selection in Arbitrary Domains
Margaritis, D. (2009) · Cited by 18
de Miranda, P. B. C., Prudêncio, R. B. C., de Carvalho, A. C. P. L. F., & Soares, C. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-31137-6_43 · Cited by 15
Recursive Feature Generation for Knowledge-based LearningPreprint
Friedman, L., & Markovitch, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00050 · Cited by 11
Modelling Multi-Component Predictive Systems as Petri Nets
Salvador, M. M., Budka, M., & Gabryś, B. (2017) · Bournemouth University Research Online (Bournemouth University) · Cited by 11
Cited by
On hyperparameter optimization of machine learning algorithms: Theory and practice
Yang, L., & Shami, A. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.061 · Cited by 3,232
AutoML: A survey of the state-of-the-art
He, X., Zhao, K., & Chu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106622 · Cited by 1,691
González, S., García, S., Del Ser, J., Rokach, L., & Herrera, F. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.07.007 · Cited by 582
Machine learning in agriculture domain: A state-of-art survey
Meshram, V., Patil, K., Meshram, V., Hanchate, D., & Ramkteke, S. D. (2021) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2021.100010 · Cited by 367
Data-centric Artificial Intelligence: A Survey
Zha, D., Bhat, Z. P., Lai, K.-H., Yang, F., Jiang, Z., Zhong, S., & Hu, X. (2025) · ACM Computing Surveys · doi:10.1145/3711118 · Cited by 334
Metadata
- Authors
- Zöller, M.-A., & Huber, M. F.
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2021-01-27
- Volume
- 70
- Pages
- 409-472
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2999356315
- Cited by
- 448
- References
- 170
- FWCI
- 26.29
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.