The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
AutoML to Date and Beyond: Challenges and Opportunities
Abstract
As big data becomes ubiquitous across domains, and more and more stakeholders aspire to make the most of their data, demand for machine learning tools has spurred researchers to explore the possibilities of automated machine learning (AutoML). AutoML tools aim to make machine learning accessible for non-machine learning experts (domain experts), to improve the efficiency of machine learning, and to accelerate machine learning research. But although automation and efficiency are among AutoML’s main selling points, the process still requires human involvement at a number of vital steps, including understanding the attributes of domain-specific data, defining prediction problems, creating a suitable training dataset, and selecting a promising machine learning technique. These steps often require a prolonged back-and-forth that makes this process inefficient for domain experts and data scientists alike and keeps so-called AutoML systems from being truly automatic. In this review article, we introduce a new classification system for AutoML systems, using a seven-tiered schematic to distinguish these systems based on their level of autonomy. We begin by describing what an end-to-end machine learning pipeline actually looks like, and which subtasks of the machine learning pipeline have been automated so far. We highlight those subtasks that are still done manually—generally by a data scientist—and explain how this limits domain experts’ access to machine learning. Next, we introduce our novel level-based taxonomy for AutoML systems and define each level according to the scope of automation support provided. Finally, we lay out a roadmap for the future, pinpointing the research required to further automate the end-to-end machine learning pipeline and discussing important challenges that stand in the way of this ambitious goal.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
A survey on feature selection methods
Chandrashekar, G., & Sahin, F. (2014) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2013.11.024 · Cited by 5,189
Data mining: practical machine learning tools and techniques with Java implementations
Witten, I. H., & Frank, E. (2002) · ACM SIGMOD Record · doi:10.1145/507338.507355 · Cited by 5,137
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487629 · Cited by 1,359
Neural Networks: Tricks of the Trade
(1998) · Lecture Notes in Computer Science · doi:10.1007/3-540-49430-8 · Cited by 810
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science
Olson, R. S., Bartley, N., Urbanowicz, R. J., & Moore, J. H. (2016) · Proceedings of the Genetic and Evolutionary Computation Conference 2016 · doi:10.1145/2908812.2908918 · Cited by 595
Google Vizier: A Service for Black-Box Optimization
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098043 · Cited by 571
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
A Survey on semi-supervised feature selection methods
Sheikhpour, R., Sarram, M. A., Gharaghani, S., & Chahooki, M. A. Z. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.003 · Cited by 505
Data Cleaning: Overview and Emerging Challenges
Chu, X., Ilyas, I. F., Krishnan, S., & Wang, J. (2016) · Proceedings of the 2016 International Conference on Management of Data · doi:10.1145/2882903.2912574 · Cited by 488
Deep feature synthesis: Towards automating data science endeavors
Kanter, J. M., & Veeramachaneni, K. (2015) · 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344858 · Cited by 421
Self-citationPMLB: a large benchmark suite for machine learning evaluation and comparison
Olson, R. S., La Cava, W., Orzechowski, P., Urbanowicz, R. J., & Moore, J. H. (2017) · BioData Mining · doi:10.1186/s13040-017-0154-4 · Cited by 365
Benchmarking State-of-the-Art Deep Learning Software Tools
Shi, S., Wang, Q., Xu, P., & Chu, X. (2016) · 2016 7th International Conference on Cloud Computing and Big Data (CCBD) · doi:10.1109/ccbd.2016.029 · Cited by 329
RapidMiner: Data Mining Use Cases and Business Analytics Applications
(2016) · Chapman & Hall/CRC eBooks · doi:10.1201/b16023 · Cited by 314
ActiveClean: interactive data cleaning for statistical modeling
Krishnan, S., Wang, J., Wu, E., Franklin, M. J., & Goldberg, K. (2016) · Proceedings of the VLDB Endowment · doi:10.14778/2994509.2994514 · Cited by 264
Beyond Manual Tuning of Hyperparameters
Hutter, F., Lücke, J., & Schmidt-Thieme, L. (2015) · KI - Künstliche Intelligenz · doi:10.1007/s13218-015-0381-0 · Cited by 215
ExploreKit: Automatic Feature Generation and Selection
Katz, G., Shin, E. C. R., & Song, D. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0123 · Cited by 202
On Application of Learning to Rank for E-Commerce Search
Santu, S. K. K., Sondhi, P., & Zhai, C. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080838 · Cited by 149
Self-citationZhang, A., Song, S., Wang, J., & Yu, P. S. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3115404.3115410 · Cited by 130
Trends in Cleaning Relational Data: Consistency and Deduplication
Ilyas, I. F., & Chu, X. (2015) · Foundations and Trends in Databases · doi:10.1561/1900000045 · Cited by 127
AutoLearn — Automated Feature Generation and Selection
Kaul, A., Maheshwary, S., & Pudi, V. (2017) · Proceedings · doi:10.1109/icdm.2017.31 · Cited by 123
ATM: A distributed, collaborative, scalable system for automated machine learning
Swearingen, T., Drevo, W., Cyphers, B., Cuesta‐Infante, A., Ross, A. A., & Veeramachaneni, K. K. (2017) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2017.8257923 · Cited by 120
Self-citationElskenThomas, Metzen, J. H., & HutterFrank (2019) · Journal of Machine Learning Research · doi:10.5555/3322706.3361996 · Cited by 105
Rafiki: machine learning as an analytics service system
Wang, W., Gao, J., Zhang, M., Wang, S., Chen, G., Ng, T. K., Ooi, B. C., Shao, J., & Reyad, M. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3282495.3282499 · Cited by 102
The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development
Smith, M. J., Sala, C., Kanter, J. M., & Veeramachaneni, K. K. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3386146 · Cited by 55
Self-citationChu, X., Morcos, J., Ilyas, I. F., et al. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824109 · Cited by 47
The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development
Smith, M. J., Sala, C., Kanter, J. M., & Veeramachaneni, K. K. (2020) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 46
Self-citationFeature selection and feature learning for high-dimensional batch reinforcement learning: A survey
Liu, D., Li, H., & Wang, D. (2015) · Machine Intelligence Research · doi:10.1007/s11633-015-0893-y · Cited by 45
Active Learning for Ranking through Expected Loss Optimization
Long, B., Bian, J., Chapelle, O., Zhang, Y., Inagaki, Y., & Chang, Y. (2014) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2365785 · Cited by 42
Haas, D., Krishnan, S., Wang, J., Franklin, M. J. M., & Wu, E. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824122 · Cited by 39
Label, Segment, Featurize: A Cross Domain Framework for Prediction Engineering
Kanter, J. M., Gillespie, O., & Veeramachaneni, K. K. (2016) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2016.54 · Cited by 22
Self-citationClearView: data cleaning for online review mining
Minnich, A., Abu-el-rub, N., Gokhale, M. B., Minnich, R. G., & Mueen, A. (2016) · doi:10.5555/3192424.3192528 · Cited by 3
Cited by
Advances and Challenges in Meta-Learning: A Technical Review
Vettoruzzo, A., Bouguelia, M.-R., Vanschoren, J., Rögnvaldsson, T., & Santosh, K. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3357847 · Cited by 314
A review of the F-measure: its history, properties, criticism, and alternatives
Christen, P., Hand, D. J., & Kirielle, N. (2024) · ACM Computing Surveys · doi:10.1145/3606367 · Cited by 236
Artificial Intelligence and Biosensors in Healthcare and Its Clinical Relevance: A Review
(2023) · doi:10.1109/access.2023.3285596 · Cited by 205
Metadata
- Authors
- Karmaker (“Santu”), S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., & Veeramachaneni, K.
- Venue
- ACM Computing Surveys
- Published
- 2021-10-04
- Volume
- 54
- Issue
- 8
- Pages
- 1-36
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3201904098
- Cited by
- 292
- References
- 36
- FWCI
- 14.73
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.