Neural Architecture Search Benchmarks: Insights and Survey
Open accessAbstract
Neural Architecture Search (NAS), a promising and fast-moving research field, aims to automate the architectural design of Deep Neural Networks (DNNs) to achieve better performance on the given task and dataset. NAS methods have been very successful in discovering efficient models for various Computer Vision, Natural Language Processing, etc. The major obstacles to the advancement of NAS techniques are the demand for large computation resources and fair evaluation of various search methods. The differences in training pipeline and setting make it challenging to compare the efficiency of two NAS algorithms. A large number of NAS Benchmarks to simulate the architecture evaluation in seconds have been released over the last few years to ease the computation burden of training neural networks and can aid in the unbiased assessment of different search methods. This paper provides and extensive review of several publicly available NAS Benchmarks in the literature. We provide technical details and a deeper understanding of each benchmark and point out future directions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several NAS benchmarks [259, 260] have been developed, which act as a lookup-table to obtain validation accuracy without actually training them.
“In this context, several NAS benchmarks [259, 260] have been developed, which act as a lookup-table to obtain the validation accuracy without actually training them.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [259]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · arXiv · doi:10.48550/arxiv.1604.01685 · Cited by 1,174 · see citing works
Elsken, T., Metzen, J. H., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_3 · Cited by 785 · see citing works
AutoFormer: Searching Transformers for Visual Recognition
Chen, M., Peng, H., Fu, J., & Ling, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01205 · Cited by 323 · see citing works
NAS-FCOS: Fast Neural Architecture Search for Object Detection
Wang, N., Gao, Y., Chen, H., Wang, P., Tian, Z., Shen, C., & Zhang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01196 · Cited by 258 · see citing works
Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.01500 · Cited by 206 · cited 1× · see citing works
“If possible, a consortium can be established for NAS Bench-marks in the future, similar to the MLperf community [149], [150], [151], [152], where all the code and benchmark datasets should be stored for quick access to researchers.”
BackgroundNeural Architecture Search Survey: A Hardware Perspective
Chitty-Venkata, K. T., & Somani, A. K. (2023) · ACM Computing Surveys · doi:10.1145/3524500 · Cited by 106 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
Self-citation
- Authors
- Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K.
- Affiliations
- Iowa State University (US)
- Argonne National Laboratory (US)
- Venue
- IEEE Access
- Published
- 2023
- Volume
- 11
- Pages
- 25217-25236
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4323644307
- Cited by
- 64
- References
- 10
- Citations by year
- 2026: 15 · 2025: 26 · 2024: 18 · 2023: 5
- FWCI
- 5.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- IEEE Access (publishedVersion, cc-by)
- OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion, public-domain)
- Iowa State University Digital Repository (Iowa State University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Philip and Virginia Sproul Professorship;
- National Science Foundation at Iowa State University (MRI 1726447, MRI 2018594)
- Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility (DEAC02-06CH11357)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- neural architecture search, NAS benchmarks, benchmark comparison, training cost reduction, benchmark evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
