NAS-bench-101: Towards reproducible neural architecture search
Open accessAbstract
Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several NAS benchmarks [259, 260] have been developed, which act as a lookup-table to obtain validation accuracy without actually training them.
“In this context, several NAS benchmarks [259, 260] have been developed, which act as a lookup-table to obtain the validation accuracy without actually training them.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [260]
Cited by
Sorted by citations
The Values Encoded in Machine Learning Research
Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bao, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533083 · Cited by 317 · see citing works
Symbolic Discovery of Optimization Algorithms
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.06675 · Cited by 303 · see citing works
“Evolution with warm-start and restart We apply regularized evolution as it is simple, scalable, and has shown success on many AutoML search tasks (Holland, 1992; Real et al., 2019, 2020; So et al., 2019; Ying et al., 2019).”
MethodSelf-citationA Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks
Seshadri, K., Akin, B., Laudon, J., Narayanaswami, R., & Yazdanbakhsh, A. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00017 · Cited by 138 · see citing works
- Authors
- Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., & Hutter, F.
- Affiliations
- University of Freiburg (DE)
- Google (United States) (US)
- Washington State University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 7105-7114
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1902.09635
- OpenAlex
- W2916118939
- Cited by
- 297
- References
- 0
- Citations by year
- 2026: 1 · 2025: 12 · 2024: 19 · 2023: 38 · 2022: 36 · 2021: 111 · 2020: 65 · 2019: 13 · 2018: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- neural architecture search, NAS-Bench-101, reproducibility, graph isomorphism, convolutional neural network architecture, benchmarking, CIFAR-10
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
