Fixing the train-test resolution discrepancy: FixEfficientNet
Open accessAbstract
This paper provides an extensive analysis of the performance of the EfficientNet image classifiers with several recent training procedures, in particular one that corrects the discrepancy between train and test images. The resulting network, called FixEfficientNet, significantly outperforms the initial architecture with the same number of parameters. For instance, our FixEfficientNet-B0 trained without additional training data achieves 79.3% top-1 accuracy on ImageNet with 5.3M parameters. This is a +0.5% absolute improvement over the Noisy student EfficientNet-B0 trained with 300M unlabeled images. An EfficientNet-L2 pre-trained with weak supervision on 300M unlabeled images and further optimized with FixRes achieves 88.5% top-1 accuracy (top-5: 98.7%), which establishes the new state of the art for ImageNet with a single crop. These improvements are thoroughly evaluated with cleaner protocols than the one usually employed for Imagenet, and particular we show that our improvement remains in the experimental setting of ImageNet-v2, that is less prone to overfitting, and with ImageNet Real Labels. In both cases we also establish the new state of the art.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Convolutional neural networks have been optimized in architecture and optimization for almost a decade, including through extensive architecture search prone to overfitting, as for EfficientNets.
“Convolutional neural networks have optimized, both in terms of architecture and optimization during almost a decade, including through extensive architecture search that is prone to overfiting, as it is the case for instance for EfficientNets [51].”
From Training data-efficient image transformers & distillation through attention (2020) · cites this paper as [51]
Section 3.3 (p. 3) says EfficientNets use neural architecture search, which enlarges the hyperparameter space, and that the pretrained models may benefit from greater overfitting to the validation set. Section 3.4 (p. 3) finds that EfficientNet models trained with Noisy Student tend to overfit. The paper does not establish that architecture search itself causes that overfitting, or substantiate the claim that CNN architecture and optimization have been improving for almost a decade. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 16 · Cited by 88
Research works printed but not linked: 14
- [1][1] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and H´ erve J´ egou, 'Fixing the train-test resolution discrepancy,' Advances in Neural Information Processing Systems , 2019. -
- [2][2] Ismet Zeki Yalniz, Herv´ e J´ egou, Kan Chen, Manohar Paluri, and Dhruv Kumar Mahajan, 'Billion-scale semisupervised learning for image classification,' arXiv preprint arXiv:1905.00546 , 2019. - · arXiv
- [3][3] Ekin Dogus Cubuk, Barret Zoph, Dandelion Man´ e, Vijay Vasudevan, and Quoc V. Le, 'Autoaugment: Learning augmentation policies from data,' arXiv preprint arXiv:1805.09501 , 2018. - · arXiv
- [5][5] Lucas Beyer, Olivier J. H´ enaff, A. Kolesnikov, Xiaohua Zhai, and Aaron van den Oord, 'Are we done with ImageNet?,' arXiv preprint arXiv:2006.07159 , 2020. - · arXiv
- [6][6] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen, 'Gpipe: Efficient training of giant neural networks using pipeline parallelism,' arXiv preprint arXiv:1811.06965 , 2018. - · arXiv
- [7][7] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten, 'Exploring the limits of weakly supervised pretraining,' in European Conference on Computer Vision , 2018. -
- [8][8] Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le, 'Self-training with noisy student improves imagenet classification,' arXiv preprint arXiv:1911.04252 , 2019. - · arXiv
- [9][9] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoc V. Le, 'Adversarial examples improve image recognition,' arXiv preprint arXiv:1911.09665 , 2019. - · arXiv
- [10][10] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei, 'Imagenet large scale visual recognition challenge,' International journal of Computer Vision , 2015. -
- [12][12] Barret Zoph, V. Vasudevan, Jonathon Shlens, and Quoc V. Le, 'Learning transferable architectures for scalable image recognition,' Conference on Computer Vision and Pattern Recognition , 2018. -
- [13][13] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy, 'Progressive neural architecture search,' in International Conference on Computer Vision , September 2018. -
- [14][14] Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Doll´ ar, 'Designing network design spaces,' arXiv preprint arXiv:2003.13678 , 2020. - · arXiv
- [15][15] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le, 'Randaugment: Practical automated data augmentation with a reduced search space,' arXiv preprint arXiv:1909.13719 , 2019. - · arXiv
- [17][17] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, 'Do imagenet classifiers generalize to imagenet?,' in International Conference on Machine Learning , 2019.
- Authors
- Touvron, H., Vedaldi, A., Douze, M., & Jégou, H.
- Affiliations
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.08237
- OpenAlex
- W3012324658
- Cited by
- 88
- References
- 16
- Citations by year
- 2024: 2 · 2023: 5 · 2022: 19 · 2021: 37 · 2020: 24 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- EfficientNet, unlabeled images, ImageNet, image classification, overfitting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
