Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Open accessThe DOI matches its OpenAlex record; 12 of the 12 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 4×
“the re1MIT CSAIL 2CUNY Graduate Center, Initiative for the Theoretical Sciences 3Facebook AI Research. Correspondence to: Jonathan Frankle, Ari S. Morcos <jfrankle@mit.edu, arimorcos@fb.com>. 1He et al. (2016) find that performance is better when BatchNorm is applied before activation rather than after in ResNets…”
BackgroundMethodInfluentialArbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076
Maass, W., Natschläger, T., & Markram, H. (2002) · Neural Computation · doi:10.1162/089976602760407955 · Cited by 4,025 · cited 3×
“Although prior work considers models with random features and a train-able affine output layer (e.g., Rahimi & Recht, 2009; Jaeger, 2003; Maass et al., 2002), we reveal the expressive power of networks configured such that trainable affine parameters appear after each layer of random features.”
BackgroundFilm: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
Rethinking the Value of Network Pruning
Liu, Z., Sun, M., Zhou, T., Huang, G., & Darrell, T. (2018) · arXiv · doi:10.48550/arxiv.1810.05270 · Cited by 1,625
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562
Cited by
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765
Normalization Techniques in Training DNNs: Methodology, Analysis and Application
Huang, L., Qin, J., Zhou, Y., Zhu, F., Liu, L., & Shao, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3250241 · Cited by 537
Wortsman, M., Ramanujan, V., Liu, R., Kembhavi, A., Rastegari, M., Yosinski, J., & Farhadi, A. (2020) · arXiv · doi:10.48550/arxiv.2006.14769 · Cited by 144
“[9] Jonathan Frankle, David J Schwab, and Ari S Morcos.”
BackgroundMethodFrozen Pretrained Transformers as Universal Computation Engines
Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20729 · Cited by 113
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben-Zaken, E., Ravfogel, S., & Goldberg, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.10199 · Cited by 11
“Michel and Neubig (2018) fine-tuned the biases of the output softmax in an NMT systems, to personalize the output vocabulary, and Frankle et al. (2020) have demonstrated that randomly-initialized CNNs achieve reasonable accuracy after training the batch-norm layers alone.”
BackgroundImproving robustness against common corruptions by covariate shift adaptation
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., & Bethge, M. (2020) · arXiv · doi:10.48550/arxiv.2006.16971
Metadata
- Authors
- Frankle, J., Schwab, D. J., & Morcos, A. S.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2020-02-29
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.00152
- OpenAlex
- W3007700590
- Cited by
- 74
- References
- 12
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.