Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 1,447

The DOI matches its Crossref record; 104 of the 104 references it cites were found in a registry or an index.

Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs

Ding, Xiaohan, Zhang, Xiangyu, Han, Jungong, Ding, Guiguang·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Abstract

We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depthwise convolutions, to design efficient high-performance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31×31, in contrast to commonly used 3×3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias. Code & models at https://github.com/megvii-research/RepLKNet.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.01166

Cites

104 of 104 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 23

  1. dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ArXiv Preprint
  2. geirhos, 2018, Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, ArXiv Preprint
  3. dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale, 9th International Conference on Learning Representations ICLR 2021
  4. cordonnier, 2019, On the relationship between self-attention and convolutional layers, ArXiv Preprint
  5. yu, 2015, Multiscale context aggregation by dilated convolutions, ArXiv Preprint
  6. liu, 2018, Darts: Differentiable architecture search, ArXiv Preprint
  7. ba, 2016, Layer normalization, ArXiv Preprint
  8. brendel, 2019, Approximating cnns with bag-of-local-features models works surprisingly well on imagenet, ArXiv Preprint
  9. howard, 2017, Mobilenets: Efficient convolutional neural networks for mobile vision applications, ArXiv Preprint
  10. rao, 2021, Global filter networks for image classification, ArXiv Preprint
  11. ramachandran, 2019, Standalone self-attention in vision models, ArXiv Preprint
  12. 2022, Official pytorch implementation of replknet
  13. raghu, 2021, Do vision transformers see like convolutional neural networks, ArXiv Preprint
  14. 2020, Megengine a fast scalable and easy-to-use deep learning framewok
  15. wu, 2019, Pay less attention with lightweight and dynamic convolutions, ArXiv Preprint
  16. wang, 0, Axial-deeplab: Standalone axial-attention for panoptic segmentation, European Conference on Computer Vision, 108
  17. vaswani, 0, Attention is all you need, Advances in neural information processing systems, 5998
  18. yu, 0, Di-lated residual networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 472
  19. chen, 0, Pretrained image processing transformer, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12299
  20. chen, 2021, An empirical study of training self-supervised visual transformers, ArXiv e-prints
  21. 2020, MMSegmentation Contributors MMSegmentation Openmmlab semantic segmentation toolbox and benchmark
  22. touvron, 0, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning, 10347
  23. trockman, 2022, Patches are all you need, ArXiv Preprint

Cited by

See all 1,447

Metadata

Authors
Ding, X., Zhang, X., Han, J., & Ding, G.
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
11953-11965
ISSN
1063-6919 (ISSN-L)
OpenAlex
W4226334005
Cited by
1,447
References
104
FWCI
115.12
Type
conference paper
Language
en
Open access
closed
Funders
National Natural Science Foundation of China (61925107,U1936202,62021002)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.