Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 1,447

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.

Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs

Ding, Xiaohan, Zhang, Xiangyu, Han, Jungong, Ding, Guiguang·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52688.2022.01166
Open this paper in CoChat — free account

Abstract

We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depthwise convolutions, to design efficient high-performance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31×31, in contrast to commonly used 3×3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias. Code & models at https://github.com/megvii-research/RepLKNet.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.01166

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Research works printed but not linked: 77

  1. 10.1109/ICCV.2017.322 · DOI · Paper page
  2. han, 2021, Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight, ArXiv Preprint
  3. dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ArXiv Preprint
  4. dong, 2021, Cswin transformer: A general vision transformer backbone with cross-shaped windows, ArXiv Preprint
  5. 10.1109/CVPR46437.2021.00098 · DOI
  6. 10.1109/CVPR46437.2021.01352 · DOI · Paper page
  7. guo, 0, Single path one-shot neural architecture search with uniform sampling, European Conference on Computer Vision, 544
  8. gong, 2021, Improve vision transformers training by suppressing over-smoothing, ArXiv Preprint
  9. geirhos, 2018, Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, ArXiv Preprint
  10. dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale, 9th International Conference on Learning Representations ICLR 2021
  11. 10.1109/ICCV48922.2021.00447 · DOI · Paper page
  12. 10.1109/ICCV.2019.00200 · DOI · Paper page
  13. 10.1109/CVPR46437.2021.01074 · DOI · Paper page
  14. cordonnier, 2019, On the relationship between self-attention and convolutional layers, ArXiv Preprint
  15. 10.1109/CVPRW50498.2020.00359 · DOI · Paper page
  16. 10.1109/CVPR46437.2021.00729 · DOI · Paper page
  17. yu, 2015, Multiscale context aggregation by dilated convolutions, ArXiv Preprint
  18. ding, 2021, Repmlpnet: Hierarchical vision mlp with re-parameterized locality, ArXiv Preprint
  19. xie, 2021, Self-supervised learning with swin transformers, ArXiv Preprint
  20. de, 2020, Batch normalization biases residual blocks towards the identity function in deep networks, ArXiv Preprint
  21. kayhan, 0, On translation invariance in cnns: Convolutional layers can exploit absolute spatial location, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14274
  22. 10.18653/v1/2020.emnlp-main.463 · DOI · Paper page
  23. liu, 2018, Darts: Differentiable architecture search, ArXiv Preprint
  24. lin, 0, Microsoft coco: Common objects in context, European Conference on Computer Vision, 740
  25. 10.1109/ICCVW54120.2021.00210 · DOI · Paper page
  26. kim, 2021, Dead pixel test using effective receptive field, ArXiv Preprint
  27. bao, 2021, Beit: Bert pre-training of image transformers, ArXiv Preprint
  28. ba, 2016, Layer normalization, ArXiv Preprint
  29. 10.1109/ICCV.2019.00338 · DOI · Paper page
  30. cai, 2019, Cascade r-cnn: High quality object detection and instance segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence
  31. brendel, 2019, Approximating cnns with bag-of-local-features models works surprisingly well on imagenet, ArXiv Preprint
  32. 10.1109/ICCV.2019.00356 · DOI · Paper page
  33. huang, 2016, Deep networks with stochastic depth, Computer Vision - ECCV 2016 – 14th European Conference, 646, 10.1007/978-3-319-46493-0_39 · DOI · Paper page
  34. hinton, 2021, How to represent part-whole hierarchies in a neural network, ArXiv Preprint
  35. hendrycks, 2016, Gaussian error linear units (gelus), ArXiv Preprint
  36. howard, 2017, Mobilenets: Efficient convolutional neural networks for mobile vision applications, ArXiv Preprint
  37. romero, 2021, Flexconv: Continuous kernel convolutions with differentiable kernel sizes, ArXiv Preprint
  38. rao, 2021, Global filter networks for image classification, ArXiv Preprint
  39. ramachandran, 2019, Standalone self-attention in vision models, ArXiv Preprint
  40. simonyan, 2014, Very deep convo-lutional networks for large-scale image recognition, ArXiv Preprint
  41. romero, 2021, Ckconv: Continuous kernel convolution for sequential data, ArXiv Preprint
  42. 10.1109/CVPR.2018.00474 · DOI · Paper page
  43. 10.1109/CVPR46437.2021.01625 · DOI · Paper page
  44. 10.1609/aaai.v31i1.11231 · DOI · Paper page
  45. 10.1109/CVPR52688.2022.01167 · DOI · Paper page
  46. loshchilov, 2017, Decoupled weight decay regularization, ArXiv Preprint
  47. mathieu, 2014, Fast training of convolutional networks through ffts, 2nd International Conference on Learning Representations ICLR 2014
  48. 2022, Official pytorch implementation of replknet
  49. raghu, 2021, Do vision transformers see like convolutional neural networks, ArXiv Preprint
  50. 2020, Megengine a fast scalable and easy-to-use deep learning framewok
  51. 10.1109/CVPR.2018.00716 · DOI · Paper page
  52. wu, 2019, Pay less attention with lightweight and dynamic convolutions, ArXiv Preprint
  53. 10.1109/CVPRW53098.2021.00344 · DOI
  54. zhang, 2020, Resnest: Split-attention networks, ArXiv Preprint
  55. zhang, 2017, mixup: Beyond empirical risk minimization, ArXiv Preprint
  56. wang, 0, Axial-deeplab: Standalone axial-attention for panoptic segmentation, European Conference on Computer Vision, 108
  57. 10.1109/ICCV.2019.00612 · DOI · Paper page
  58. veit, 0, Residual networks behave like ensembles of relatively shallow networks, Advances in neural information processing systems, 550
  59. yuan, 2021, Volo: Vision outlooker for visual recog-nition, ArXiv Preprint
  60. vaswani, 0, Attention is all you need, Advances in neural information processing systems, 5998
  61. yu, 2021, Metaformer is actually what you need for vision, ArXiv Preprint
  62. yu, 0, Di-lated residual networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 472
  63. zhao, 2021, A battle of network structures: An empirical study of cnn, transformer, and mlp, ArXiv Preprint
  64. zhang, 2021, Collaboration of experts: Achieving 80% top-1 accuracy on imagenet with 100m flops, ArXiv Preprint
  65. xie, 2021, Segformer: Simple and ef-ficient design for semantic segmentation with transformers, ArXiv Preprint
  66. 10.1109/CVPR.2017.634 · DOI · Paper page
  67. 10.1109/ICCV48922.2021.00009 · DOI · Paper page
  68. chen, 0, Pretrained image processing transformer, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12299
  69. 10.29007/3b2l · DOI · Paper page
  70. chen, 2017, Rethinking atrous convolution for semantic image segmentation, ArXiv Preprint
  71. chen, 2021, An empirical study of training self-supervised visual transformers, ArXiv e-prints
  72. 2020, MMSegmentation Contributors MMSegmentation Openmmlab semantic segmentation toolbox and benchmark
  73. 10.1109/CVPR.2015.7298594 · DOI · Paper page
  74. 10.1109/ICCV.2019.00679 · DOI · Paper page
  75. 10.1109/CVPR46437.2021.01270 · DOI · Paper page
  76. touvron, 0, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning, 10347
  77. trockman, 2022, Patches are all you need, ArXiv Preprint

Cited by

See all 1,447

Metadata

Authors
Ding, X., Zhang, X., Han, J., & Ding, G.
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
11953-11965
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52688.2022.01166
OpenAlex
W4226334005
Cited by
1,447
References
44
FWCI
115.12
Type
conference paper
Language
en
Open access
closed
Licence
Funders
National Natural Science Foundation of China (61925107,U1936202,62021002)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs. https://app.cochat.ai/papers/10.1109/cvpr52688.2022.01166
The paper
Ding, X., Zhang, X., Han, J., & Ding, G. (2022). Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52688.2022.01166

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.