The DOI matches its Crossref record; 108 of the 108 references it cites were found in a registry or an index.
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs
Abstract
We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depthwise convolutions, to design efficient high-performance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31×31, in contrast to commonly used 3×3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias. Code & models at https://github.com/megvii-research/RepLKNet.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Self-citationVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Research works printed but not linked: 22
- dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ArXiv Preprint
- geirhos, 2018, Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, ArXiv Preprint
- dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale, 9th International Conference on Learning Representations ICLR 2021
- cordonnier, 2019, On the relationship between self-attention and convolutional layers, ArXiv Preprint
- yu, 2015, Multiscale context aggregation by dilated convolutions, ArXiv Preprint
- liu, 2018, Darts: Differentiable architecture search, ArXiv Preprint
- ba, 2016, Layer normalization, ArXiv Preprint
- brendel, 2019, Approximating cnns with bag-of-local-features models works surprisingly well on imagenet, ArXiv Preprint
- rao, 2021, Global filter networks for image classification, ArXiv Preprint
- ramachandran, 2019, Standalone self-attention in vision models, ArXiv Preprint
- 2022, Official pytorch implementation of replknet
- raghu, 2021, Do vision transformers see like convolutional neural networks, ArXiv Preprint
- 2020, Megengine a fast scalable and easy-to-use deep learning framewok
- wu, 2019, Pay less attention with lightweight and dynamic convolutions, ArXiv Preprint
- wang, 0, Axial-deeplab: Standalone axial-attention for panoptic segmentation, European Conference on Computer Vision, 108
- vaswani, 0, Attention is all you need, Advances in neural information processing systems, 5998
- yu, 0, Di-lated residual networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 472
- chen, 0, Pretrained image processing transformer, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12299
- chen, 2021, An empirical study of training self-supervised visual transformers, ArXiv e-prints
- 2020, MMSegmentation Contributors MMSegmentation Openmmlab semantic segmentation toolbox and benchmark
- touvron, 0, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning, 10347
- trockman, 2022, Patches are all you need, ArXiv Preprint
Cited by
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00721 · Cited by 13,198
“Model re-parametrization techniques [71, 31, 75, 19, 33, 11, 4, 24, 13, 12, 10, 29, 14, 78] merge multiple computational modules into one at inference stage.”
MethodYOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2022) · arXiv · doi:10.48550/arxiv.2207.02696 · Cited by 13,205
“Model re-parametrization techniques [71, 31, 75, 19, 33, 11, 4, 24, 13, 12, 10, 29, 14, 78] merge multiple computational modules into one at inference stage.”
MethodHussain, M. (2023) · Machines · doi:10.3390/machines11070677 · Cited by 1,217
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080
Guo, M.-H., Lu, C.-Z., Liu, Z.-N., Cheng, M.-M., & Hu, S.-M. (2023) · Computational Visual Media · doi:10.1007/s41095-023-0364-2 · Cited by 1,073
Large Selective Kernel Network for Remote Sensing Object Detection
Li, Y., Hou, Q., Zheng, Z., Cheng, M.-M., Yang, J., & Li, X. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01540 · Cited by 893
Rep ViT: Revisiting Mobile CNN From ViT Perspective
Wang, A., Chen, H., Lin, Z., Han, J., & Ding, G. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01506 · Cited by 752
Self-citationMambaIR: A Simple Baseline for Image Restoration with State-Space Model
Guo, H., Li, J., Dai, T., Ouyang, Z., Ren, X., & Xia, S.-T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72649-1_13 · Cited by 642
Large Separable Kernel Attention: Rethinking the Large Kernel Attention design in CNN
Lau, K. W., Po, L.-M., & Rehman, Y. A. U. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121352 · Cited by 569
Wavelet Convolutions for Large Receptive Fields
Finder, S. E., Amoyal, R., Treister, E., & Freifeld, O. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72949-2_21 · Cited by 514
InceptionNeXt: When Inception Meets ConvNeXt
Yu, W., Zhou, P., Yan, S., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00542 · Cited by 423
TransNeXt: Robust Foveal Visual Perception for Vision Transformers
Shi, D. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01683 · Cited by 423
TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution
Xiao, Y., Yuan, Q., Jiang, K., He, J., Lin, C.-W., & Zhang, L. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3349004 · Cited by 413
Li, J., Chen, J., Tang, Y., Wang, C., Landman, B. A., & Zhou, S. K. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102762 · Cited by 389
EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction
Cai, H., Li, J., Hu, M., Gan, C., & Han, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01587 · Cited by 356
Drone-YOLO: An Efficient Neural Network Method for Target Detection in Drone Images
Zhang, Z. (2023) · Drones · doi:10.3390/drones7080526 · Cited by 306
Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution
Sun, L., Dong, J., Tang, J., & Pan, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01213 · Cited by 289
Omni-Kernel Network for Image Restoration
Cui, Y., Ren, W., & Knoll, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27907 · Cited by 279
Overview of Deep Learning-Based CSI Feedback in Massive MIMO Systems
Guo, J., Wen, C.-K., Jin, S., & Li, G. Y. (2022) · IEEE Transactions on Communications · doi:10.1109/tcomm.2022.3217777 · Cited by 278
NTIRE 2023 Challenge on Efficient Super-Resolution: Methods and Results
Li, Y., Zhang, Y., Timofte, R., Van Gool, L., Yu, L., Li, Y., Li, X., Jiang, T., Wu, Q., Han, M., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00189 · Cited by 72
Self-citationMake RepVGG Greater Again: A Quantization-aware Approach
Chu, X., Li, L., & Zhang, B. (2022) · arXiv · doi:10.48550/arxiv.2212.01593 · Cited by 53
Make RepVGG Greater Again: A Quantization-Aware Approach
Chu, X., Li, L., & Zhang, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29045 · Cited by 53
Tang, J., Han, J., Xie, B., Xue, J., Zhou, H., Jiang, Y., Hu, L., Chen, C., Zhang, K., Zhu, F., et al. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20032377 · Cited by 13
Self-citation
Metadata
- Authors
- Ding, X., Zhang, X., Han, J., & Ding, G.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 11953-11965
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4226334005
- Cited by
- 1,447
- References
- 108
- FWCI
- 115.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- National Natural Science Foundation of China (61925107,U1936202,62021002)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.