The DOI matches its Crossref record; 104 of the 104 references it cites were found in a registry or an index.
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs
Abstract
We revisit large kernel design in modern convolutional neural networks (CNNs). Inspired by recent advances in vision transformers (ViTs), in this paper, we demonstrate that using a few large convolutional kernels instead of a stack of small kernels could be a more powerful paradigm. We suggested five guidelines, e.g., applying re-parameterized large depthwise convolutions, to design efficient high-performance large-kernel CNNs. Following the guidelines, we propose RepLKNet, a pure CNN architecture whose kernel size is as large as 31×31, in contrast to commonly used 3×3. RepLKNet greatly closes the performance gap between CNNs and ViTs, e.g., achieving comparable or superior results than Swin Transformer on ImageNet and a few typical downstream tasks, with lower latency. RepLKNet also shows nice scalability to big data and large models, obtaining 87.8% top-1 accuracy on ImageNet and 56.0% mIoU on ADE20K, which is very competitive among the state-of-the-arts with similar model sizes. Our study further reveals that, in contrast to small-kernel CNNs, large-kernel CNNs have much larger effective receptive fields and higher shape bias rather than texture bias. Code & models at https://github.com/megvii-research/RepLKNet.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Self-citationVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
Self-citationHoward, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
Deformable Convolutional Networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.89 · Cited by 7,513
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_8 · Cited by 7,356 · cited 1×
“It is true for conventional DW 3 × 3 kernels [46, 77, 114], because DW operations introduce low ratio of computation vs . memory access cost [66], which is not friendly to modern computing architecture.”
BackgroundSelf-citationFCOS: Fully Convolutional One-Stage Object Detection
Tian, Z., Shen, C., Chen, H., & He, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00972 · Cited by 7,013
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“been greatly challenged by Vision Transformers (ViTs) [34, 59, 86, 94], which have shown leading performances on many visual tasks – not only image classification [34, 104] and representation learning [4, 9, 16, 100], but also many downstream tasks such as object detection [24, 59], semantic segmentation [94, 98] and…”
BackgroundCutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 2×
“been greatly challenged by Vision Transformers (ViTs) [34, 59, 86, 94], which have shown leading performances on many visual tasks – not only image classification [34, 104] and representation learning [4, 9, 16, 100], but also many downstream tasks such as object detection [24, 59], semantic segmentation [94, 98] and…”
BackgroundSegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
Deep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894
SwinIR: Image Restoration Using Swin Transformer
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., & Timofte, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw54120.2021.00210 · Cited by 4,734
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223
RepVGG: Making VGG-style ConvNets Great Again
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., & Sun, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01352 · Cited by 2,907
Self-citationSwin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01170 · Cited by 2,484
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,444
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373
Pre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274
MMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227
Unified Perceptual Parsing for Scene Understanding
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., & Sun, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_26 · Cited by 2,201 · cited 2×
“Specifi-cally, we use the UperNet [102] implemented by MMSeg-mentation [20] with the 80K-iteration training schedule for Cityscapes and 160K for ADE20K.”
MethodDeep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
MLP-Mixer: An all-MLP Architecture for Vision
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01601 · Cited by 2,087
Designing Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Cascade R-CNN: High Quality Object Detection and Instance Segmentation
Cai, Z., & Vasconcelos, N. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2956516 · Cited by 1,935
Semantic Understanding of Scenes Through the ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., & Torralba, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1140-0 · Cited by 1,831
Yu, F., Koltun, V., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.75 · Cited by 1,804
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network
Peng, C., Zhang, X., Yu, G., Luo, G., & Sun, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.189 · Cited by 1,750
Self-citationResNeSt: Split-Attention Networks
Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00309 · Cited by 1,648
Hybrid Task Cascade for Instance Segmentation
Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00511 · Cited by 1,587
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
Luo, W., Li, Y., Urtasun, R., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1701.04128 · Cited by 1,575
BEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
Bottleneck Transformers for Visual Recognition
Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., & Vaswani, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01625 · Cited by 1,385
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01181 · Cited by 1,358
MetaFormer is Actually What You Need for Vision
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01055 · Cited by 1,255
Attention Augmented Convolutional Networks
Bello, I., Zoph, B., Le, Q., Vaswani, A., & Shlens, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00338 · Cited by 1,205
Dynamic Head: Unifying Object Detection Heads with Attentions
Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00729 · Cited by 1,044
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977
ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
Ding, X., Guo, Y., Ding, G., & Han, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00200 · Cited by 912
Self-citationSingle Path One-Shot Neural Architecture Search with Uniform Sampling
Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., & Sun, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_32 · Cited by 813
Self-citationAxial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., & Chen, L.-C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_7 · Cited by 809
ResMLP: Feedforward networks for image classification with data-efficient training
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.03404 · Cited by 807
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Izacard, G., Joulin, A., Synnaeve, G., Verbeek, J., et al. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206148 · Cited by 807
Residual Networks Behave Like Ensembles of Relatively Shallow Networks
Veit, A., Wilber, M., & Belongie, S. (2016) · arXiv · doi:10.48550/arxiv.1605.06431 · Cited by 705
An Empirical Study of Spatial Attention Mechanisms in Deep Networks
Zhu, X., Cheng, D., Zhang, Z., Lin, S., & Dai, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00679 · Cited by 659
Local Relation Networks for Image Recognition
Hu, H., Zhang, Z., Xie, Z., & Lin, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00356 · Cited by 570
Diverse Branch Block: Building a Convolution as an Inception-like Unit
Ding, X., Zhang, X., Han, J., & Ding, G. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01074 · Cited by 520
Self-citationConditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489
Scaling Local Self-Attention for Parameter Efficient Visual Backbones
Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., & Shlens, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01270 · Cited by 453
Fast Training of Convolutional Networks through FFTs
Mathieu, M., Henaff, M., & LeCun, Y. (2014) · International Conference on Learning Representations · Cited by 347
VOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 288
Vision Transformers Are Robust Learners
Paul, S., & Chen, P.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20103 · Cited by 244 · cited 1×
“They provided empirical results to demonstrate that, MHSA is more flexible [50], capable (less inductive bias) [20], more robust to distortions [66, 98], or able to model long-range dependencies [69, 90].”
BackgroundSelf-Supervised Learning with Swin TransformersPreprint
Xie, Z., Lin, Y., Yao, Z., Zhang, Z., Dai, Q., Cao, Y., & Hu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04553 · Cited by 233
Efficient Cipher-Only Attack of a Stream Cipher in RDHEI Based on Pixel Smoothness Estimation
Chen, Y., Li, Z., Liu, B., Zhang, J., Yin, B., & He, H. (2019) · EasyChair Preprints · doi:10.29007/3b2l · Cited by 232
How to Represent Part-Whole Hierarchies in a Neural Network
Hinton, G. E. (2022) · Neural Computation · doi:10.1162/neco_a_01557 · Cited by 213
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location
Semih Kayhan, O., & van Gemert, J. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01428 · Cited by 193
ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting
Ding, X., Hao, T., Tan, J., Liu, J., Han, J., Guo, Y., & Ding, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00447 · Cited by 177
Self-citationUnderstanding the Difficulty of Training Transformers
Liu, L., Liu, X., Gao, J., Chen, W., & Han, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.463 · Cited by 170
Self-citationA Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLPPreprint
Zhao, Y., Wang, G., Tang, C., Luo, C., Zeng, W., & Zha, Z.-J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13002 · Cited by 125
Fast and Accurate Model Scaling
Dollár, P., Singh, M., & Girshick, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00098 · Cited by 122
Are Convolutional Neural Networks or Transformers more like human vision?
Tuli, S., Dasgupta, I., Grant, E., & Griffiths, T. L. (2021) · arXiv · doi:10.48550/arxiv.2105.07197 · Cited by 119
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep NetworksPreprint
De, S., & Smith, S. L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10444 · Cited by 107
RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality
Ding, X., Chen, H., Zhang, X., Han, J., & Ding, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00066 · Cited by 91
Self-citationFlexConv: Continuous Kernel Convolutions with Differentiable Kernel SizesPreprint
Romero, D. W., Bruintjes, R.-J., Tomczak, J. M., Bekkers, E. J., Hoogendoorn, M., & van Gemert, J. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08059 · Cited by 72
Vision Transformers with Patch Diversification
Gong, C., Wang, D., Li, M., Chandra, V., & Liu, Q. (2021) · arXiv · doi:10.48550/arxiv.2104.12753 · Cited by 41
CKConv: Continuous Kernel Convolution For Sequential Data
Romero, D. W., Kuzina, A., Bekkers, E. J., Tomczak, J. M., & Hoogendoorn, M. (2021) · arXiv · doi:10.48550/arxiv.2102.02611 · Cited by 39
Dead pixel test using effective receptive field
Kim, B. J., Choi, H., Jang, H., Lee, D. G., Jeong, W., & Kim, S. W. (2023) · Pattern Recognition Letters · doi:10.1016/j.patrec.2023.02.018 · Cited by 36
Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic WeightPreprint
Han, Q., Fan, Z., Dai, Q., Sun, L., Cheng, M., Liu, J., & Wang, J. (2021) · arXiv (Cornell University) · Cited by 26
Improve Vision Transformers Training by Suppressing Over-smoothingPreprint
Gong, C., Wang, D., Li, M., Chandra, V., & Liu, Q. (2021) · arXiv (Cornell University) · Cited by 18
Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPsPreprint
Zhang, Y., Chen, Z., & Zhong, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03815 · Cited by 11
BasisNet: Two-stage Model Synthesis for Efficient Inference
Zhang, M., Chu, C.-T., Zhmoginov, A., Howard, A., Jou, B., Zhu, Y., Zhang, L., Hwa, R., & Kovashka, A. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00344 · Cited by 7
Research works printed but not linked: 23
- dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ArXiv Preprint
- geirhos, 2018, Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, ArXiv Preprint
- dosovitskiy, 2021, An image is worth 16×16 words: Transformers for image recognition at scale, 9th International Conference on Learning Representations ICLR 2021
- cordonnier, 2019, On the relationship between self-attention and convolutional layers, ArXiv Preprint
- yu, 2015, Multiscale context aggregation by dilated convolutions, ArXiv Preprint
- liu, 2018, Darts: Differentiable architecture search, ArXiv Preprint
- ba, 2016, Layer normalization, ArXiv Preprint
- brendel, 2019, Approximating cnns with bag-of-local-features models works surprisingly well on imagenet, ArXiv Preprint
- howard, 2017, Mobilenets: Efficient convolutional neural networks for mobile vision applications, ArXiv Preprint
- rao, 2021, Global filter networks for image classification, ArXiv Preprint
- ramachandran, 2019, Standalone self-attention in vision models, ArXiv Preprint
- 2022, Official pytorch implementation of replknet
- raghu, 2021, Do vision transformers see like convolutional neural networks, ArXiv Preprint
- 2020, Megengine a fast scalable and easy-to-use deep learning framewok
- wu, 2019, Pay less attention with lightweight and dynamic convolutions, ArXiv Preprint
- wang, 0, Axial-deeplab: Standalone axial-attention for panoptic segmentation, European Conference on Computer Vision, 108
- vaswani, 0, Attention is all you need, Advances in neural information processing systems, 5998
- yu, 0, Di-lated residual networks, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 472
- chen, 0, Pretrained image processing transformer, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12299
- chen, 2021, An empirical study of training self-supervised visual transformers, ArXiv e-prints
- 2020, MMSegmentation Contributors MMSegmentation Openmmlab semantic segmentation toolbox and benchmark
- touvron, 0, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning, 10347
- trockman, 2022, Patches are all you need, ArXiv Preprint
Cited by
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00721 · Cited by 13,198
“Model re-parametrization techniques [71, 31, 75, 19, 33, 11, 4, 24, 13, 12, 10, 29, 14, 78] merge multiple computational modules into one at inference stage.”
MethodYOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2022) · arXiv · doi:10.48550/arxiv.2207.02696 · Cited by 13,205
“Model re-parametrization techniques [71, 31, 75, 19, 33, 11, 4, 24, 13, 12, 10, 29, 14, 78] merge multiple computational modules into one at inference stage.”
MethodHussain, M. (2023) · Machines · doi:10.3390/machines11070677 · Cited by 1,217
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080
Guo, M.-H., Lu, C.-Z., Liu, Z.-N., Cheng, M.-M., & Hu, S.-M. (2023) · Computational Visual Media · doi:10.1007/s41095-023-0364-2 · Cited by 1,073
Metadata
- Authors
- Ding, X., Zhang, X., Han, J., & Ding, G.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 11953-11965
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4226334005
- Cited by
- 1,447
- References
- 104
- FWCI
- 115.12
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- National Natural Science Foundation of China (61925107,U1936202,62021002)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.