The DOI matches its Crossref record; 118 of the 118 references it cites were found in a registry or an index.
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
Abstract
We present MaX-DeepLab, the first end-to-end model for panoptic segmentation. Our approach simplifies the current pipeline that depends heavily on surrogate sub-tasks and hand-designed components, such as box detection, non-maximum suppression, thing-stuff merging, etc. Although these sub-tasks are tackled by area experts, they fail to comprehensively solve the target task. By contrast, our MaX-DeepLab directly predicts class-labeled masks with a mask transformer, and is trained with a panoptic quality inspired loss via bipartite matching. Our mask transformer employs a dual-path architecture that introduces a global memory path in addition to a CNN path, allowing direct communication with any CNN layers. As a result, MaX-DeepLab shows a significant 7.1% PQ gain in the box-free regime on the challenging COCO dataset, closing the gap between box-based and box-free methods for the first time. A small variant of MaX-DeepLab improves 3.0% PQ over DETR with similar parameters and M-Adds. Furthermore, MaX-DeepLab, without test time augmentation, achieves new state-of-the-art 51.3% PQ on COCO test-dev set.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Cited by
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Masked-attention Mask Transformer for Universal Image Segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., & Girdhar, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00135 · Cited by 3,565
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,051
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S.-N. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19827-4_41 · Cited by 1,965
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526
Multiscale vision transformers
Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., & Feichtenhofer, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00675 · Cited by 1,407
“An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video instance segmentation…”
BackgroundChen, J., Mei, J., Li, X., Lu, Y., Yu, Q., Wei, Q., Luo, X., Xie, Y., Adeli, E., Wang, Y., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103280 · Cited by 1,365
Self-citationLearning Spatio-Temporal Transformer for Visual Tracking
Yan, B., Peng, H., Fu, J., Wang, D., & Lu, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01028 · Cited by 1,280
“Recently, transformer has shown their great potential in vision tasks like image classification [12], object detection [5], semantic segmentation [49], multiple object tracking [44, 35], etc.”
BackgroundPoint-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., & Lu, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01871 · Cited by 801
FAT-Net: Feature adaptive transformers for automated skin lesion segmentation
Wu, H., Chen, S., Chen, G., Wang, W., Lei, B., & Wen, Z. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102327 · Cited by 590
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation
Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L. M., & Shum, H.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00297 · Cited by 560
Liu, N., Zhang, N., Wan, K., Shao, L., & Han, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00468 · Cited by 549
A Survey of Visual Transformers
Liu, Y., Zhang, Y., Wang, Y., Hou, F., Yuan, J., Tian, J., Zhang, Y., Shi, Z., Fan, J., & He, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3227717 · Cited by 549
TransFG: A Transformer Architecture for Fine-Grained Recognition
He, J., Chen, J.-N., Liu, S., Kortylewski, A., Yang, C., Bai, Y., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19967 · Cited by 551
Voxel Transformer for 3D Object Detection
Mao, J., Xue, Y., Niu, M., Bai, H., Feng, J., Liang, X., Xu, H., & Xu, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00315 · Cited by 520
OneFormer: One Transformer to Rule Universal Image Segmentation
Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., & Shi, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00292 · Cited by 504
A survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 451
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414
Twin Adversarial Contrastive Learning for Underwater Image Enhancement and Beyond
Liu, R., Jiang, Z., Yang, S., & Fan, X. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3190209 · Cited by 390
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Xu, J., Liu, S., Vahdat, A., Byeon, W., Wang, X., & De Mello, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00289 · Cited by 359
Large-scale pancreatic cancer detection via non-contrast CT and deep learning
Cao, K., Xia, Y., Yao, J., Han, X., Lambert, L., Zhang, T., Tang, W., Jin, G., Jiang, H., Fang, X., et al. (2023) · Nature Medicine · doi:10.1038/s41591-023-02640-w · Cited by 358
“…diagnosis of pancreatic lesions if any abnormalities are detected in the second stage, integrated with an auxiliary memory transformer branch 24,25 to automatically encode the feature prototypes of the pancreatic lesions, such as local texture, position and pancreas shape, for more accurate…”
Self-citationMPViT: Multi-Path Vision Transformer for Dense Prediction
Lee, Y., Kim, J., Willette, J., & Hwang, S. J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00714 · Cited by 351
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
Ghiasi, G., Gu, X., Cui, Y., & Lin, T.-Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_31 · Cited by 339
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
Zhang, P., Dai, X., Yang, J., Xiao, B., Yuan, L., Zhang, L., & Gao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00299 · Cited by 330
Fang, Y., Yang, S., Wang, X., Li, Y., Fang, C., Shan, Y., Feng, B., & Liu, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00683 · Cited by 320
P2T: Pyramid Pooling Transformer for Scene Understanding
Wu, Y.-H., Liu, Y., Zhan, X., & Cheng, M.-M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3202765 · Cited by 309
Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification
Zhu, H., Ke, W., Li, D., Liu, J., Tian, L., & Shan, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00465 · Cited by 277
CAT: Cross Attention in Vision Transformer
Lin, H., Cheng, X., Wu, X., & Shen, D. (2022) · 2022 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme52920.2022.9859720 · Cited by 269
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer
Zhao, C., Zhang, Y., Poggi, M., Tosi, F., Guo, X., Zhu, Z., Huang, G., Tang, Y., & Mattoccia, S. (2022) · 2022 International Conference on 3D Vision (3DV) · doi:10.1109/3dv57658.2022.00077 · Cited by 267
Focal Self-attention for Local-Global Interactions in Vision Transformers
Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00641 · Cited by 256
Lite Vision Transformer with Enhanced Self-Attention
Yang, C., Wang, Y., Zhang, J., Zhang, H., Wei, Z., Lin, Z., & Yuille, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01169 · Cited by 164
Self-citationPTQ4ViT: Post-Training Quantization for Vision Transformers with Twin Uniform Quantization
Yuan, Z., Xue, C., Chen, Y., Wu, Q., & Sun, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_12 · Cited by 163
“Vision transformers have been successfully applied to downstream tasks [15,24,12].”
MethodDenoising Pretraining for Semantic Segmentation
Brempong, E. A., Kornblith, S., Chen, T., Parmar, N., Minderer, M., & Norouzi, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00462 · Cited by 120
Chasing Sparsity in Vision Transformers: An End-to-End Exploration
Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2106.04533 · Cited by 111
Revisiting Weakly Supervised Pre-Training of Visual Perception Models
Singh, M., Gustafson, L., Adcock, A., De Freitas Reis, V., Gedik, B., Kosaraju, R. P., Mahajan, D., Girshick, R., Dollar, P., & Van Der Maaten, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00088 · Cited by 99
“Fully supervised pre-training was pioneered by [19, 59] and is now the de-facto standard approach to a variety of visual-recognition tasks, including fine-grained image classification [30, 62], object detection [61], image segmentation [32, 70], image captioning [46], visual question answering [40], video classificati…”
BackgroundASFormer: Transformer for Action Segmentation
Yi, F., Wen, H., & Jiang, T. (2021) · Proceedings of the British Machine Vision Conference 2021 · doi:10.5244/c.35.49 · Cited by 170
Transformers for Vision: A Survey on Innovative Methods for Computer Vision
Palanisamy, B., Hassija, V., Chatterjee, A., Mandal, A., Chakraborty, D., Pandey, A., Chalapathi, G. S. S., & Kumar, D. (2025) · IEEE Access · doi:10.1109/access.2025.3571735 · Cited by 83
“It achieves advanced performance using a ‘‘dual-path transformer’’ that combines CNNs with transformers [62].”
Zhu, Y., Pan, Y., Hu, T., & Liu, Y. (2025) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2025.3628397 · Cited by 3
Self-citation
Metadata
- Authors
- Wang, H., Zhu, Y., Adam, H., Yuille, A., & Chen, L.-C.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 5459-5470
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3168649818
- Cited by
- 497
- References
- 118
- FWCI
- 36.39
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.