The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
General Multi-label Image Classification with Transformers
Open accessAbstract
Multi-label image classification is the task of predicting a set of labels corresponding to objects, attributes or other entities present in an image. In this work we propose the Classification Transformer (C-Tran), a general framework for multi-label image classification that leverages Transformers to exploit the complex dependencies among visual features and labels. Our approach consists of a Transformer encoder trained to predict a set of target labels given an input set of masked labels, and visual features from a convolutional neural network. A key ingredient of our method is a label mask training objective that uses a ternary encoding scheme to represent the state of the labels as positive, negative, or unknown during training. Our model shows state-of-the-art performance on challenging datasets such as COCO and Visual Genome. Moreover, because our model explicitly represents the label state during training, it is more general by allowing us to produce improved results for images with partial or extra label annotations during inference. We demonstrate this additional capability in the COCO, Visual Genome, News-500, and CUB image datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
(2008) · Journal of Machine Learning Research · Cited by 33,051
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169
Multi-Label Classification: An Overview
Tsoumakas, G., & Katakis, I. (2007) · International Journal of Data Warehousing and Mining · doi:10.4018/jdwm.2007070101 · Cited by 2,508
“Cloze Procedure”: A New Tool for Measuring Readability
Taylor, W. L. (1953) · Journalism Quarterly · doi:10.1177/107769905303000401 · Cited by 2,332
Large Margin Methods for Structured and Interdependent Output Variables
Tsochantaridis, I., Joachims, T., Hofmann, T., & Altün, Y. (2005) · Journal of Machine Learning Research · doi:10.5555/1046920.1088722 · Cited by 2,225
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849
A Kernel Method for Multi-Labelled Classification
Elisseeff, A., & Weston, J. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0092 · Cited by 1,342
Multi-Label Image Recognition With Graph Convolutional Networks
Chen, Z.-M., Wei, X.-S., Wang, P., & Guo, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00532 · Cited by 1,261
CNN-RNN: A Unified Framework for Multi-label Image Classification
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., & Xu, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.251 · Cited by 1,198
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D. (2018) · arXiv · doi:10.48550/arxiv.1802.05751 · Cited by 1,085
CNN: Single-label to Multi-label
Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., & Yan, S. (2014) · arXiv · doi:10.48550/arxiv.1406.5726 · Cited by 784
Classifier Chains for Multi-label Classification
Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-04174-7_17 · Cited by 607
(2021) · International Conference on Learning Representations · Cited by 449
Learning Spatial Regularization with Image-Level Supervisions for Multi-label Image Classification
Zhu, F., Li, H., Ouyang, W., Yu, N., & Wang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.219 · Cited by 380
Bayes Optimal Multilabel Classification via Probabilistic Classifier Chains
Cheng, W., llermeier, E. H., & Dembczyński, K. (2010) · International Conference on Machine Learning · Cited by 362
Large-Scale Object Classification Using Label Relation Graphs
Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., & Adam, H. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_4 · Cited by 347
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2007.04612 · Cited by 338
Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition
Chen, T., Xu, M., Hui, X., Wu, H., & Lin, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00061 · Cited by 332
Multi-label Image Recognition by Recurrently Discovering Attentional Regions
Wang, Z., Chen, T., Li, G., Xu, R., & Lin, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.58 · Cited by 326
Multi-label Zero-Shot Learning with Structured Knowledge Graphs
Lee, C.-W., Fang, W., Yeh, C.-K., & Wang, Y.-C. F. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00170 · Cited by 305
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, E., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1032 · Cited by 279
Sparse local embeddings for extreme multi-label classification
Bhatia, K., Jain, H., Kar, P., Varma, M., & Jain, P. (2015) · neural information processing systems · Cited by 264
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships
Liu, Y., Wang, R., Shan, S., & Chen, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00730 · Cited by 252
Learning Deep Latent Space for Multi-Label Classification
Yeh, C., Wu, W.-C., Ko, W.-J., & Wang, Y.-C. F. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10769 · Cited by 246
Learning a Deep ConvNet for Multi-Label Classification With Partial Labels
Durand, T., Mehrasa, N., & Mori, G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00074 · Cited by 245
Ge, W., Yang, S., & Yu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00139 · Cited by 210
Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition
Chen, T., Lin, L., Chen, R., Hui, X., & Wu, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3025814 · Cited by 163
Order-Free RNN With Visual Attention for Multi-Label Classification
Chen, S., Chen, Y., Yeh, C., & Wang, Y.-C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12230 · Cited by 159
ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
Stock, P., & Cisse, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_31 · Cited by 158
Instance-level Image Retrieval using Reranking Transformers
Tan, F., Yuan, J., & Ordóñez, V. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01189 · Cited by 143
Learning Structured Inference Neural Networks with Label Relations
Hu, H., Zhou, G.-T., Deng, Z., Liao, Z., & Mori, G. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.323 · Cited by 142
Maximizing Subset Accuracy with Recurrent Neural Networks in Multi-label Classification
Nam, J., Mencía, E. L., Kim, H. J., & Fürnkranz, J. (2017) · neural information processing systems · Cited by 128
Conditional Graphical Lasso for Multi-label Image Classification
Li, Q., Qiao, M., Bian, W., & Tao, D. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.325 · Cited by 124
Multi-label image classification with a probabilistic label enhancement model
Li, X., Zhao, F., & Guo, Y. (2014) · Cited by 94
Neural Message Passing for Multi-label Classification
Lanchantin, J., Sekhon, A., & Qi, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46147-8_9 · Cited by 55
Self-citationCommonly Uncommon: Semantic Sparsity in Situation Recognition
Yatskar, M., Ordóñez, V., Zettlemoyer, L., & Farhadi, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.671 · Cited by 47
Neural Networks with Recurrent Generative FeedbackPreprint
Huang, Y., Gornet, J., Dai, S., Yu, Z., Nguyen, T. M., Tsao, D. Y., & Anandkumar, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.09200 · Cited by 32
Predicting Entry-Level Categories
Ordóñez, V., Liu, W., Deng, J., Choi, Y., Berg, A. C., & Berg, T. L. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0815-z · Cited by 22
Exploiting weakly supervised visual patterns to learn from partial annotations
Kundu, K., & Tighe, J. (2020) · neural information processing systems · Cited by 22
Nimmagadda, T., & Anandkumar, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00308 · Cited by 11
Feedback-Prop: Convolutional Neural Network Inference Under Partial Evidence
Wang, T., Yamaguchi, K., & Ordóñez, V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00100 · Cited by 10
Self-citationPlugin Networks for Inference under Partial Evidence
Koperski, M., Konopczyński, T., Nowak, R., Semberecki, P., & Trzcinski, T. P. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093644 · Cited by 5
Using Visual Feature Space as a Pivot Across Languages
Yang, Z., Pinto-Alva, L., Dernoncourt, F., et al. (2020) · doi:10.18653/v1/2020.findings-emnlp.328 · Cited by 4
Cited by
Feng, Y., Xu, H., Jiang, J., Liu, H., & Zheng, J. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3168331 · Cited by 339
Transparent medical image AI via an image–text foundation model grounded in medical literature
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, J. A., Cai, Z. R., Daneshjou, R., & Lee, S.-I. (2024) · Nature Medicine · doi:10.1038/s41591-024-02887-x · Cited by 125
A Simple and Interactive Transformer for Fine-Grained Emotion Detection
Sui, D., Li, B., Yang, H., Chen, Y., Liu, K., Chu, D., & Zhao, J. (2025) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/taslp.2024.3487418 · Cited by 2
Metadata
- Authors
- Lanchantin, J., Wang, T., Ordonez, V., & Qi, Y.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 16473-16483
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3167456680
- Cited by
- 350
- References
- 60
- FWCI
- 21.83
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.