Recognize Anything: A Strong Image Tagging Model
Open accessAbstract
We present the Recognize Anything Model (RAM): a strong foundation model for image tagging. RAM makes a substantial step for foundation models in computer vision, demonstrating the zero-shot ability to recognize any common category with high accuracy. By leveraging large-scale image-text pairs for training instead of manual annotations, RAM introduces a new paradigm for image tagging.The development of RAM comprises four key steps. Firstly, annotation-free image tags are obtained at scale through automatic text semantic parsing. Subsequently, a preliminary model is trained for automatic annotation by unifying the captioning and tagging tasks, supervised by the original texts and parsed tags, respectively. Thirdly, a data engine is employed to generate additional annotations and clean incorrect ones. Lastly, the model is retrained with the processed data and fine-tuned using a smaller but higher-quality dataset.We evaluate the tagging capability of RAM on numerous benchmarks and observe an impressive zero-shot performance, which significantly outperforms CLIP and BLIP. Remarkably, RAM even surpasses fully supervised models and exhibits a competitive performance compared with the Google tagging API. We have released RAM at https://recognize-anything.github.io/ to foster the advancement of foundation models in computer vision.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
By using an LVLM to add a description of the current camera image to the text prompt, the robot can answer visual queries.
“By using an LVLM [55] to add a description of the current camera image to the text prompt, the robot can also answer visual queries.”
From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [55]
The Model Architecture section (page unavailable) says RAM includes a text-generation encoder-decoder for image captioning, supporting the idea that it can describe an image. It does not describe adding a camera-image description to a robot’s text prompt or answering visual queries, so those parts of the attributed claim are unsupported. (LLM, conf=0.82)
Experimental — accuracy not yet measuredObjects are assigned language tags by leveraging large language models and large vision-language models, which provide semantically rich descriptions and enable free-form language querying, all while using off-the-shelf models without training or finetuning.
“Objects are assigned language tags by leveraging large language models (LLMs) [32] and large vision-language models (LVLMs) [55], which provide semantically rich descriptions and enable free-form language querying, all while using offthe-shelf models (no training/finetuning).”
From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [55]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThe node labels are accurate about 70% of the time; most of the errors are incurred due to errors made by the LVLM employed (LLaVA).
“The node labels are accurate about 70% of the time; most of the errors are incurred due to errors made by the LVLM employed (LLaVA [55]).”
From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [55]
The Label System and Evaluation Metrics sections discuss RAM’s image tags and tagging performance, not node-label accuracy or errors caused by LLaVA. Searches for “LLaVA,” “node labels,” and “70%” found no passage in our copy addressing the attributed claim. (LLM, conf=0.77)
Experimental — accuracy not yet measuredThe LLaVA-7B model used for node captioning misclassifies a non-negligible number of small objects as toothbrushes or pairs of scissors.
“In particular, the LLaVA-7B [55] model used for node captioning misclassifies a non-negligible number of small objects as toothbrushes or pairs of scissors .”
From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [55]
The Model Ablation Study (section 4.3; page unavailable) discusses RAM’s tagging ability, while the comparison with vision-language models discusses CLIP and BLIP. Searches for LLaVA-7B, toothbrushes, scissors, and small-object misclassification found no passage addressing the claimed captioning errors. (LLM, conf=0.76)
Experimental — accuracy not yet measuredThe computational and economic costs of ConceptGraphs include multiple LVLM and one or more proprietary LLM inferences when building and querying the scene graph, which may be significant.
“Additionally, the computational and economic costs of our system include multiple LVLM (LLaVA [55]) and one or more proprietary LLM inference(s) when building and querying the scenegraph, which may be significant.”
From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [55]
Section 2.3, “Model Efficiency” (page unavailable), discusses RAM’s training and image-tagging inference efficiency, not ConceptGraphs’ scene-graph costs. Searches for ConceptGraphs, LLaVA, proprietary LLM inference, and scene-graph querying found no passage addressing the attributed claim. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Node captioning incurs errors due to the current limitations of LVLMs like LLaVA.
“First, node captioning incurs errors due to the current limitations of LVLMs like LLaVA [55].”
From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [55]
The “Compared with Vision-Language Models” section discusses accuracy limitations of CLIP and BLIP, and the Figure 2 caption says BLIP’s tag recall is limited by caption generation. Neither passage addresses node-captioning errors or attributes them to limitations of LLaVA; searches for LLaVA and captioning errors found no passage that does. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 1× · see citing works
“The 4M setting includes two human-annotated datasets, COCO [16] (113K images, 557K captions) and Visual Genome [13] (101K images, 822K captions), along with two large-scale web-based datasets, Conceptual Cap-tions [6] (3M images, 3M captions) and SBU Cap-tions [21] (849K images, 849K captions).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
Self-citationLi, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · see citing works
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · cited 1× · see citing works
“To address the issue of incorrect tags, we initially employ Grounding-DINO [28] to identify and crop regions corresponding to a specific category within all images.”
Self-citation
Flagged first, then by citations
Research works printed but not linked: 22
- David Arthur and Sergei Vassilvitskii. K-means++ the ad- vantages of careful seeding. In Proceedings of the eigh- teenth annual ACM-SIAM symposium on Discrete algo- rithms, pages 1027-1035, 2007.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877-1901, 2020.
- Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021.
- Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Bo Ren, and Shu-Tao Xia. Open-vocabulary multi-label classification via multi-modal knowledge transfer. CoRR, abs/2207.01887, 2022. · arXiv
- Xinyu Huang, Youcai Zhang, Ying Cheng, Weiwei Tian, Ruiwei Zhao, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Xiaobo Zhang. Idea: Increasing text diversity via online multi-label recognition for vision-language pre- training. In Proceedings of the 30th ACM International Con- ference on Multimedia, pages 4573-4583, 2022.
- Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657, 2023. · arXiv
- Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
- Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. · arXiv
- Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label clas- sification. arXiv preprint arXiv:2107.10834, 2021. · arXiv
- Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. Advances in neural information processing sys- tems, 24, 2011.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
- Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben Baruch, and Asaf Noy. Ml-decoder: Scalable and versatile classification head. In IEEE/CVF Winter Conference on Ap- plications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, pages 32-41. IEEE, 2023.
- Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, Xiaogang Wang, and Yu Qiao. Internim- age: Exploring large-scale vision foundation models with deformable convolutions. CoRR, abs/2211.05778, 2022. · arXiv
- Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6609-6618, 2019.
- Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xi- aolong Wang, and Shalini De Mello. Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Mod- els. arXiv preprint arXiv:2303.04803, 2023. · arXiv
- Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. In NeurIPS, 2022.
- Yuhang Zang, W ei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part IX, pages 106-122. Springer, 2022.
- Youcai Zhang, Yuhao Cheng, Xinyu Huang, Fei Wen, Rui Feng, Yaqian Li, and Yandong Guo. Simple and robust loss design for multi-label learning with missing labels. arXiv preprint arXiv:2112.07368, 2021. · arXiv
- Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
- Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
- Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything every- where all at once. CoRR, abs/2304.06718, 2023. · arXiv
Cited by
Sorted by citations
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., & Zhao, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00987 · Cited by 1,296 · see citing works
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610243 · Cited by 255 · see citing works
“We also develop a variant of our system, ConceptGraphs - Detector ( CG-D ), where we employ an image tagging model (RAM [55]) to list the object classes present in the image and an open-vocabulary 2D detector (Grounding DINO [34]) to obtain object bounding boxes 6 .”
MethodBridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill
Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., & Dong, H. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610499 · Cited by 63 · see citing works
PneumoLLM: Harnessing the power of large language model for pneumoconiosis diagnosis
Song, M., Wang, J., Yu, Z., Wang, J., Yang, L., Lu, Y., Li, B., Wang, X., Wang, X., Huang, Q., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103248 · Cited by 31 · see citing works
Self-citation
- Authors
- Zhang, Y., Huang, X., Ma, J., Li, Z., Luo, Z., Xie, Y., Qin, Y., Luo, T., Li, Y., Liu, S., Guo, Y., & Zhang, L.
- Affiliations
- OPPO Research Institute
- International Digital Economy Academy (IDEA)
- AI2Robotics
- ORCID
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2024-06-17
- Pages
- 1724-1732
- OpenAlex
- W4402917081
- Cited by
- 242
- References
- 6
- Citations by year
- 2026: 70 · 2025: 104 · 2024: 64 · 2023: 4
- FWCI
- 44.28
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- image tagging, foundation models, zero-shot recognition, image-text pairs, fine-tuning, CLIP, BLIP
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
