CoChat
Sign Up
Cited by 66Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesAdvanced Neural Network Applications

FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

Open access
Rotstein, Noam, Bensaïd, David, Brody, Shaked, Ganz, Roy, Kimmel, Ron·2024·2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
DOI 10.1109/wacv57701.2024.00559
Open this paper in CoChat — free accountOpen PDF

Abstract

The advent of vision-language pre-training techniques enhanced substantial progress in the development of models for image captioning. However, these models frequently produce generic captions and may omit semantically important image details. This limitation can be traced back to the image-text datasets; while their captions typically offer a general description of image content, they frequently omit salient details. Considering the magnitude of these datasets, manual reannotation is impractical, emphasizing the need for an automated approach. To address this challenge, we leverage existing captions and explore augmenting them with visual details using "frozen" vision experts including an object detector, an attribute recognizer, and an Optical Character Recognizer (OCR). Our proposed method, FuseCap, fuses the outputs of such vision experts with the original captions using a large language model (LLM), yielding comprehensive image descriptions. We automatically curate a training set of 12M image-enriched caption pairs. These pairs undergo extensive evaluation through both quantitative and qualitative analyses. Subsequently, this data is utilized to train a captioning generation BLIP-based model. This model outperforms current state-of-the-art approaches, producing more precise and detailed descriptions, demonstrating the effectiveness of the proposed data-centric approach. We release this large-scale dataset of enriched image-caption pairs for the community.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/wacv57701.2024.00559

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Concerns exist regarding the reliability and appropriateness of AI-generated alt text in academic contexts.

    “However, concerns exist regarding the reliability and appropriateness of AI-generated alt text in academic contexts [27, 48, 66].”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [66]

    The introductory discussion says current image-captioning models often omit key semantic elements, and Section 6 (Limitations) notes that FuseCap can miss relationships between objects and may replicate biases. These support concerns about the reliability of generated descriptions, but the paper does not assess the appropriateness of AI-generated alt text in academic contexts. Page numbers were not provided in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Challenges of domain-specific alt text creation present an opportunity to apply these methods to established practices to support rather than replace professionals' work.

    “At the same time, challenges of domainspecific alt text creation present an opportunity to apply these methods to established practices to support rather than replace professionals' work [6, 66].”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [66]

    The introduction (‘Original: Mhmm, some clouds in the sky’; page unavailable) describes FUSECAP as automatically augmenting existing captions, including captions in a human-annotated dataset. That supports applying the method to existing captioning work, but the paper does not address domain-specific alt text or propose supporting rather than replacing professionals’ work. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Little is known about writing support for non-experts, such as DSO professionals, in figure accessibility.

    “While previous research in figure accessibility has mostly focused on readers [2, 32, 66], or authors of papers [31, 71], little is known about the writing support for non-experts , such as DSO professionals.”

    From How University Disability Services Professionals Write Image Descriptions for HCI Figures Using Generative AI (2026) · cites this paper as [66]

    The Introduction and Conclusions discuss automated image caption generation, not writing support for scientific figures. Searches for figure accessibility, non-expert writers, and DSO professionals found no passage addressing the claimed research gap. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 70

  1. Aviad Aberdam, David Bensaïd, Alona Golts, Roy Ganz, Oren Nuriel, Royee Tichauer, Shai Mazor, and Ron Litman. Clipter: Looking at the bigger picture in scene text recogni- tion. arXiv preprint arXiv:2301.07464, 2023. 2 · arXiv
  2. Aviad Aberdam, Roy Ganz, Shai Mazor, and Ron Litman. Multimodal semi-supervised learning for text recognition. arXiv preprint arXiv:2205.03873, 2022. 2 · arXiv
  3. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. Nocaps: Novel object caption- ing at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8948-8957, 2019. 1
  4. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018. 2
  5. Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detec- tion. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 9365-9374, 2019. 4, 13
  6. K. Barnard and D. Forsyth. Learning the semantics of words and pictures. In Proceedings Eighth IEEE International Con- ference on Computer Vision. ICCV 2001, volume 2, pages 408-415 vol.2, 2001. 1
  7. Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In European Conference on Computer Vision, pages 178-196, Cham, 10 2022. Springer Nature Switzerland. 2, 4, 13
  8. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291-4301, 2019. 4
  9. Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022. 3 · arXiv
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2, 3, 4
  11. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021. 2, 3, 4
  12. Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38-56, 2023. 2
  13. Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Brad- bury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model. CoRR, 2022. 1
  14. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In An- drea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision -ECCV 2020 -16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part XXX. Springer, 2020. 1
  15. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104- 120. Springer, 2020. 2
  16. Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In Interna- tional Conference on Machine Learning, pages 1931-1942. PMLR, 2021. 2, 4
  17. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional net- works for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 2625-2634, 2015. 2
  18. Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18166-18176, 2022. 2
  19. Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Pe- ter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. Every picture tells a story: Generating sentences from images. In Computer Vision-ECCV 2010: 11th Eu- ropean Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pages 15-29. Springer, 2010. 2
  20. Roy Ganz and Michael Elad. Clipag: Towards generator-free text-to-image generation. arXiv preprint arXiv:2306.16805, 2023. 2 · arXiv
  21. Roy Ganz, Oren Nuriel, Aviad Aberdam, Yair Kittenplon, Shai Mazor, and Ron Litman. Towards models that can see and read. arXiv preprint arXiv:2301.07389, 2023. 2, 4 · arXiv
  22. Micah Hodosh, Peter Young, and Julia Hockenmaier. Fram- ing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853-899, 2013. 2
  23. Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision- language pre-training for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17980-17989, 2022. 1
  24. Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense caption- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4565-4574, 2016. 2
  25. Andrej Karpathy, Armand Joulin, and Li Fei-Fei. Deep frag- ment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Sys- tems 2014, December 8-13 2014, Montreal, Quebec, Canada, 2014. 1
  26. Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. Baby talk: Understanding and generating simple image descriptions. In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, 2011. 1
  27. Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple im- age descriptions. IEEE transactions on pattern analysis and machine intelligence, 35(12):2891-2903, 2013. 2
  28. Polina Kuznetsova, Vicente Ordonez, Alexander C. Berg, Tamara L. Berg, and Yejin Choi. Collective generation of natural image descriptions. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea -Volume 1: Long Papers, 2012. 1
  29. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 2, 5, 7 · arXiv
  30. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In International Con- ference on Machine Learning, pages 12888-12900. PMLR, 2022. 1, 2, 5, 7, 13
  31. Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu-Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan, editors, Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Decem- ber 6-14, 2021, virtual, 2021. 1
  32. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. 2, 4 · arXiv
  33. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020. 4
  34. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004. 5
  35. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014. 1, 2, 3
  36. Lixin Liu, Jiajun Tang, Xiaojun Wan, and Zongming Guo. Generating diverse and descriptive image captions using vi- sual paraphrases. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 4240-4249, 2019. 3
  37. Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02506, 2023. 2, 4, 7 · arXiv
  38. Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xi- aogang Wang. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In Proceedings of the European conference on computer vision (ECCV), pages 338-354, 2018. 3
  39. Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training de- scriptive captions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6964-6974, 2018. 3
  40. Rebecca Mason and Eugene Charniak. Nonparametric method for data-driven image captioning. In Proceedings of the 52nd Annual Meeting of the Association for Computa- tional Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, 2014. 1
  41. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011. 3
  42. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002. 5
  43. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. 4 · arXiv
  44. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 1, 2, 6
  45. Krishnan Ramnath, Simon Baker, Lucy Vanderwende, Mo- taz Ahmad El-Saban, Sudipta N. Sinha, Anitha Kannan, No- ran Hassan, Michel Galley, Yi Yang, Deva Ramanan, Alessan- dro Bergamo, and Lorenzo Torresani. Autocaption: Auto- matic caption generation for personal photos. In IEEE Winter Conference on Applications of Computer Vision, Steamboat Springs, CO, USA, March 24-26, 2014, 2014. 1
  46. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Can- cedda, and Thomas Scialom. Toolformer: Language mod- els can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. 3 · arXiv
  47. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018. 2, 3, 4
  48. Zhan Shi, Hui Liu, and Xiaodan Zhu. Enhancing descrip- tive image captioning with natural language inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 269-277, 2021. 3
  49. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhi- nav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017. 3
  50. Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019. 4 · arXiv
  51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2
  52. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalu- ation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015. 5
  53. Tomás F Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras. Large-scale training of shadow detec- tors with noisily-annotated shadow examples. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 816-832. Springer, 2016. 2
  54. Alex Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-guided text prompt for vision- language pre-training. arXiv preprint arXiv:2212.09737, 2022. 4 · arXiv
  55. Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. 1 · arXiv
  56. Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. CoRR, 2022. 2, 7
  57. Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al. Caption anything: Interactive image de- scription with diverse multimodal controls. arXiv preprint arXiv:2305.02677, 2023. 4 · arXiv
  58. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021. 2 · arXiv
  59. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In The Tenth In- ternational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022. 1
  60. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. 3 · arXiv
  61. Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017. 4
  62. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gener- ation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015. 2
  63. Linjie Yang, Kevin Tang, Jianchao Yang, and Li-Jia Li. Dense captioning with joint inference and visual context. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 2193-2202, 2017. 2
  64. Zhilin Yang, Ye Yuan, Yuexin Wu, William W Cohen, and Russ R Salakhutdinov. Review networks for caption genera- tion. Advances in neural information processing systems, 29, 2016. 2
  65. Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, and Xia Hu. Data-centric ai: Perspectives and challenges. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 945-948. SIAM, 2023. 2, 3
  66. Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data- centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023. 2, 3 · arXiv
  67. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2021, virtual, June 19-25, 2021, 2021. 1, 5
  68. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021. 2, 4
  69. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained trans- former language models. arXiv preprint arXiv:2205.01068, 2022. 7 · arXiv
  70. Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, blip- 2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594, 2023. · arXiv

Cited by

Sorted by citations

Authors
Rotstein, N., Bensaïd, D., Brody, S., Ganz, R., & Kimmel, R.
Affiliations
Technion – Israel Institute of Technology (IL)
ORCID
Kimmel, Ron
Venue
2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Publisher
IEEE
Published
2024-01-03
Pages
5677-5688
ISSN
2472-6737 (ISSN-L)
DOI
10.1109/wacv57701.2024.00559
OpenAlex
W4394625850
Cited by
66
References
11
Citations by year
2026: 14 · 2025: 32 · 2024: 20
FWCI
13.62
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
image captioning, large language models, vision-language models, object detection, attribute identification, optical character recognition, image-text dataset, BLIP, data augmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review