CoChat
Sign Up
Cited by 373Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesGenerative Adversarial Networks and Image Synthesis

From Show to Tell: A Survey on Deep Learning-Based Image Captioning

Open access
Stefanini, Matteo, Cornia, Marcella, Baraldi, Lorenzo, Cascianelli, Silvia, Fiameni, Giuseppe, Cucchiara, Rita·2023·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2022.3148210
Open this paper in CoChat — free accountOpen PDF

Abstract

Connecting Vision and Language plays an essential role in Generative Intelligence. For this reason, large research efforts have been devoted to image captioning, i.e. describing images with syntactically and semantically meaningful sentences. Starting from 2015 the task has generally been addressed with pipelines composed of a visual encoder and a language model for text generation. During these years, both components have evolved considerably through the exploitation of object regions, attributes, the introduction of multi-modal connections, fully-attentive approaches, and BERT-like early-fusion strategies. However, regardless of the impressive results, research in image captioning has not reached a conclusive answer yet. This work aims at providing a comprehensive overview of image captioning approaches, from visual encoding and text generation to training strategies, datasets, and evaluation metrics. In this respect, we quantitatively compare many relevant state-of-the-art approaches to identify the most impactful technical innovations in architectures and training strategies. Moreover, many variants of the problem and its open challenges are discussed. The final goal of this work is to serve as a tool for understanding the existing literature and highlighting the future directions for a research area where Computer Vision and Natural Language Processing can find an optimal synergy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2022.3148210

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 268 · Cited by 373

Research works printed but not linked: 219

  1. [3][3] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, 'Every picture tells a story: Generating sentences from images,' in ECCV , 2010. -
  2. [4][4] V. Ordonez, G. Kulkarni, and T. Berg, 'Im2text: Describing images using 1 million captioned photographs,' in NeurIPS , 2011. -
  3. [5][5] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, 'DeViSE: a deep visual-semantic embedding model,' in NeurIPS , 2013. -
  4. [6][6] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying visualsemantic embeddings with multimodal neural language models,' in NeurIPS Workshops , 2014. -
  5. [7][7] A. Karpathy, A. Joulin, and L. Fei-Fei, 'Deep fragment embeddings for bidirectional image sentence mapping,' in NeurIPS , 2014. -
  6. [8][8] B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu, 'I2T: Image parsing to text description,' Proceedings of the IEEE , 2010. -
  7. [9][9] A. Aker and R. Gaizauskas, 'Generating image descriptions using dependency relational patterns,' in ACL , 2010. -
  8. [10][10] Y. Yang, C. Teo, H. Daum´ e III, and Y. Aloimonos, 'Corpus-guided sentence generation of natural images,' in EMNLP , 2011. -
  9. [12][12] A. Gupta, Y. Verma, and C. Jawahar, 'Choosing linguistics over vision to describe images,' in AAAI , 2012. -
  10. [13][13] M. Mitchell, J. Dodge, A. Goyal, K. Yamaguchi, K. Stratos, X. Han, A. Mensch, A. Berg, T. Berg, and H. Daum´ e III, 'Midge: Generating image descriptions from computer vision detections,' in ACL , 2012. -
  11. [14][14] G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, 'BabyTalk: Understanding and generating simple image descriptions,' IEEE Trans. PAMI , 2013. -
  12. [15][15] P. Kuznetsova, V. Ordonez, T. L. Berg, and Y. Choi, 'Treetalk: Composition and compression of trees for image descriptions,' TACL , vol. 2, pp. 351-362, 2014. -
  13. [16][16] R. Bernardi, R. Cakici, D. Elliott, A. Erdem, E. Erdem, N. IkizlerCinbis, F. Keller, A. Muscat, and B. Plank, 'Automatic description generation from images: A survey of models, datasets, and evaluation measures,' JAIR , vol. 55, pp. 409-442, 2016. -
  14. [17][17] S. Bai and S. An, 'A survey on automatic image caption generation,' Neurocomputing , vol. 311, pp. 291-304, 2018. -
  15. [21][21] X. Liu, Q. Xu, and N. Wang, 'A survey on deep neural networkbased image captioning,' The Visual Computer , vol. 35, no. 3, pp. 445-470, 2019. -
  16. [24][24] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, 'Going deeper with convolutions,' in CVPR , 2015. -
  17. [27][27] J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, 'Deep Captioning with Multimodal Recurrent Neural Networks (mRNN),' in ICLR , 2015. -
  18. [28][28] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, 'Long-term recurrent convolutional networks for visual recognition and description,' in CVPR , 2015. -
  19. [29][29] K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' in ICLR , 2015. -
  20. [30][30] X. Chen and C. Lawrence Zitnick, 'Mind's Eye: A Recurrent Visual Representation for Image Caption Generation,' in CVPR , 2015. -
  21. [31][31] H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Doll´ ar, J. Gao, X. He, M. Mitchell, J. C. Platt et al. , 'From captions to visual concepts and back,' in CVPR , 2015. -
  22. [32][32] X. Jia, E. Gavves, B. Fernando, and T. Tuytelaars, 'Guiding the Long-Short Term Memory model for Image Caption Generation,' in ICCV , 2015. -
  23. [33][33] Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, 'Image captioning with semantic attention,' in CVPR , 2016. -
  24. [34][34] Q. Wu, C. Shen, L. Liu, A. Dick, and A. Van Den Hengel, 'What Value Do Explicit High Level Concepts Have in Vision to Language Problems?' in CVPR , 2016. -
  25. [35][35] J. Gu, G. Wang, J. Cai, and T. Chen, 'An Empirical Study of Language CNN for Image Captioning,' in ICCV , 2017. -
  26. [37][37] F. Chen, R. Ji, X. Sun, Y. Wu, and J. Su, 'GroupCap: Group-based Image Captioning with Structured Relevance and Diversity Constraints,' in CVPR , 2018. -
  27. [38][38] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, 'Selfcritical sequence training for image captioning,' in CVPR , 2017. -
  28. [40][40] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, 'Boosting image captioning with attributes,' in ICCV , 2017. -
  29. [41][41] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng, 'Semantic Compositional Networks for Visual Captioning,' in CVPR , 2017. -
  30. [43][43] J. Lu, C. Xiong, D. Parikh, and R. Socher, 'Knowing when to look: Adaptive attention via a visual sentinel for image captioning,' in CVPR , 2017. -
  31. [44][44] B. Dai, D. Ye, and D. Lin, 'Rethinking the form of latent states in image captioning,' in ECCV , 2018. -
  32. [45][45] D. Bahdanau, K. Cho, and Y. Bengio, 'Neural machine translation by jointly learning to align and translate,' in ICLR , 2014. -
  33. [46][46] X. Chen, L. Ma, W. Jiang, J. Yao, and W. Liu, 'Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present,' in CVPR , 2018. -
  34. [47][47] Y. Wang, Z. Lin, X. Shen, S. Cohen, and G. W. Cottrell, 'Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition,' in CVPR , 2017. -
  35. [48][48] H. Ge, Z. Yan, K. Zhang, M. Zhao, and L. Sun, 'Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style,' in ICCV , 2019. -
  36. [49][49] J. Gu, J. Cai, G. Wang, and T. Chen, 'Stack-Captioning: Coarseto-Fine Learning for Image Captioning,' in AAAI , 2018. -
  37. [50][50] Z. Yang, Y. Yuan, Y. Wu, W. W. Cohen, and R. R. Salakhutdinov, 'Review Networks for Caption Generation,' in NeurIPS , 2016. -
  38. [51][51] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T.S. Chua, 'SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning,' in CVPR , 2017. -
  39. [52][52] W. Jiang, L. Ma, Y.-G. Jiang, W. Liu, and T. Zhang, 'Recurrent Fusion Network for Image Captioning,' in ECCV , 2018. -
  40. [53][53] Y. Sugano and A. Bulling, 'Seeing with Humans: Gaze-Assisted Neural Image Captioning,' arXiv preprint arXiv:1608.05203 , 2016. - · arXiv
  41. [54][54] H. R. Tavakoli, R. Shetty, A. Borji, and J. Laaksonen, 'Paying attention to descriptions generated by image captioning models,' in ICCV , 2017. -
  42. [55][55] V. Ramanishka, A. Das, J. Zhang, and K. Saenko, 'Top-down visual saliency guided by captions,' in CVPR , 2017. -
  43. [56][56] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, 'Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention,' ACM TOMM , vol. 14, no. 2, pp. 1-21, 2018. -
  44. [57][57] S. Chen and Q. Zhao, 'Boosted attention: Leveraging human attention for image captioning,' in ECCV , 2018. -
  45. [58][58] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, 'Bottom-up and top-down attention for image captioning and visual question answering,' in CVPR , 2018. -
  46. [59][59] S. Ren, K. He, R. Girshick, and J. Sun, 'Faster R-CNN: Towards real-time object detection with region proposal networks,' in NeurIPS , 2015. -
  47. [62][62] L. Ke, W. Pei, R. Li, X. Shen, and Y.-W. Tai, 'Reflective Decoding Network for Image Captioning,' in ICCV , 2019. -
  48. [63][63] Y. Qin, J. Du, Y. Zhang, and H. Lu, 'Look Back and Predict Forward in Image Captioning,' in CVPR , 2019. -
  49. [64][64] L. Huang, W. Wang, Y. Xia, and J. Chen, 'Adaptively Aligned Image Captioning via Adaptive Attention Time,' in NeurIPS , 2019. -
  50. [65][65] L. Wang, Z. Bai, Y. Zhang, and H. Lu, 'Show, Recall, and Tell: Image Captioning with Recall Mechanism,' in AAAI , 2020. -
  51. [66][66] Z.-J. Zha, D. Liu, H. Zhang, Y. Zhang, and F. Wu, 'Context-aware visual policy network for fine-grained image captioning,' IEEE Trans. PAMI , 2019. -
  52. [67][67] M. Pedersoli, T. Lucas, C. Schmid, and J. Verbeek, 'Areas of Attention for Image Captioning,' in ICCV , 2017. -
  53. [68][68] T. Yao, Y. Pan, Y. Li, and T. Mei, 'Exploring Visual Relationship for Image Captioning,' in ECCV , 2018. -
  54. [69][69] L. Guo, J. Liu, J. Tang, J. Li, W. Luo, and H. Lu, 'Aligning linguistic words and visual semantic units for image captioning,' in ACM Multimedia , 2019. -
  55. [70][70] T. N. Kipf and M. Welling, 'Semi-supervised classification with graph convolutional networks,' in ICLR , 2017. -
  56. [71][71] X. Yang, K. Tang, H. Zhang, and J. Cai, 'Auto-Encoding Scene Graphs for Image Captioning,' in CVPR , 2019. -
  57. [72][72] Z. Shi, X. Zhou, X. Qiu, and X. Zhu, 'Improving Image Captioning with Better Use of Captions,' in ACL , 2020. -
  58. [73][73] T. Yao, Y. Pan, Y. Li, and T. Mei, 'Hierarchy Parsing for Image Captioning,' in ICCV , 2019. -
  59. [74][74] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' in NeurIPS , 2017. -
  60. [75][75] X. Yang, H. Zhang, and J. Cai, 'Learning to Collocate Neural Modules for Image Captioning,' in ICCV , 2019. -
  61. [76][76] G. Li, L. Zhu, P. Liu, and Y. Yang, 'Entangled Transformer for Image Captioning,' in ICCV , 2019. -
  62. [77][77] S. Herdade, A. Kappeler, K. Boakye, and J. Soares, 'Image Captioning: Transforming Objects into Words,' in NeurIPS , 2019. -
  63. [78][78] L. Guo, J. Liu, X. Zhu, P. Yao, S. Lu, and H. Lu, 'Normalized and Geometry-Aware Self-Attention Network for Image Captioning,' in CVPR , 2020. -
  64. [81][81] M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara, 'MeshedMemory Transformer for Image Captioning,' in CVPR , 2020. -
  65. [83][83] F. Liu, X. Ren, X. Wu, S. Ge, W. Fan, Y. Zou, and X. Sun, 'Prophet Attention: Predicting Attention with Future Attention,' in NeurIPS , 2020. -
  66. [84][84] M. Cornia, L. Baraldi, and R. Cucchiara, 'SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability,' in ICRA , 2020. -
  67. [85][85] J. Ji, Y. Luo, X. Sun, F. Chen, G. Luo, Y. Wu, Y. Gao, and R. Ji, 'Improving Image Captioning by Leveraging Intra- and Interlayer Global Representation in Transformer Network,' in AAAI , 2021. -
  68. [86][86] Y. Luo, J. Ji, X. Sun, L. Cao, Y. Wu, F. Huang, C.-W. Lin, and R. Ji, 'Dual-Level Collaborative Transformer for Image Captioning,' in AAAI , 2021. -
  69. [87][87] F. Liu, Y. Liu, X. Ren, X. He, and X. Sun, 'Aligning visual regions and textual concepts for semantic-grounded image representations,' in NeurIPS , 2019. -
  70. [88][88] H. Jiang, I. Misra, M. Rohrbach, E. Learned-Miller, and X. Chen, 'In defense of grid features for visual question answering,' in CVPR , 2020. -
  71. [89][89] X. Zhang, X. Sun, Y. Luo, J. Ji, Y. Zhou, Y. Wu, F. Huang, and R. Ji, 'RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words,' in CVPR , 2021. -
  72. [91][91] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J´ egou, 'Training data-efficient image transformers & distillation through attention,' in ICML , 2021. -
  73. [92][92] W. Liu, S. Chen, L. Guo, X. Zhu, and J. Liu, 'CPTR: Full Transformer Network for Image Captioning,' arXiv preprint arXiv:2101.10804 , 2021. - · arXiv
  74. [93][93] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, 'Learning Transferable Visual Models From Natural Language Supervision,' arXiv preprint arXiv:2103.00020 , 2021. - · arXiv
  75. [94][94] Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao, 'SimVLM: Simple visual language model pretraining with weak supervision,' arXiv preprint arXiv:2108.10904 , 2021. - · arXiv
  76. [95][95] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer, 'How Much Can CLIP Benefit Visionand-Language Tasks?' arXiv preprint arXiv:2107.06383 , 2021. - · arXiv
  77. [96][96] R. Mokady, A. Hertz, and A. H. Bermano, 'ClipCap: CLIP Prefix for Image Captioning,' arXiv preprint arXiv:2111.09734 , 2021. - · arXiv
  78. [97][97] M. Cornia, L. Baraldi, G. Fiameni, and R. Cucchiara, 'Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation,' arXiv preprint arXiv:2111.12727 , 2021. - · arXiv
  79. [98][98] H. Tan and M. Bansal, 'Lxmert: Learning cross-modality encoder representations from transformers,' EMNLP , 2019. -
  80. [99][99] J. Lu, D. Batra, D. Parikh, and S. Lee, 'Vilbert: Pretraining taskagnostic visiolinguistic representations for vision-and-language tasks,' in NeurIPS , 2019. -
  81. [100][100] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei et al. , 'Oscar: Object-semantics aligned pretraining for vision-language tasks,' in ECCV , 2020. -
  82. [101][101] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. J. Corso, and J. Gao, 'Unified Vision-Language Pre-Training for Image Captioning and VQA,' in AAAI , 2020. -
  83. [103][103] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, 'VinVL: Revisiting visual representations in visionlanguage models,' in CVPR , 2021. -
  84. [104][104] X. Hu, Z. Gan, J. Wang, Z. Yang, Z. Liu, Y. Lu, and L. Wang, 'Scaling Up Vision-Language Pre-training for Image Captioning,' arXiv preprint arXiv:2111.12233 , 2021. - · arXiv
  85. [105][105] S. Hochreiter and J. Schmidhuber, 'Long Short-Term Memory,' Neural Computation , vol. 9, no. 8, pp. 1735-1780, 1997. -
  86. [106][106] J. Lu, J. Yang, D. Batra, and D. Parikh, 'Neural Baby Talk,' in CVPR , 2018. -
  87. [107][107] X. Zhu, W. Wang, L. Guo, and J. Liu, 'AutoCaption: Image Captioning with Neural Architecture Search,' arXiv preprint arXiv:2012.09742 , 2020. - · arXiv
  88. [108][108] J. Aneja, A. Deshpande, and A. G. Schwing, 'Convolutional image captioning,' in CVPR , 2018. -
  89. [110][110] Z.-c. Fei, 'Fast Image Caption Generation with Position Alignment,' AAAI Workshops , 2019. -
  90. [111][111] L. Guo, J. Liu, X. Zhu, X. He, J. Jiang, and H. Lu, 'Nonautoregressive image captioning with counterfactuals-critical multi-agent learning,' IJCAI , 2020. -
  91. [112][112] Z. Fei, 'Iterative Back Modification for Faster Image Captioning,' in ACM Multimedia , 2020. -
  92. [113][113] L. Guo, J. Liu, X. Zhu, and H. Lu, 'Fast Sequence Generation with Multi-Agent Reinforcement Learning,' arXiv preprint arXiv:2101.09698 , 2021. - · arXiv
  93. [114][114] P. Koehn, Statistical Machine Translation . Cambridge University Press, 2009. -
  94. [115][115] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba, 'Sequence level training with recurrent neural networks,' in ICLR , 2016. -
  95. [118][118] C.-Y. Lin, 'Rouge: A package for automatic evaluation of summaries,' in ACL Workshops , 2004. -
  96. [119][119] Z. Ren, X. Wang, N. Zhang, X. Lv, and L.-J. Li, 'Deep reinforcement learning-based image captioning with embedding reward,' in CVPR , 2017. -
  97. [121][121] P. Anderson, B. Fernando, M. Johnson, and S. Gould, 'SPICE: Semantic Propositional Image Caption Evaluation,' in ECCV , 2016. -
  98. [122][122] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, 'CIDEr: Consensus-based Image Description Evaluation,' in CVPR , 2015. -
  99. [123][123] L. Zhang, F. Sung, F. Liu, T. Xiang, S. Gong, Y. Yang, and T. M. Hospedales, 'Actor-Critic Sequence Training for Image Captioning,' in NeurIPS , 2017. -
  100. [124][124] J. Gao, S. Wang, S. Wang, S. Ma, and W. Gao, 'Self-critical n-step Training for Image Captioning,' in CVPR , 2019. -
  101. [125][125] Q. Xia, H. Huang, N. Duan, D. Zhang, L. Ji, Z. Sui, E. Cui, T. Bharti, and M. Zhou, 'XGPT: Cross-modal Generative PreTraining for Image Captioning,' arXiv preprint arXiv:2003.01473 , 2020. - · arXiv
  102. [126][126] K. Desai and J. Johnson, 'VirTex: Learning Visual Representations From Textual Annotations,' in CVPR , 2021. -
  103. [127][127] R. Sennrich, B. Haddow, and A. Birch, 'Neural Machine Translation of Rare Words with Subword Units,' in ACL , 2016. -
  104. [128][128] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft COCO: Common Objects in Context,' in ECCV , 2014. -
  105. [129][129] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, 'From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,' TACL , 2014. -
  106. [130][130] P. Sharma, N. Ding, S. Goodman, and R. Soricut, 'Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,' in ACL , 2018. -
  107. [131][131] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, 'Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts,' in CVPR , 2021. -
  108. [132][132] D. Gurari, Y. Zhao, M. Zhang, and N. Bhattacharya, 'Captioning Images Taken by People Who Are Blind,' in ECCV , 2020. -
  109. [133][133] S. Reed, Z. Akata, H. Lee, and B. Schiele, 'Learning deep representations of fine-grained visual descriptions,' in CVPR , 2016. -
  110. [134][134] X. Yang, H. Zhang, D. Jin, Y. Liu, C.-H. Wu, J. Tan, D. Xie, J. Wang, and X. Wang, 'Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards,' in ECCV , 2020. -
  111. [135][135] A. Ramisa, F. Yan, F. Moreno-Noguer, and K. Mikolajczyk, 'BreakingNews: Article Annotation by Image and Text Processing,' IEEE Trans. PAMI , vol. 40, no. 5, pp. 1072-1085, 2017. -
  112. [137][137] O. Sidorov, R. Hu, M. Rohrbach, and A. Singh, 'TextCaps: a Dataset for Image Captioning with Reading Comprehension,' in ECCV , 2020. -
  113. [138][138] J. Pont-Tuset, J. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari, 'Connecting vision and language with localized narratives,' in ECCV , 2020. -
  114. [139][139] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, 'Zero-Shot Text-to-Image Generation,' arXiv preprint arXiv:2102.12092 , 2021. - · arXiv
  115. [140][140] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, 'YFCC100M: The new data in multimedia research,' Communications of the ACM , vol. 59, no. 2, pp. 64-73, 2016. -
  116. [141][141] K. Srinivasan, K. Raman, J. Chen, M. Bendersky, and M. Najork, 'WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning,' arXiv preprint arXiv:2103.01913 , 2021. - · arXiv
  117. [145][145] T.-H. Chen, Y.-H. Liao, C.-Y. Chuang, W.-T. Hsu, J. Fu, and M. Sun, 'Show, adapt and tell: Adversarial training of crossdomain image captioner,' in ICCV , 2017. -
  118. [146][146] L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell, 'Generating visual explanations,' in ECCV , 2016. -
  119. [148][148] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, 'Generative adversarial text to image synthesis,' in ICML , 2016. -
  120. [149][149] J. Kasai, K. Sakaguchi, L. Dunagan, J. Morrison, R. L. Bras, Y. Choi, and N. A. Smith, 'Transparent human evaluation for image captioning,' arXiv preprint arXiv:2111.08940 , 2021. - · arXiv
  121. [150][150] S. Banerjee and A. Lavie, 'METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,' in ACL Workshops , 2005. -
  122. [151][151] N. Sharif, L. White, M. Bennamoun, and S. A. A. Shah, 'NNEval: Neural network based evaluation metric for image captioning,' in ECCV , 2018. -
  123. [152][152] Y. Cui, G. Yang, A. Veit, X. Huang, and S. Belongie, 'Learning to evaluate image captioning,' in CVPR , 2018. -
  124. [153][153] O. Caglayan, P. Madhyastha, and L. Specia, 'Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale,' in COLING , 2020. -
  125. [154][154] R. Shetty, M. Rohrbach, L. Anne Hendricks, M. Fritz, and B. Schiele, 'Speaking the same language: Matching machine to human captions by adversarial training,' in ICCV , 2017. -
  126. [155][155] E. Van Miltenburg, D. Elliott, and P. Vossen, 'Measuring the diversity of automatic image descriptions,' in COLING , 2018. -
  127. [156][156] Q. Wang and A. B. Chan, 'Describing like humans: on diversity in image captioning,' in CVPR , 2019. -
  128. [157][157] Q. Wang, J. Wan, and A. B. Chan, 'On Diversity in Image Captioning: Metrics and Methods,' IEEE Trans. PAMI , 2020. -
  129. [159][159] M. Jiang, J. Hu, Q. Huang, L. Zhang, J. Diesner, and J. Gao, 'REORelevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning,' in EMNLP-IJCNLP , 2019. -
  130. [160][160] Z. Wang, B. Feng, K. Narasimhan, and O. Russakovsky, 'Towards unique and informative captioning of images,' in ECCV , 2020. -
  131. [161][161] M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger, 'From word embeddings to document distances,' in ICML , 2015. -
  132. [162][162] M. Kilickaya, A. Erdem, N. Ikizler-Cinbis, and E. Erdem, 'Reevaluating automatic metrics for image captioning,' in ACL , 2017. -
  133. [163][163] M. Cornia, L. Baraldi, and R. Cucchiara, 'Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions,' in CVPR , 2019. -
  134. [165][165] H. Lee, S. Yoon, F. Dernoncourt, D. S. Kim, T. Bui, and K. Jung, 'ViLBERTScore: Evaluating Image Caption Using Vision-andLanguage BERT,' in EMNLP Workshops , 2020. -
  135. [166][166] Y. Yi, H. Deng, and J. Hu, 'Improving image captioning evaluation by considering inter references variance,' in ACL , 2020. -
  136. [167][167] S. Wang, Z. Yao, R. Wang, Z. Wu, and X. Chen, 'FAIEr: Fidelity and Adequacy Ensured Image Caption Evaluation,' in CVPR , 2021. -
  137. [168][168] H. Lee, S. Yoon, F. Dernoncourt, T. Bui, and K. Jung, 'UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning,' in ACL , 2021. -
  138. [169][169] B. Dai, S. Fidler, and D. Lin, 'A neural compositional paradigm for image captioning,' in NeurIPS , 2018. -
  139. [170][170] B. Dai, S. Fidler, R. Urtasun, and D. Lin, 'Towards Diverse and Natural Image Descriptions via a Conditional GAN,' in ICCV , 2017. -
  140. [172][172] I. J. Unanue, J. Parnell, and M. Piccardi, 'BERTTune: Fine-Tuning Neural Machine Translation with BERTScore,' ACL , 2021. -
  141. [173][173] M. Jiang, Q. Huang, L. Zhang, X. Wang, P. Zhang, Z. Gan, J. Diesner, and J. Gao, 'TIGEr: text-to-image grounding for image caption evaluation,' in ACL , 2019. -
  142. [175][175] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi, 'CLIPScore: A Reference-free Evaluation Metric for Image Captioning,' arXiv preprint arXiv:2104.08718 , 2021. - · arXiv
  143. [176][176] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, 'Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data,' in CVPR , 2016. -
  144. [177][177] S. Venugopalan, L. A. Hendricks, M. Rohrbach, R. Mooney, T. Darrell, and K. Saenko, 'Captioning Images with Diverse Objects,' in CVPR , 2017. -
  145. [178][178] H. Agrawal, K. Desai, X. Chen, R. Jain, D. Batra, D. Parikh, S. Lee, and P. Anderson, 'nocaps: novel object captioning at scale,' in ICCV , 2019. -
  146. [179][179] T. Yao, Y. Pan, Y. Li, and T. Mei, 'Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects,' in CVPR , 2017. -
  147. [180][180] Y. Li, T. Yao, Y. Pan, H. Chao, and T. Mei, 'Pointing Novel Objects in Image Captioning,' in CVPR , 2019. -
  148. [181][181] Y. Wu, L. Zhu, L. Jiang, and Y. Yang, 'Decoupled Novel Object Captioner,' in ACM Multimedia , 2018. -
  149. [182][182] P. Anderson, B. Fernando, M. Johnson, and S. Gould, 'Guided Open Vocabulary Image Captioning with Constrained Beam Search,' in EMNLP , 2017. -
  150. [183][183] X. Hu, X. Yin, K. Lin, L. Wang, L. Zhang, J. Gao, and Z. Liu, 'VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning,' AAAI , 2020. -
  151. [184][184] J. Gu, S. Joty, J. Cai, and G. Wang, 'Unpaired image captioning by language pivoting,' in ECCV , 2018. -
  152. [185][185] Y. Feng, L. Ma, W. Liu, and J. Luo, 'Unsupervised image captioning,' in CVPR , 2019. -
  153. [187][187] J. Gu, S. Joty, J. Cai, H. Zhao, X. Yang, and G. Wang, 'Unpaired image captioning via scene graph alignments,' in ICCV , 2019. -
  154. [188][188] D. Guo, Y. Wang, P. Song, and M. Wang, 'Recurrent relational memory network for unsupervised image captioning,' IJCAI , 2020. -
  155. [189][189] D.-J. Kim, J. Choi, T.-H. Oh, and I. S. Kweon, 'Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach,' in EMNLP , 2019. -
  156. [190][190] H. Ben, Y. Pan, Y. Li, T. Yao, R. Hong, M. Wang, and T. Mei, 'Unpaired Image Captioning with Semantic-Constrained SelfLearning,' IEEE Trans. Multimedia , 2021. -
  157. [191][191] R. Del Chiaro, B. Twardowski, A. D. Bagdanov, and J. van de Weijer, 'RATT: Recurrent Attention to Transient Tasks for Continual Image Captioning,' in NeurIPS , 2020. -
  158. [192][192] J. Johnson, A. Karpathy, and L. Fei-Fei, 'DenseCap: Fully convolutional Localization Networks for Dense Captioning,' in CVPR , 2016. -
  159. [193][193] L. Yang, K. Tang, J. Yang, and L.-J. Li, 'Dense captioning with joint inference and visual context,' in CVPR , 2017. -
  160. [194][194] X. Li, S. Jiang, and J. Han, 'Learning object context for dense captioning,' in AAAI , 2019. -
  161. [195][195] G. Yin, L. Sheng, B. Liu, N. Yu, X. Wang, and J. Shao, 'Context and attribute grounded dense captioning,' in CVPR , 2019. -
  162. [196][196] D.-J. Kim, J. Choi, T.-H. Oh, and I. S. Kweon, 'Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning,' in CVPR , 2019. -
  163. [197][197] J. Krause, J. Johnson, R. Krishna, and L. Fei-Fei, 'A hierarchical approach for generating descriptive image paragraphs,' in CVPR , 2017. -
  164. [198][198] X. Liang, Z. Hu, H. Zhang, C. Gan, and E. P. Xing, 'Recurrent topic-transition GAN for visual paragraph generation,' in ICCV , 2017. -
  165. [199][199] Y. Mao, C. Zhou, X. Wang, and R. Li, 'Show and Tell More: TopicOriented Multi-Sentence Image Captioning,' in IJCAI , 2018. -
  166. [200][200] M. Chatterjee and A. G. Schwing, 'Diverse and coherent paragraph generation from images,' in ECCV , 2018. -
  167. [201][201] Y. Luo, Z. Huang, Z. Zhang, Z. Wang, J. Li, and Y. Yang, 'Curiosity-driven reinforcement learning for diverse visual paragraph generation,' in ACM Multimedia , 2019. -
  168. [202][202] Z. Yang, Y. Lu, J. Wang, X. Yin, D. Florencio, L. Wang, C. Zhang, L. Zhang, and J. Luo, 'TAP: Text-Aware Pre-training for TextVQA and Text-Caption,' in CVPR , 2021. -
  169. [203][203] J. Wang, J. Tang, and J. Luo, 'Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning,' in ACM Multimedia , 2020. -
  170. [204][204] J. Wang, J. Tang, M. Yang, X. Bai, and J. Luo, 'Improving OCRbased Image Captioning by Incorporating Geometrical Relationship,' in CVPR , 2021. -
  171. [205][205] Q. Zhu, C. Gao, P. Wang, and Q. Wu, 'Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps,' in AAAI , 2021. -
  172. [206][206] G. Xu, S. Niu, M. Tan, Y. Luo, Q. Du, and Q. Wu, 'Towards Accurate Text-based Image Captioning with Content Diversity Exploration,' in CVPR , 2021. -
  173. [207][207] H. Jhamtani and T. Berg-Kirkpatrick, 'Learning to describe differences between pairs of similar images,' in EMNLP , 2018. -
  174. [208][208] D. H. Park, T. Darrell, and A. Rohrbach, 'Robust Change Captioning,' in CVPR , 2019. -
  175. [209][209] X. Shi, X. Yang, J. Gu, S. Joty, and J. Cai, 'Finding It at Another Side: A Viewpoint-Adapted Matching Encoder for Change Captioning,' in ECCV , 2020. -
  176. [210][210] Q. Huang, Y. Liang, J. Wei, C. Yi, H. Liang, H.-f. Leung, and Q. Li, 'Image Difference Captioning with Instance-Level Fine-Grained Feature Representation,' IEEE Trans. Multimedia , 2021. -
  177. [211][211] H. Kim, J. Kim, H. Lee, H. Park, and G. Kim, 'ViewpointAgnostic Change Captioning With Cycle Consistency,' in ICCV , 2021. -
  178. [212][212] M. Hosseinzadeh and Y. Wang, 'Image Change Captioning by Learning from an Auxiliary Task,' in CVPR , 2021. -
  179. [213][213] A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. J. Crandall, and D. Batra, 'Diverse Beam Search for Improved Description of Complex Scenes,' in AAAI , 2018. -
  180. [214][214] B. Dai and D. Lin, 'Contrastive learning for image captioning,' in NeurIPS , 2017. -
  181. [215][215] L. Liu, J. Tang, X. Wan, and Z. Guo, 'Generating diverse and descriptive image captions using visual paraphrases,' in ICCV , 2019. -
  182. [216][216] L. Wang, A. G. Schwing, and S. Lazebnik, 'Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space,' in NeurIPS , 2017. -
  183. [217][217] J. Aneja, H. Agrawal, D. Batra, and A. Schwing, 'Sequential latent spaces for modeling the intention during diverse image captioning,' in ICCV , 2019. -
  184. [218][218] F. Chen, R. Ji, J. Ji, X. Sun, B. Zhang, X. Ge, Y. Wu, F. Huang, and Y. Wang, 'Variational structured semantic inference for diverse image captioning,' in NeurIPS , 2019. -
  185. [219][219] S. Mahajan and S. Roth, 'Diverse image captioning with contextobject split latent spaces,' in NeurIPS , 2020. -
  186. [220][220] A. Deshpande, J. Aneja, L. Wang, A. G. Schwing, and D. Forsyth, 'Fast, diverse and accurate image captioning guided by part-ofspeech,' in CVPR , 2019. -
  187. [221][221] D. Elliott, S. Frank, and E. Hasler, 'Multilingual image description with neural sequence models,' ICLR , 2015. -
  188. [222][222] X. Li, C. Xu, X. Wang, W. Lan, Z. Jia, G. Yang, and J. Xu, 'COCOCN for cross-lingual image tagging, captioning, and retrieval,' IEEE Trans. Multimedia , vol. 21, no. 9, pp. 2347-2360, 2019. -
  189. [223][223] T. Miyazaki and N. Shimizu, 'Cross-lingual image caption generation,' in ACM Multimedia , 2016. -
  190. [224][224] D. Elliott, S. Frank, K. Sima'an, and L. Specia, 'Multi30K: Multilingual English-German Image Descriptions,' in ACL Workshops , 2016. -
  191. [225][225] W. Lan, X. Li, and J. Dong, 'Fluency-guided cross-lingual image captioning,' in ACM Multimedia , 2017.
  192. [226][226] Y. Song, S. Chen, Y. Zhao, and Q. Jin, 'Unpaired cross-lingual image caption generation with self-supervised rewards,' in ACM Multimedia , 2019.
  193. [228][228] F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou, 'Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation,' in CVPR , 2021.
  194. [230][230] Z. Bai, Y. Nakashima, and N. Garcia, 'Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation,' in ICCV , 2021.
  195. [231][231] Y. Feng and M. Lapata, 'Automatic Caption Generation for News Images,' IEEE Trans. PAMI , vol. 35, no. 4, pp. 797-812, 2012.
  196. [232][232] A. Tran, A. Mathews, and L. Xie, 'Transform and Tell: EntityAware News Image Captioning,' in CVPR , 2020.
  197. [233][233] F. Liu, Y. Wang, T. Wang, and V. Ordonez, 'Visual News: Benchmark and Challenges in News Image Captioning,' in EMNLP , 2021.
  198. [234][234] X. Yang, S. Karaman, J. Tetreault, and A. Jaimes, 'Journalistic Guidelines Aware News Image Captioning,' arXiv preprint arXiv:2109.02865 , 2021. · arXiv
  199. [235][235] S. Wu, J. Wieland, O. Farivar, and J. Schiller, 'Automatic alttext: Computer-generated image descriptions for blind users on a social network service,' in CSCW , 2017.
  200. [236][236] C. Chunseong Park, B. Kim, and G. Kim, 'Attend to you: Personalized image captioning with context sequence memory networks,' in CVPR , 2017.

200 of 219 shown

Authors
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R.
Affiliations
  • University of Modena and Reggio Emilia (IT)
  • NVIDIA (Italy) (IT)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023-01-01
Volume
45
Issue
1
Pages
539-559
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2022.3148210
OpenAlex
W3216130706
Cited by
373
References
268
Citations by year
2026: 41 · 2025: 110 · 2024: 113 · 2023: 89 · 2022: 20
FWCI
7.04
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • IRIS UNIMORE (University of Modena and Reggio Emilia) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • European Commission (951847, 952026)
  • Fondazione di Modena
  • Italian Ministry of Foreign Affairs and International Cooperation
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
image captioning, vision-language models, vision encoder, text generation, object properties, attention mechanism, training strategies, evaluation metrics, open problems
MeSH terms
Deep Learning, Algorithms, Language, Natural Language Processing, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review