Good News, Everyone! Context Driven Entity-Aware Captioning for News Images
Open accessAbstract
Current image captioning systems perform at a merely descriptive level, essentially enumerating the objects in the scene and their relations. Humans, on the contrary, interpret images by integrating several sources of prior knowledge of the world. In this work, we aim to take a step closer to producing captions that offer a plausible interpretation of the scene, by integrating such contextual information into the captioning pipeline. For this we focus on the captioning of images used to illustrate news articles. We propose a novel captioning method that is able to leverage contextual information provided by the text of news articles associated with an image. Our model is able to selectively draw information from the article guided by visual cues, and to dynamically extend the output dictionary to out-of-vocabulary named entities that appear in the context source. Furthermore we introduce ``GoodNews'', the largest news image captioning dataset in the literature and demonstrate state-of-the-art results.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Biten and colleagues combined visual and textual data to generate captions in two stages: template caption generation and entity insertion.
“Most recently, Biten et al. [52] combined both visual and textual data to generate captions across two stages: template caption generation stage and entity insertion stage.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
(2004) · Cited by 9,718 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,682 · see citing works
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550 · see citing works
A survey of named entity recognition and classification
Nadeau, D., & Sekine, S. (2007) · Lingvisticae Investigationes · doi:10.1075/li.30.1.03nad · Cited by 2,507 · see citing works
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.345 · Cited by 1,724 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
(2017) · International Conference on Learning Representations · Cited by 1,075 · see citing works
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · see citing works
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2587640 · Cited by 875 · see citing works
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, B., Çakıcı, R., Elliott, D., Erdem, A., Erdem, E., İkizler-Cinbiş, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 350 · see citing works
StyleNet: Generating Attractive Visual Captions with Styles
Gan, C., Gan, Z., He, X., Gao, J., & Deng, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.108 · Cited by 287 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and VQA.Preprint
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2017) · arXiv (Cornell University) · Cited by 252 · see citing works
Terman, L. M. (2008) · Medical Entomology and Zoology · Cited by 249 · see citing works
A survey on automatic image caption generation
Bai, S., & An, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.080 · Cited by 243 · see citing works
SentiCap: Generating Image Descriptions with Sentiments
Mathews, A., Xie, L., & He, X. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10475 · Cited by 215 · see citing works
Deep Learning and Convolutional Neural Networks for Medical Image Computing
Lu, L., Zheng, Y., Carneiro, G., & Yang, L. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-42999-1 · Cited by 208 · see citing works
Language Models for Image Captioning: The Quirks and What Works
Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015) · doi:10.3115/v1/p15-2017 · Cited by 207 · see citing works
Exploring Nearest Neighbor Approaches for Image CaptioningPreprint
Devlin, J., Gupta, S., Girshick, R., Mitchell, M., & Zitnick, C. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04467 · Cited by 148 · see citing works
Learning to Evaluate Image Captioning
Cui, Y., Yang, G., Veit, A., Huang, X., & Belongie, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00608 · Cited by 140 · see citing works
Comparing Automatic Evaluation Measures for Image Description
Elliott, D., & Keller, F. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2074 · Cited by 137 · see citing works
SemStyle: Learning to Generate Stylised Image Captions Using Unaligned Text
Mathews, A., Xie, L., & He, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00896 · Cited by 127 · see citing works
Automatic Caption Generation for News Images
Feng, Y., & Lapata, M. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.118 · Cited by 117 · see citing works
Rich Image Captioning in the Wild
Tran, K., He, X., Zhang, L., & Sun, J. Z. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.61 · Cited by 110 · see citing works
Language Models for Image Captioning: The Quirks and What WorksPreprint
Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.01809 · Cited by 98 · see citing works
BreakingNews: Article Annotation by Image and Text Processing
Arnau Ramisa, A., Yan, F., Moreno-Noguer, F., & Mikolajczyk, K. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2721945 · Cited by 94 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question AnsweringPreprint
Anderson, P., He, X., Buehler, C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07998 · Cited by 94 · see citing works
Entity-aware Image Caption Generation
Lu, D., Whitehead, S., Huang, L., Ji, H., & Chang, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1435 · Cited by 67 · see citing works
A Context-Driven Extractive Framework for Generating Realistic Image Descriptions
Tariq, A., & Foroosh, H. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2628585 · Cited by 46 · see citing works
Rich Image Captioning in the WildPreprint
Tran, K., He, X., Zhang, L., Sun, J., Carapcea, C., Thrasher, C., Buehler, C., & Sienkiewicz, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.09016 · Cited by 34 · see citing works
Re-evaluating Automatic Metrics for Image CaptioningPreprint
Kilickaya, M., Erdem, A., İkizler-Cinbiş, N., & Erdem, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07600 · Cited by 14 · see citing works
Entity-aware Image Caption GenerationPreprint
Lu, D., Whitehead, S., Huang, L., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07889 · Cited by 13 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373 · see citing works
Liu, H., Zheng, C., Li, D., Shen, X., Lin, K., Wang, J., Zhang, Z., Zhang, Z., & Xiong, N. N. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3128240 · Cited by 260 · see citing works
Das, S., Tariq, A., Santos, T., Kantareddy, S. S., & Banerjee, I. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_4 · Cited by 195 · see citing works
Transform and Tell: Entity-Aware News Image Captioning
Tran, A., Mathews, A., & Xie, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01305 · Cited by 91 · see citing works
Retrieval Topic Recurrent Memory Network for Remote Sensing Image Captioning
Wang, B., Zheng, X., Qu, B., & Lu, X. (2020) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2959208 · Cited by 79 · see citing works
Visual News: Benchmark and Challenges in News Image Captioning
Liu, F., Wang, Y., Wang, T., & Ordóñez, V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.542 · Cited by 78 · see citing works
Visuals to Text: A Comprehensive Review on Automatic Image Captioning
Ming, Y., Hu, N., Fan, C., Feng, F., Zhou, J., & Yu, H. L. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105734 · Cited by 80 · see citing works
Boosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph
Zhao, W., & Wu, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301279 · Cited by 73 · see citing works
Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Manzoor, M. A., Albarri, S., Xian, Z., Meng, Z., Nakov, P., & Liang, S. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3617833 · Cited by 76 · see citing works
Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News
Tan, R., Plummer, B. A., & Saenko, K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.163 · Cited by 62 · see citing works
Knowledge-aware multimodal pre-training for fake news detection
Zhang, L., Zhang, X., Zhou, Z., Zhang, X., Yu, P. S., & Li, C. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102715 · Cited by 57 · see citing works
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection
Fung, Y., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., & Sil, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.133 · Cited by 52 · see citing works
WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types
Wang, X., Tian, J., Gui, M., Li, Z., Wang, R., Yan, M., Chen, L., & Xiao, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.328 · Cited by 53 · see citing works
Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation
Bai, Z., Nakashima, Y., & García, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00537 · Cited by 48 · see citing works
Dual-CNN: A Convolutional language decoder for paragraph image captioning
Li, R., Liang, H., Shi, Y., Feng, F., & Wang, X. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.02.041 · Cited by 44 · see citing works
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning
Biten, A. F., Gomez, L., & Karatzas, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00253 · Cited by 43 · see citing works
Self-citationKnowing What it is: Semantic-Enhanced Dual Attention Transformer
Ma, Y., Ji, J., Sun, X., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3164787 · Cited by 39 · see citing works
Explain and improve: LRP-inference fine-tuning for image captioning models
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.008 · Cited by 47 · see citing works
Generating Accurate Caption Units for Figure Captioning
Qian, X., Koh, E., Du, F., Kim, S., Chan, J., Rossi, R. A., Malik, S., & Lee, T. Y. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449923 · Cited by 34 · see citing works
Transparent Human Evaluation for Image Captioning
Kasai, J., Sakaguchi, K., Dunagan, L., Morrison, J., Bras, R. J. L., Choi, Y., & Smith, N. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.254 · Cited by 34 · see citing works
Preserving Semantic Neighborhoods for Robust Cross-Modal Retrieval
Thomas, C., & Kovashka, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_19 · Cited by 32 · see citing works
Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective
Uchendu, A., Le, T., & Lee, D. (2023) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3606274.3606276 · Cited by 34 · see citing works
Context-Aware Image Descriptions for Web Accessibility
Mohanbabu, A. G., & Pavel, A. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675658 · Cited by 29 · see citing works
Journalistic Guidelines Aware News Image Captioning
Yang, X., Karaman, S., Tetreault, J., & Jaimes, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.419 · Cited by 29 · see citing works
Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities
Hu, H., Luan, Y., Chen, Y., Khandelwal, U., Joshi, M., Lee, K., Toutanova, K., & Chang, M.-W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01108 · Cited by 27 · see citing works
Visual Entity Linking via Multi-modal Learning
Zheng, Q., Wen, H., Wang, M., & Qi, G. (2021) · Data Intelligence · doi:10.1162/dint_a_00114 · Cited by 26 · see citing works
Kreiss, E., Bennett, C., Hooshmand, S., Zelikman, E., Ringel Morris, M., & Potts, C. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.309 · Cited by 24 · see citing works
ICECAP: Information Concentrated Entity-aware Image Captioning
Hu, A., Chen, S., & Jin, Q. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413576 · Cited by 22 · see citing works
Cheng, K., Song, W., Ma, Z., Zhu, W., Zhu, Z., & Zhang, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611987 · Cited by 20 · see citing works
Open-Vocabulary Object Detection With an Open Corpus
Wang, J., Zhang, H., Hong, H., Jin, X., He, Y., Xue, H., & Zhao, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00622 · Cited by 19 · see citing works
Context-Driven Image Caption With Global Semantic Relations of the Named Entities
Yun, J., Xu, Z., & Guanglai, G. (2020) · IEEE Access · doi:10.1109/access.2020.3013321 · Cited by 19 · see citing works
Concadia: Towards Image-Based Text Generation with a Purpose
Kreiss, E., Fang, F., Goodman, N. D., & Potts, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.308 · Cited by 21 · see citing works
Jamil, A., Rehman, S. U., Mahmood, K., Villar, M. G., Prola, T. A., Díez, I. D. L. T., Samad, M. A., & Ashraf, I. (2024) · IEEE Access · doi:10.1109/access.2024.3365528 · Cited by 23 · see citing works
RetractedAugmented Human and Human-Machine Co-evolution: Efficiency and Ethics
Dengel, A. R., Devillers, L. Y., & Schaal, L. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69128-8_13 · Cited by 18 · see citing works
Fine-tuning with Multi-modal Entity Prompts for News Image Captioning
Zhang, J., Fang, S., Mao, Z., Zhang, Z., & Zhang, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3547883 · Cited by 18 · see citing works
Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection
Yang, Y., Bao, R., Guo, W., Zhan, D., Yin, Y., & Yang, J. J. (2023) · Science China Information Sciences · doi:10.1007/s11432-021-3530-7 · Cited by 17 · see citing works
Fusing Pre-Trained Language Models with Multimodal Prompts through Reinforcement Learning
Yu, Y., Chung, J., Yun, H., Hessel, J., Park, J. S., Lu, X., Zellers, R., Ammanabrolu, P., Bras, R. J. L., Kim, G., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01044 · Cited by 16 · see citing works
A Strong and Robust Baseline for Text-Image Matching
Liu, F., & Ye, R. (2019) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p19-2023 · Cited by 16 · see citing works
Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTAPreprint
Fain, M., Twomey, N., Ponikar, A., Fox, R., & Bollegala, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.12763 · Cited by 18 · see citing works
Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
Shukor, M., Thome, N., & Cord, M. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104071 · Cited by 14 · see citing works
Review of Recent Deep Learning Based Methods for Image-Text Retrieval
Chen, J., Zhang, L., Bai, C., & Kpalma, K. (2020) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr49039.2020.00042 · Cited by 15 · see citing works
Focus! Relevant and Sufficient Context Selection for News Image Captioning
Zhou, M., Luo, G., Rohrbach, A., & Zhou, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.450 · Cited by 13 · see citing works
SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning
Yang, Z., Dabre, R., Tanaka, H., & Okazaki, N. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1140 · Cited by 12 · see citing works
Enhancing Neural Machine Translation With Dual-Side Multimodal Awareness
Song, Y., Chen, S., Jin, Q., Luo, W., Xie, J., & Huang, F. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3092187 · Cited by 12 · see citing works
Zhang, Z., Zhang, H., Wang, J., Sun, Z., & Yang, Z. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108429 · Cited by 12 · see citing works
VisualNews : Benchmark and Challenges in Entity-aware Image Captioning.Preprint
Liu, F., Wang, Y., Wang, T., & Ordóñez, V. (2020) · arXiv (Cornell University) · Cited by 11 · see citing works
News Image-Text Matching With News Knowledge Graph
Zhao, Y., Yun, J., Gao, S., & Liu, L. (2021) · IEEE Access · doi:10.1109/access.2021.3093650 · Cited by 10 · see citing works
Visual Named Entity Linking: A New Dataset and A Baseline
Sun, W., Fan, Y., Guo, J., Zhang, R., & Cheng, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.178 · Cited by 10 · see citing works
Image Caption Generation for News Articles
Yang, Z., & Okazaki, N. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.176 · Cited by 10 · see citing works
Vision to Language: Methods, Metrics and Datasets
Sharif, N., Nadeem, U., Shah, S. A. A., Bennamoun, M., & Liu, W. (2020) · Learning and Analytics in Intelligent Systems · doi:10.1007/978-3-030-49724-8_2 · Cited by 9 · see citing works
EAES: Effective Augmented Embedding Spaces for Text-Based Image Captioning
Nguyen, K. T. T. M., Bui, D. C., Trinh, T., & Vo, N. D. (2022) · IEEE Access · doi:10.1109/access.2022.3158763 · Cited by 9 · see citing works
Spatial-aware topic-driven-based image Chinese caption for disaster news
Zhou, J., Zhu, Y., Zhang, Y., Yang, C., & Pan, H. (2023) · Neural Computing and Applications · doi:10.1007/s00521-022-08072-w · Cited by 8 · see citing works
Kalarani, A. R., Bhattacharyya, P., Chhaya, N., & Shekhar, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.67 · Cited by 8 · see citing works
Show, Interpret and Tell: Entity-Aware Contextualised Image Captioning in Wikipedia
Nguyen, K., Biten, A. F., Mafla, A., Gómez, L., & Karatzas, D. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25285 · Cited by 8 · see citing works
Self-citationDealing with Semantic Underspecification in Multimodal NLP
Pezzelle, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.675 · Cited by 8 · see citing works
G3: Geolocation via Guidebook Grounding
Luo, G., Biamby, G., Darrell, T., Fried, D., & Rohrbach, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.430 · Cited by 9 · see citing works
Understanding Image Captioning Models beyond Visualizing AttentionPreprint
Sun, J., Lapuschkin, S., Samek, W., & Binder, A. (2020) · arXiv (Cornell University) · Cited by 8 · see citing works
Transform, contrast and tell: Coherent entity-aware multi-image captioning
Chen, J. (2023) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2023.103878 · Cited by 7 · see citing works
A comprehensive survey on deep-learning-based visual captioning
Xin, B., Xu, N., Zhai, Y., Zhang, T., Lu, Z., Liu, J., Nie, W., Li, X., & Liu, A.-A. (2023) · Multimedia Systems · doi:10.1007/s00530-023-01175-x · Cited by 7 · see citing works
Deep Cross-Modal Retrieval Between Spatial Image and Acoustic Speech
Qian, X., Xue, W., Zhang, Q., Tao, R., & Li, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3323876 · Cited by 7 · see citing works
CapOnImage: Context-driven Dense-Captioning on Image
Gao, Y., Hou, X., Zhang, Y., Ge, T., Jiang, Y., & Wang, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.226 · Cited by 7 · see citing works
Tell as You Imagine: Sentence Imageability-Aware Image Captioning
Umemura, K., Kastner, M. A., Ide, I., Kawanishi, Y., Hirayama, T., Doman, K., Deguchi, D., & Murase, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67835-7_6 · Cited by 7 · see citing works
Vision and Structured-Language Pretraining for Cross-Modal Food RetrievalPreprint
Shukor, M., Thome, N., & Cord, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4511116 · Cited by 7 · see citing works
Exploring the Impact of Vision Features in News Image Captioning
Zhang, J., & Wan, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.818 · Cited by 6 · see citing works
An Attentive Fourier-Augmented Image-Captioning Transformer
Raymond, O. I., Yang, Z., & Long, J. (2021) · Applied Sciences · doi:10.3390/app11188354 · Cited by 6 · see citing works
Berger, U., Stanovsky, G., Abend, O., & Frermann, L. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.52 · Cited by 5 · see citing works
Xu, N., Zhang, T., Tian, H., & Liu, A.-A. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3426655 · Cited by 5 · see citing works
A transformer-based deep learning model for evaluation of accessibility of image descriptions
Shrestha, R. (2022) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3529836.3529856 · Cited by 5 · see citing works
Emphasizing Complementary Samples for Non-literal Cross-modal Retrieval
Thomas, C., & Kovashka, A. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00509 · Cited by 5 · see citing works
Mao, S., Zhang, C., Su, H., Song, H., Shalyminov, I., & Cai, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_27 · Cited by 4 · see citing works
Sequential Vision to Language as Story: A Storytelling Dataset and Benchmarking
Malakan, Z. M., Anwar, S., Hassan, G. M., & Mian, A. S. (2023) · IEEE Access · doi:10.1109/access.2023.3293646 · Cited by 4 · see citing works
Towards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 4 · see citing works
Eswaraiah, P., & Syed, H. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1805 · Cited by 5 · see citing works
Image-Relevant Entities Knowledge-Aware News Image Captioning
Ajankar, S., & Dutta, T. (2024) · IEEE Multimedia · doi:10.1109/mmul.2024.3363429 · Cited by 4 · see citing works
Cross-Modal Coherence-Enhanced Feedback Prompting for News Captioning
Xu, N., Gao, Y., Zhang, T., Tian, H., & Liu, A.-A. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681497 · Cited by 4 · see citing works
Geo-Aware Image Caption Generation
Nikiforova, S., Deoskar, T., Paperno, D., & Winter, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.280 · Cited by 4 · see citing works
Multimodal Geolocation Estimation of News Photos
Tahmasebzadeh, G., Hakimov, S., Ewerth, R., & Müller-Budack, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_14 · Cited by 4 · see citing works
A Systematic Literature Review on Image Captioning
Ding, Y. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-36004-6_54 · Cited by 4 · see citing works
Cao, Z., Xia, J., & Zhou, M. (2025) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2025.3553260 · Cited by 3 · see citing works
Khan, A. S., Abbass, M. J., & Khan, A. (2025) · Sensors · doi:10.3390/s25195992 · Cited by 4 · see citing works
An Analysis of the Use of Feed-Forward Sub-Modules for Transformer-Based Image Captioning Tasks
Raymond, O. I., Yang, Z., & Long, J. (2021) · Applied Sciences · doi:10.3390/app112411635 · Cited by 3 · see citing works
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Burns, A., Srinivasan, K., Ainslie, J., Brown, G., Plummer, B. A., Saenko, K., Ni, J., & Guo, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.119 · Cited by 3 · see citing works
NewsStories: Illustrating Articles with Visual Summaries
Tan, R., Plummer, B. A., Saenko, K., Lewis, J. P., Sud, A., & Leung, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_37 · Cited by 4 · see citing works
News Image Captioning via Separate Attention on Entity Categories
Ajankar, S., & Dutta, T. (2025) · IEEE Access · doi:10.1109/access.2025.3582655 · Cited by 2 · see citing works
Event-Enriched Image Analysis Grand Challenge At ACM Multimedia 2025
Tran, T.-P., Nguyen, M.-Q., Tran, M., Nguyen, T., Do, T.-L., Ly, D.-N., Huynh, V.-T., Le, K.-D., Tran, M.-K., & Le, T.-N. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3762067 · Cited by 2 · see citing works
Visual Moral Inference and Communication
Zhu, W., Ramezani, A., & Xu, Y. (2025) · Topics in Cognitive Science · doi:10.1111/tops.70031 · Cited by 2 · see citing works
Multimodal Information Retrieval
Luo, M., Gokhale, T., Varshney, N., Yang, Y., & Baral, C. R. (2024) · Synthesis lectures on computer vision · doi:10.1007/978-3-031-57816-8_3 · Cited by 2 · see citing works
The Optimal Choice of the Encoder–Decoder Model Components for Image Captioning
Bartosiewicz, M., & Marcin Iwanowski, C. (2024) · Information · doi:10.3390/info15080504 · Cited by 2 · see citing works
ReCap: Event-Aware Image Captioning with Article Retrieval and Semantic Gaussian Normalization
Nguyen, T.-P., Nguyen, T.-H., Dinh, G., Nguyen, L.-H., Tran, M., & Le, T.-N. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3762039 · Cited by 2 · see citing works
Scene Change Captioning in Real Scenarios
Qiu, Y., Nakashima, K., Satoh, Y., Suzuki, R., Iwata, K., & Kataoka, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05643-7_26 · Cited by 2 · see citing works
BIG-C: a Multimodal Multi-Purpose Dataset for Bemba
Sikasote, C., Mukonde, E., Alam, M. M. I., & Anastasopoulos, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.115 · Cited by 2 · see citing works
Who You Are Decides How You Tell
Wu, S., Fan, S., Shen, Z., Kankanhalli, M., & Tung, A. K. H. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413589 · Cited by 2 · see citing works
Think as People: Context-Driven Multi-Image News Captioning with Adaptive Dual Attention
Yang, Q., Wu, X., Chen, X., Gao, X., & Zhang, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446024 · Cited by 1 · see citing works
Multimodal Geolocation Estimation in News Documents
Tahmasebzadeh, G., Müller-Budack, E., & Ewerth, R. (2024) · Springer eBooks · doi:10.1007/978-3-031-64451-1_2 · Cited by 1 · see citing works
MEKA: Multimodal External Knowledge Augmentation for Social Media Image Captioning
Dash, M., Pant, T., & Parganiha, H. (2025) · International Conference on Information and Computer Technologies (ICICT) · doi:10.1109/icict64582.2025.00081 · Cited by 1 · see citing works
EAMA: Entity-Aware Multimodal Alignment Based Approach for News Image Captioning
Zhang, J. Y., Zhang, H., Yin, X., & Wan, X. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3801548 · Cited by 1 · see citing works
News Video Description Based on Template Generation and Entity Insertion
Yuan, Q., Zhai, P., Zheng, D., & Fang, Y. K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13870-6_28 · Cited by 1 · see citing works
Improving language-supervised object detection with linguistic structure analysis
Rai, A., & Kovashka, A. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00588 · Cited by 1 · see citing works
- Authors
- Biten, A. F., Gomez, L., Rusiñol, M., & Karatzas, D.
- Affiliations
- Computer Vision Center (ES)
- Universitat Autónoma de Barcelona
- Universitat Autònoma de Barcelona (ES)
- Computer Vision Centre
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 12458-12467
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2928602793
- Cited by
- 136
- References
- 43
- Citations by year
- 2026: 5 · 2025: 17 · 2024: 16 · 2023: 36 · 2022: 23 · 2021: 21 · 2020: 16 · 2019: 2
- FWCI
- 5.05
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Fundación BBVA
- Centres de Recerca de Catalunya
- Generalitat de Catalunya
- Banco Bilbao Vizcaya Argentaria
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- image captioning, news images, contextual information, named entities, visual cues
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
