The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Open accessAbstract
Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
(2016) · Cited by 31,590
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Cited by
Attention U-Net: Learning Where to Look for the Pancreas
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.03999 · Cited by 5,458
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
“Natural language processing (NLP): There are few NLPbased subjects such as neural machine translation [508] and some other interdisciplinary subjects such as image captioning [509], [510], [511], [512], visual question answering (VQA) [513], [514], [515], [516], [517] which are closely associated with NLP.”
BackgroundLXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72970-6_3 · Cited by 2,074
Self-citationUNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,542
Self-citationStacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
Self-citationMeshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
Self-citationVisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Kim, W., Son, B., & Kim, I. (2021) · arXiv · doi:10.48550/arxiv.2102.03334 · Cited by 951
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864
Self-citationThe design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769
Kim, J.-H., Jun, J., & Zhang, B.-T. (2018) · arXiv · doi:10.48550/arxiv.1805.07932 · Cited by 761
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795
X-Linear Attention Networks for Image Captioning
Pan, Y., Yao, T., Li, Y., & Mei, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01098 · Cited by 720
Generating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693
An Attentive Survey of Attention Models
Chaudhari, S., Mithal, V., Polatkan, G., & Ramanath, R. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3465055 · Cited by 689
MirrorGAN: Learning Text-To-Image Generation by Redescription
Qiao, T., Zhang, J., Xu, D., & Tao, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00160 · Cited by 644
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundMethodImage as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511
Self-citationMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381
“We use a popular implementation of the BottomUpToDown (Anderson et al., 2018) VQA model.”
MethodCounterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification
Rao, Y., Chen, G., Lu, J., & Zhou, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00106 · Cited by 380
From Show to Tell: A Survey on Deep Learning-Based Image Captioning
Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., & Cucchiara, R. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3148210 · Cited by 373
Cross-modal Memory Networks for Radiology Report Generation
Chen, Z., Shen, Y., Song, Y., & Wan, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.459 · Cited by 369
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., & Batra, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00008 · Cited by 410
MUREL: Multimodal Relational Reasoning for Visual Question Answering
Cadene, R., Ben-younes, H., Cord, M., & Thome, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00209 · Cited by 330
Dual-level Collaborative Transformer for Image Captioning
Luo, Y., Ji, J., Sun, X., Cao, L., Wu, Y., Huang, F., Lin, C.-W., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16328 · Cited by 343
Data-driven materials research enabled by natural language processing and information extraction
Olivetti, E. A., Cole, J. M., Kim, E., Kononova, O., Ceder, G., Han, T. Y.-J., & Hiszpanski, A. M. (2020) · Applied Physics Reviews · doi:10.1063/5.0021106 · Cited by 302
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329
RSVQA: Visual Question Answering for Remote Sensing Data
Lobry, S., Marcos, D., Murray, J., & Tuia, D. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2020.2988782 · Cited by 305
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291
Multi-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 291
UniT: Multimodal Multitask Learning with a Unified Transformer
Hu, R., & Singh, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00147 · Cited by 303
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00210 · Cited by 293
Towards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 280
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., & Tao, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019127 · Cited by 281
“…shown recent success, with many approaches proposed to investigate the key components of this task, e.g. , discriminative feature representation (Anderson et al. 2018), multi-modal fusion (Kim et al. 2017a; Yu et al. 2017; Yu et al. 2018a) and visual reasoning (Nam, Ha, and Kim 2016; Lu et al.…”
BackgroundTowards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training
Hao, W., Li, C., Li, X., Carin, L., & Gao, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01315 · Cited by 264
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation
Hong, Y., Wu, Q., Qi, Y., Rodriguez-Opazo, C., & Gould, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00169 · Cited by 264
“[3]) according to the positions of objects provided in REVERIE [47].”
BackgroundMethodSelf-citationImage Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
Liu, Z., Rodriguez-Opazo, C., Teney, D., & Gould, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00213 · Cited by 238
Self-citationVision-language navigation with self-supervised auxiliary reasoning tasks
Zhu, F., Zhu, Y., Chang, X., & Liang, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01003 · Cited by 223
Dynamic Graph Enhanced Contrastive Learning for Chest X-Ray Report Generation
Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., & Chang, X. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00325 · Cited by 206
Text Recognition in the Wild: A Survey
Chen, X., Jin, L., Zhu, Y., Luo, C., & Wang, T. (2022) · ACM Computing Surveys · doi:10.1145/3440756 · Cited by 205
“structured data, is the main object of NLP. There is a wide range of text-based applications of NLP, including machine translation [98], [99], automatic summarization [100], [101], question answering [102], [103], and relationship extraction [104], [105]. 2.3 Applications Text, as the most important carrier of communi…”
BackgroundInfluentialImproving Vision-and-Language Navigation with Image-Text Pairs from the Web
Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., & Batra, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58539-6_16 · Cited by 201
Self-citationScanQA: 3D Question Answering for Spatial Scene Understanding
Azuma, D., Miyanishi, T., Kurita, S., & Kawanabe, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01854 · Cited by 197
METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens
Wang, Z., Liu, L., Wang, L., & Zhou, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01112 · Cited by 184
“Specifically, we compare METransformer with 5 state-of-the-art (SOTA) image captioning methods, including Show-tell [32], AdaAtt [24], Att2in [1], Transformer [7] and M2transformer [6].”
BackgroundMethodInfluentialCausal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175
ArtEmis: Affective Language for Visual Art
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01140 · Cited by 161
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
ULIP-2: Towards Scalable Multimodal Pre-Training for 3D Understanding
Xue, L., Yu, N., Zhang, S., Panagopoulou, A., Li, J., Martín-Martín, R., Wu, J., Xiong, C., Xu, R., Niebles, J. C., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02558 · Cited by 144
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00166 · Cited by 139
Good News, Everyone! Context Driven Entity-Aware Captioning for News Images
Biten, A. F., Gomez, L., Rusiñol, M., & Karatzas, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01275 · Cited by 136
Region-Aware Image Captioning via Interaction Learning
Liu, A.-A., Zhai, Y., Xu, N., Nie, W., Li, W., & Zhang, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3107035 · Cited by 128
Towards Unsupervised Image Captioning With Shared Multimodal Embeddings
Laina, I., Rupprecht, C., & Navab, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00751 · Cited by 111
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100
From General to Specific: Informative Scene Graph Generation via Balance Adjustment
Guo, Y., Gao, L., Wang, X., Hu, Y., Xu, X., Lu, X., Shen, H. T., & Song, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01607 · Cited by 97
Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation
Yan, B., & Pei, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20204 · Cited by 94
GridMM: Grid Memory Map for Vision-and-Language Navigation
Wang, Z., Li, X., Yang, J., Liu, Y., & Jiang, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01432 · Cited by 93
Ding, Z., Chen, W., Hu, T., & Xu, X. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.116660 · Cited by 85
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84
Vision-Language Navigation with Random Environmental Mixup
Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y.-D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00167 · Cited by 80
Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation
Otani, M., Togashi, R., Sawai, Y., Ishigami, R., Nakashima, Y., Rahtu, E., Heikkilä, J., & Satoh, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01372 · Cited by 78
On hallucination and predictive uncertainty in conditional language generation
Xiao, Y., & Wang, W. Y. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.236 · Cited by 79
Li, Y., Wang, H., & Luo, Y. (2020) · 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm49941.2020.9313289 · Cited by 81
Qi, Y., Pan, Z., Hong, Y., Yang, M.-H., van den Hengel, A., & Wu, Q. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00168 · Cited by 80
Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning
Dong, X., Long, C., Xu, W., & Xiao, C. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475439 · Cited by 74
Faithful Multimodal Explanation for Visual Question Answering
Wu, J., & Mooney, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4812 · Cited by 73
XGPT: Cross-modal Generative Pre-Training for Image Captioning
Xia, Q., Huang, H., Duan, N., Zhang, D., Ji, L., Sui, Z., Cui, E., Bharti, T., & Zhou, M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-88480-2_63 · Cited by 70
TallyQA: Answering Complex Counting Questions
Acharya, M., Kafle, K., & Kanan, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018076 · Cited by 69
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68
FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
Rotstein, N., Bensaïd, D., Brody, S., Ganz, R., & Kimmel, R. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00559 · Cited by 66
Assessing Image Quality Issues for Real-World Problems
Chiu, T.-Y., Zhao, Y., & Gurari, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00370 · Cited by 63
Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
Gat, I., Schwartz, I., Schwing, A., & Hazan, T. (2020) · arXiv · doi:10.48550/arxiv.2010.10802 · Cited by 63
Melfi, G., Müller, K., Schwarz, T., Jaworek, G., & Stiefelhagen, R. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376508 · Cited by 63
Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
Sachdeva, E., Agarwal, N., Chundi, S., Roelofs, S., Li, J., Kochenderfer, M., Choi, C., & Dariush, B. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00734 · Cited by 63
Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation
Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., & Chang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02000 · Cited by 60
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!
Hessel, J., & Lee, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.62 · Cited by 57
Protecting Intellectual Property of Language Generation APIs with Lexical Watermark
He, X., Xu, Q., Lyu, L., Wu, F., & Wang, C. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21321 · Cited by 56
Self-citationInfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection
Fung, Y., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., & Sil, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.133 · Cited by 52
Diagnosing the Environment Bias in Vision-and-Language Navigation
Zhang, Y., Tan, H., & Bansal, M. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/124 · Cited by 51
RUBi: Reducing Unimodal Biases for Visual Question Answering
(2019) · HAL · Cited by 50
Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network
Li, Y., Pan, Y., Yao, T., Chen, J., & Mei, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17034 · Cited by 49
Image captioning by diffusion models: a survey
Daneshfar, F., Bartani, A., & Lotfi, P. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109288 · Cited by 49
ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT
Lee, H., Yoon, S., Dernoncourt, F., Kim, D. S., Bui, T., & Jung, K. (2020) · Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems · doi:10.18653/v1/2020.eval4nlp-1.4 · Cited by 43
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning
Biten, A. F., Gomez, L., & Karatzas, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00253 · Cited by 43
AutoAD: Movie Description in Context
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01815 · Cited by 43
Video Object Grounding Using Semantic Roles in Language Description
Sadhu, A., Chen, K., & Nevatia, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01043 · Cited by 36
Lin, Z., Zhang, D., Shi, D., Xu, R., Tao, Q., Wu, L., He, M., & Ge, Z. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104281 · Cited by 39
Concept-Aware Video Captioning: Describing Videos With Effective Prior Information
Yang, B., Cao, M., & Zou, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3307969 · Cited by 39
Towards Visual Dialog for Radiology
Kovaleva, O., Shivade, C., Kashyap, S., Kanjaria, K., Wu, J., Ballah, D., Coy, A., Karargyris, A., Guo, Y., Beymer, D. B., et al. (2020) · Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing · doi:10.18653/v1/2020.bionlp-1.6 · Cited by 38
Metadata
- Authors
- Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 6077-6086
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2745461083
- Cited by
- 4,948
- References
- 53
- FWCI
- 159.08
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
