Deep visual-semantic alignments for generating image descriptions
The DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index.
Abstract
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On the COCO dataset, the top-10 detection methods are all transformer-based 5 .”
From Object Detection in 20 Years: A Survey (2023) · cites this paper as [5]
In the citing paper
“These advances have spurred growth in other domains of CV, such as multimodal learning, which combines vision with other modalities such as language (Fig. 1a) 26 , time-series data, and genomic data 5 .”
From Deep learning-enabled medical computer vision (2021) · cites this paper as [26]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citationGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Self-citationGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Research works printed but not linked: 2
- 10.1016/0364-0213(90)90002-E · doi:10.1016/0364-0213(90)90002-e · Paper page
- 10.1007/3-540-33486-6_6 · doi:10.1007/3-540-33486-6_6 · Paper page
Cited by
Items 301–400 of 1,041
Every Pixel Matters: Center-Aware Feature Alignment for Domain Adaptive Object Detector
Hsu, C.-C., Tsai, Y.-H., Lin, Y.-Y., & Yang, M.-H. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58545-7_42 · Cited by 232
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02240 · Cited by 247
Fake News Identification on Twitter with Hybrid CNN and RNN Models
Ajao, O., Bhowmik, D., & Zargari, S. (2018) · Proceedings of the 9th International Conference on Social Media and Society · doi:10.1145/3217804.3217917 · Cited by 253
ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
Wang, Z., Fang, Z., Wang, J., & Yang, Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_24 · Cited by 232
Bottom-Up and Top-Down Attention for Image Captioning and VQA.Preprint
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2017) · arXiv (Cornell University) · Cited by 252
Multimodal Residual Learning for Visual QA
Kim, J.-H., Lee, S., Kwak, D., Heo, M.-O., Kim, J., Ha, J.-W., & Zhang, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01455 · Cited by 209
Hierarchical LSTMs with Adaptive Attention for Visual Captioning
Gao, L., Li, X., Song, J., & Shen, H. T. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894139 · Cited by 245
Situation Recognition: Visual Semantic Role Labeling for Image Understanding
Yatskar, M., Zettlemoyer, L., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.597 · Cited by 242
Deep Computational Phenotyping
Che, Z., Kale, D., Li, W., Bahadori, M. T., & Liu, Y. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783365 · Cited by 239
An, Q., Tao, Z., Xu, X., El Mansori, M., & Chen, M. (2020) · Measurement · doi:10.1016/j.measurement.2019.107461 · Cited by 247
RUBi: Reducing Unimodal Biases in Visual Question Answering
Cadène, R., Dancette, C., Ben-younes, H., Cord, M., & Parikh, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10169 · Cited by 194
Visual question answering: Datasets, algorithms, and future challenges
Kafle, K., & Kanan, C. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.005 · Cited by 245
Machine health monitoring with LSTM networks
Zhao, R., Wang, J., Yan, R., & Mao, K. (2016) · 2016 10th International Conference on Sensing Technology (ICST) · doi:10.1109/icsenst.2016.7796266 · Cited by 229
pix2code: Generating Code from a Graphical User Interface Screenshot
Beltramelli, T. (2018) · Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems · doi:10.1145/3220134.3220135 · Cited by 237
A survey on automatic image caption generation
Bai, S., & An, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.080 · Cited by 241
Khanam, R., Hussain, M., Hill, R., & Allen, P. (2024) · IEEE Access · doi:10.1109/access.2024.3425166 · Cited by 237
Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos
Zhang, Z., Lin, Z., Zhao, Z., & Xiao, Z. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331235 · Cited by 234
X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval
Gorti, S. K., Vouitsis, N., Ma, J., Golestan, K., Volkovs, M., Garg, A., & Yu, G. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00495 · Cited by 229
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM
Huang, Y., Wang, W., & Wang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.767 · Cited by 233
ViP-CNN: Visual Phrase Guided Convolutional Neural Network
Li, Y., Ouyang, W., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.766 · Cited by 229
Feng, Y., Ma, L., Liu, W., & Luo, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00425 · Cited by 225
Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
Shetty, R., Rohrbach, M., Hendricks, L. A., Fritz, M., & Schiele, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.445 · Cited by 222
Wang, Y., Jiang, Z., Li, Y., Hwang, J.-N., Xing, G., & Liu, H. (2021) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2021.3058895 · Cited by 228
TGIF: A New Dataset and Benchmark on Animated GIF Description
Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.502 · Cited by 226
Deep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230
Review Networks for Caption Generation
Yang, Z., Yuan, Y., Wu, Y., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07912 · Cited by 62
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
Huang, Z., Zeng, Z., Huang, Y., Liu, B., Fu, D., & Fu, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01278 · Cited by 221
Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching
Nagrani, A., Albanie, S., & Zisserman, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00879 · Cited by 220
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2991965 · Cited by 221
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_9 · Cited by 226
Self-citationCan a Machine Generate Humanlike Language Descriptions for a Remote Sensing Image?
Shi, Z., & Zou, Z. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2017.2677464 · Cited by 225
Unsupervised learning of spoken language with visual context
Harwath, D. F., Torralba, A. B., & Glass, J. R. (2016) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 271
Grounding of Textual Phrases in Images by Reconstruction
Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., & Schiele, B. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_49 · Cited by 221
Tellex, S., Gopalan, N., Kress-Gazit, H., & Matuszek, C. (2020) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-101119-071628 · Cited by 220
Khan, W., Daud, A., Khan, K., Muhammad, S., & Haq, R. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100026 · Cited by 217
Semantic content-based image retrieval: A comprehensive study
Alzu’bi, A., Amira, A., & Ramzan, N. (2015) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2015.07.012 · Cited by 220
Nguyen, A. M., Yosinski, J., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03616 · Cited by 152
Diverse Beam Search for Improved Description of Complex Scenes
Vijayakumar, A., Cogswell, M., Selvaraju, R., Sun, Q., Lee, S., Crandall, D., & Batra, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12340 · Cited by 219
Hierarchy Parsing for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00271 · Cited by 206
Hao, S., Ge, F.-X., Li, Y., & Jiang, J. (2020) · Measurement · doi:10.1016/j.measurement.2020.107802 · Cited by 218
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 216
Learning multi-scale features for foreground segmentation
Lim, L. A., & Keles, H. Y. (2020) · Pattern Analysis and Applications · doi:10.1007/s10044-019-00845-9 · Cited by 174
Dibia, V., & Demiralp, C. (2019) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2019.2924636 · Cited by 206
Areas of Attention for Image Captioning
Pedersoli, M., Lucas, T., Schmid, C., & Verbeek, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.140 · Cited by 208
Multi-Horizon Time Series Forecasting with Temporal Attention Learning
Fan, C., Zhang, Y., Pan, Y., Li, X., Zhang, C., Yuan, R., Wu, D., Wang, W., Pei, J., & Huang, H. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330662 · Cited by 205
SentiCap: Generating Image Descriptions with Sentiments
Mathews, A., Xie, L., & He, X. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10475 · Cited by 213
Negative-Aware Attention Framework for Image-Text Matching
Zhang, K., Mao, Z., Wang, Q., & Zhang, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01521 · Cited by 202
Video Captioning by Adversarial LSTM
Yang, Y., Zhou, J., Ai, J., Bin, Y., Hanjalic, A., Shen, H. T., & Ji, Y. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2855422 · Cited by 214
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
Cornia, M., Baraldi, L., & Cucchiara, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00850 · Cited by 205
Spine-GAN: Semantic segmentation of multiple spinal structures
Han, Z., Wei, B., Mercado, A., Leung, S., & Li, S. (2018) · Medical Image Analysis · doi:10.1016/j.media.2018.08.005 · Cited by 211
Recurrent Topic-Transition GAN for Visual Paragraph Generation
Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.364 · Cited by 205
Adversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 208
Language Models for Image Captioning: The Quirks and What Works
Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.3115/v1/p15-2017 · Cited by 201
Omnivore: A Single Model for Many Visual Modalities
Girdhar, R., Singh, M., Ravi, N., van der Maaten, L., Joulin, A., & Misra, I. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01563 · Cited by 196
Pixels to Graphs by Associative Embedding
Newell, A., & Deng, J. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.07365 · Cited by 178
Weakly Supervised Video Moment Retrieval From Text Queries
Mithun, N. C., Paul, S., & Roy-Chowdhury, A. K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01186 · Cited by 202
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235
Multilabel Image Classification With Regional Latent Semantic Dependencies
Zhang, J., Wu, Q., Shen, C., Zhang, J., & Lu, J. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2812605 · Cited by 203
Position Focused Attention Network for Image-Text Matching
Wang, Y., Yang, H., Qian, X., Ma, L., Lu, J., Li, B., & Fan, X. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/526 · Cited by 195
Understanding Blind People's Experiences with Computer-Generated Captions of Social Media Images
MacLeod, H., Bennett, C. L., Morris, M. R., & Cutrell, E. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025814 · Cited by 200
Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
Gu, J., Cai, J., Wang, G., & Chen, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12266 · Cited by 196
Hierarchical Boundary-Aware Neural Encoder for Video Captioning
Baraldi, L., Grana, C., & Cucchiara, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.339 · Cited by 199
You Said That?: Synthesising Talking Faces from Audio
Jamaludin, A., Chung, J. S., & Zisserman, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01150-y · Cited by 189
Smart-home automation using IoT-based sensing and monitoring platform
Al-Kuwari, M., Ramadan, A., Ismael, Y., Al-Sughair, L., Gastli, A., & Benammar, M. (2018) · 2018 IEEE 12th International Conference on Compatibility, Power Electronics and Power Engineering (CPE-POWERENG 2018) · doi:10.1109/cpe.2018.8372548 · Cited by 191
R2GenGPT: Radiology Report Generation with frozen LLMs
Wang, Z., Liu, L., Wang, L., & Zhou, L. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100033 · Cited by 192
Predicting Visual Features From Text for Image and Video Caption Retrieval
Dong, J., Li, X., & Snoek, C. G. M. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2832602 · Cited by 194
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01750 · Cited by 189
Wang, W., Cui, Z., Yan, Y., Feng, J., Yan, S., Shu, X., & Sebe, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.261 · Cited by 185
Dynamic Modality Interaction Modeling for Image-Text Retrieval
Qu, L., Liu, M., Wu, J., Gao, Z., & Nie, L. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3462829 · Cited by 186
Das, S., Tariq, A., Santos, T., Kantareddy, S. S., & Banerjee, I. (2023) · Neuromethods · doi:10.1007/978-1-0716-3195-9_4 · Cited by 187
Discriminability Objective for Training Descriptive Captions
Luo, R., Shakhnarovich, G., Cohen, S., & Price, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00728 · Cited by 191
Development of UAV-Based Target Tracking and Recognition Systems
Wang, S., Jiang, F., Zhang, B., Ma, R., & Hao, Q. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2927838 · Cited by 192
Deep learning in generating radiology reports: A survey
Monshi, M. M. A., Poon, J., & Chung, V. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101878 · Cited by 192
Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
Wang, H., Zhang, Y., Ji, Z., Pang, Y., & Ma, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58586-0_2 · Cited by 174
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
Wang, X., Zhu, L., & Yang, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00504 · Cited by 193
Attention Correctness in Neural Image Captioning
Liu, C., Mao, J., Sha, F., & Yuille, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11197 · Cited by 210
Ji, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16258 · Cited by 183
Image Search With Text Feedback by Visiolinguistic Attention Learning
Chen, Y., Gong, S., & Bazzani, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00307 · Cited by 184
Domain Adaptation Without Source Data
Kim, Y., Cho, D., Han, K., Panda, P., & Hong, S. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3110179 · Cited by 190
Semantic Proposal for Activity Localization in Videos via Sentence Query
Chen, S., & Jiang, Y.-G. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018199 · Cited by 187
Huang, Z., Yang, F., Xu, F., Song, X., & Tsui, K.-L. (2019) · IEEE Access · doi:10.1109/access.2019.2928037 · Cited by 188
Variants of RMSProp and Adagrad with Logarithmic Regret BoundsPreprint
Mukkamala, M. C., & Hein, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05507 · Cited by 310
Deep Learning Techniques for Future Intelligent Cross-Media Retrieval
Rehman, S. U., Waqas, M. M., Tu, S., Koubâa, A., Rehman, O. U., Ahmad, J. E., Hanif, M. I., & Han, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01191 · Cited by 190
Wang, X., Xiong, W., Wang, H., & Wang, W. Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_3 · Cited by 186
Know More Say Less: Image Captioning Based on Scene Graphs
Li, X., & Jiang, S. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896516 · Cited by 187
Concepts and Compositionality: In Search of the Brain's Language of Thought
Frankland, S. M., & Greene, J. D. (2020) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011829 · Cited by 184
Temporally Grounding Language Queries in Videos by Contextual Boundary-Aware Prediction
Wang, J., Ma, L., & Jiang, W. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6897 · Cited by 181
Multimodal Contrastive Training for Visual Representation Learning
Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., & Faieta, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00692 · Cited by 179
MSMO: Multimodal Summarization with Multimodal Output
Zhu, J., Li, H., Liu, T., Zhou, Y., Zhang, J., & Zong, C. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1448 · Cited by 179
Tracking by Natural Language Specification
Li, Z., Tao, R., Gavves, E., Snoek, C. G. M., & Smeulders, A. W. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.777 · Cited by 177
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
Mei, H., Bansal, M., & Walter, M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10364 · Cited by 180
Fang, K., & Shen, C. (2020) · Journal of Hydrometeorology · doi:10.1175/jhm-d-19-0169.1 · Cited by 182
Weakly-Supervised Learning of Visual Relations
Peyre, J., Laptev, I., Schmid, C., & Sivic, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.554 · Cited by 172
A Review of Deep Learning Research
(2019) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2019.04.001 · Cited by 178
Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
Liu, C., Mao, Z., Liu, A.-A., Zhang, T., Wang, B., & Zhang, Y. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350869 · Cited by 182
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2024) · Artificial Intelligence in the Age of Neural Networks and Brain Computing · doi:10.1016/b978-0-323-96104-2.00002-6 · Cited by 179
Captioning Images with Diverse Objects
Venugopalan, S., Hendricks, L. A., Rohrbach, M., Mooney, R., Darrell, T., & Saenko, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.130 · Cited by 179
Uncovering the Temporal Context for Video Question Answering
Zhu, L., Xu, Z., Yang, Y., & Hauptmann, A. G. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1033-7 · Cited by 181
Cross-Modality Bridging and Knowledge Transferring for Image Understanding
Yan, C., Li, L., Zhang, C., Liu, B., Zhang, Y., & Dai, Q. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2903448 · Cited by 181
MAC: Mining Activity Concepts for Language-Based Temporal Localization
Ge, R., Gao, J., Chen, K., & Nevatia, R. (2019) · 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2019.00032 · Cited by 175
Metadata
- Authors
- Karpathy, A., & Fei-Fei, L.
- Affiliations
- Stanford University (US)
- ORCID
- Fei-Fei, Li
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3128-3137
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1905882502
- Cited by
- 4,902
- References
- 67
- Citations by year
- 2026: 145 · 2025: 313 · 2024: 387 · 2023: 464 · 2022: 427 · 2021: 542 · 2020: 509 · 2019: 621 · 2018: 555 · 2017: 471 · 2016: 324 · 2015: 141
- FWCI
- 203.84
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- image description generation, visual-semantic alignment, multimodal alignment, convolutional neural networks, bidirectional recurrent neural network, multimodal embeddings, image retrieval, Flickr8k, Flickr30k, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.