The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Deep correlation for matching images and text
Abstract
This paper addresses the problem of matching images and captions in a joint latent space learnt with deep canonical correlation analysis (DCCA). The image and caption data are represented by the outputs of the vision and text based deep neural networks. The high dimensionality of the features presents a great challenge in terms of memory and speed complexity when used in DCCA framework. We address these problems by a GPU implementation and propose methods to deal with overfitting. This makes it possible to evaluate DCCA approach on popular caption-image matching benchmarks. We compare our approach to other recently proposed techniques and present state of the art results on three datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
(2004) · Cited by 9,718
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Deep Visual-Semantic Alignments for Generating Image DescriptionsPreprint
Karpathy, A., & Fei-Fei, L. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.1412.2306 · Cited by 6,303
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2014.131 · Cited by 4,451
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
Large Margin Methods for Structured and Interdependent Output Variables
Tsochantaridis, I., Joachims, T., Hofmann, T., & Altün, Y. (2005) · MPG.PuRe (Max Planck Society) · Cited by 1,879
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · Cited by 1,602
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,314
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,204
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., & Saenko, K. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1173 · Cited by 959
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 874
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 777
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 739
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995466 · Cited by 521
Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., & Saenko, K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.337 · Cited by 500
Midge: Generating Image Descriptions From Computer Vision Detections
Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T. L., & Daumé, H. (2012) · Cited by 416
Automatic Attribute Discovery and Characterization from Noisy Web Data
Berg, T. L., Berg, A. C., & Shih, J. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_48 · Cited by 411
WordsEye: an automatic text-to-scene conversion system
Coyne, B., & Sproat, R. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383316 · Cited by 377
Translating Video Content to Natural Language Descriptions
Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., & Schiele, B. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.61 · Cited by 377
Corpus-Guided Sentence Generation of Natural Images
Yang, Y., Teo, C. L., Daumé, H., & Aloimonos, Y. (2011) · Cited by 356
Composing Simple Image Descriptions using Web-scale N-grams
Li, S., Kulkarni, G. A., Berg, T. L., Berg, A. C., & Choi, Y. (2011) · Cited by 353
Explain Images with Multimodal Recurrent Neural Networks
Mao, J., Xu, W., Yang, Y., Wang, J., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.1090 · Cited by 341
Collective Generation of Natural Image Descriptions
Кузнецова, П., Ordóñez, V., Berg, A., Berg, T. L., & Choi, Y. (2012) · Cited by 319
The IAPR Benchmark : A New Evaluation Resource for Visual Information Systems
Grubinger, M. (2006) · RWTH Publications (RWTH Aachen) · Cited by 285
Das, P., Xu, C., Doell, R. F., & Corso, J. J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.340 · Cited by 282
Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections
Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., & Lazebnik, S. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_35 · Cited by 281
T ree T alk: Composition and Compression of Trees for Image Descriptions
Kuznetsova, P., Ordonez, V., Berg, T. L., & Choi, Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00188 · Cited by 267
Generating Natural-Language Video Descriptions Using Text-Mined Knowledge
Krishnamoorthy, N., Malkarnenkar, G., Mooney, R., Saenko, K., & Guadarrama, S. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8679 · Cited by 232
Coherent Multi-sentence Video Description with Variable Level of Detail
Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Pinkal, M., & Schiele, B. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11752-2_15 · Cited by 206
Reiter, E., & Belz, A. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35405 · Cited by 196
CIDEr: Consensus-based Image Description EvaluationPreprint
Vedantam, R., Zitnick, C. L., & Parikh, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5726 · Cited by 163
Learning a Recurrent Visual Representation for Image Caption Generation
Chen, X., & Zitnick, C. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5654 · Cited by 160
Show and Tell: A Neural Image Caption GeneratorPreprint
Vinyals, O., Toshev, A., Samy Bengio, A., & Erhan, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4555 · Cited by 156
Comparing Automatic Evaluation Measures for Image Description
Elliott, D., & Keller, F. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2074 · Cited by 137
Choosing Linguistics over Vision to Describe Images
Gupta, A., Verma, Y., & Jawahar, C. V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8205 · Cited by 120
Nonparametric Method for Data-driven Image Captioning
Mason, R., & Eugene Charniak, D. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2097 · Cited by 108
A Dataset for Movie DescriptionPreprint
Rohrbach, A., Rohrbach, M., Tandon, N., & Schiele, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.02530 · Cited by 58
Im2Text and Text2Im: Associating Images and Texts for Cross-Modal Retrieval
Verma, Y., & Jawahar, C. V. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.97 · Cited by 54
Leveraging High Level Visual Information for Matching Images and Captions
Yan, F., & Mikolajczyk, K. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16865-4_40 · Cited by 4
Self-citation
Cited by
Adversarial Cross-Modal Retrieval
Wang, B., Yang, Y., Xu, X., Hanjalic, A., & Shen, H. T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123326 · Cited by 779
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.541 · Cited by 760
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
Multi-Modality Cross Attention Network for Image and Sentence Matching
Wei, X., Zhang, T., Li, Y., Zhang, Y., & Wu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01095 · Cited by 451
An Overview of Cross-Media Retrieval: Concepts, Methodologies, Benchmarks, and Challenges
Peng, Y., Huang, X., & Zhao, Y. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2705068 · Cited by 341
Metadata
- Authors
- Yan, F., & Mikolajczyk, K.
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 3441-3450
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1949478088
- Cited by
- 395
- References
- 56
- FWCI
- 18.98
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.