The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Semantic Compositional Networks for Visual Captioning
Open accessAbstract
A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
(2004) · Cited by 9,718
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
(2005) · Cited by 4,152
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Fang, C., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.503 · Cited by 1,803
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Self-citationGenerating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,264
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., & Saenko, K. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1173 · Cited by 959
Collecting Highly Parallel Data for Paraphrase Evaluation
Chen, D. J., & Dolan, W. B. (2011) · Cited by 904
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 777
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 620
Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. (2014) · Cited by 592
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.496 · Cited by 564
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.497 · Cited by 563
Variational Autoencoder for Deep Learning of Images, Labels and Captions
Pu, Y., Gan, Z., Henao, R., Yuan, X., Li, C., Stevens, A., & Carin, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08976 · Cited by 562
Self-citationMind's eye: A recurrent visual representation for image caption generation
Chen, X., & Zitnick, C. L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298856 · Cited by 498
What Value Do Explicit High Level Concepts Have in Vision to Language Problems?
Wu, Q., Shen, C., Liu, L., Dick, A., & Van Den Hengel, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.29 · Cited by 452
Guiding the Long-Short Term Memory Model for Image Caption Generation
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.277 · Cited by 426
Factored conditional restricted Boltzmann Machines for modeling motion style
Taylor, G. W., & Hinton, G. E. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553505 · Cited by 358
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R., Saenko, K., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.8 · Cited by 286
StyleNet: Generating Attractive Visual Captions with Styles
Gan, C., Gan, Z., He, X., Gao, J., & Deng, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.108 · Cited by 283
Self-citationAttention Correctness in Neural Image Captioning
Liu, C., Mao, J., Sha, F., & Yuille, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11197 · Cited by 210
Language Models for Image Captioning: The Quirks and What Works
Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.3115/v1/p15-2017 · Cited by 201
Self-citationLearning Attributes Equals Multi-Source Domain Generalization
Gan, C., Yang, T., & Gong, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.17 · Cited by 189
Self-citationUnsupervised Learning of Image Transformations
Memisevic, R., & Hinton, G. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383036 · Cited by 175
Rich Image Captioning in the Wild
Tran, K., He, X., Zhang, L., & Sun, J. Z. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.61 · Cited by 110
Self-citationJin, J., Fu, K., Cui, R., Sha, F., & Zhang, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06272 · Cited by 97
Early Embedding and Late Reranking for Video Captioning
Dong, J., Li, X., Lan, W., Huo, Y., & Snoek, C. G. M. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2984064 · Cited by 75
Guiding Long-Short Term Memory for Image Caption GenerationPreprint
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.04942 · Cited by 67
A Multiplicative Model for Learning Distributed Text-Based Attribute RepresentationsPreprint
Kiros, R., Zemel, R. S., & Salakhutdinov, R. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2710 · Cited by 37
Cited by
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
Self-citationAttention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879
Video Captioning With Attention-Based LSTM and Semantic Consistency
Gao, L., Guo, Z., Zhang, H., Xu, X., & Shen, H. T. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2729019 · Cited by 637
End-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605
Metadata
- Authors
- Gan, Z., Gan, C., He, X., Pu, Y., Tran, K., Gao, J., Carin, L., & Deng, L.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 1141-1150
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2558834163
- Cited by
- 453
- References
- 53
- FWCI
- 18.52
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.