The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Video Captioning with Transferred Semantic Attributes
Open accessAbstract
Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community. Most recent progress in this problem has been achieved through employing 2-D and/or 3-D Convolutional Neural Networks (CNNs) to encode video content and Recurrent Neural Networks (RNNs) to decode a sentence. In this paper, we present Long Short-Term Memory with Transferred Semantic Attributes (LSTM-TSA) - a novel deep architecture that incorporates the transferred semantic attributes learnt from images and videos into the CNN plus RNN framework, by training them in an end-to-end manner. The design of LSTM-TSA is highly inspired by the facts that 1) semantic attributes play a significant contribution to captioning, and 2) images and videos carry complementary semantics and thus can reinforce each other for captioning. To boost video captioning, we propose a novel transfer unit to model the mutually correlated attributes learnt from images and videos. Extensive experiments are conducted on three public datasets, i.e., MSVD, M-VAD and MPIIMD. Our proposed LSTM-TSA achieves to-date the best published performance in sentence generation on MSVD: 52.8% and 74.0% in terms of BLEU@4 and CIDEr-D. Superior results are also reported on M-VAD and MPII-MD when compared to state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
(2005) · Cited by 4,152
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Fang, C., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.503 · Cited by 1,803
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
Self-citationMicrosoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Describing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., & Saenko, K. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1173 · Cited by 959
Collecting Highly Parallel Data for Paraphrase Evaluation
Chen, D. J., & Dolan, W. B. (2011) · Cited by 904
Parikh, D., & Grauman, K. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126281 · Cited by 851
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 733
Self-citationMultiple Instance Boosting for Object Detection
Zhang, C., Platt, J. C., & Viola, P. A. (2005) · neural information processing systems · Cited by 687
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.496 · Cited by 564
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.497 · Cited by 563
Self-citationGuadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., & Saenko, K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.337 · Cited by 500
What Value Do Explicit High Level Concepts Have in Vision to Language Problems?
Wu, Q., Shen, C., Liu, L., Dick, A., & Van Den Hengel, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.29 · Cited by 452
A dataset for Movie Description
Rohrbach, A., Rohrbach, M., Tandon, N., & Schiele, B. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298940 · Cited by 435
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
Pan, P., Xu, Z., Yang, Y., Wu, F., & Zhuang, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.117 · Cited by 412
Translating Video Content to Natural Language Descriptions
Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., & Schiele, B. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.61 · Cited by 377
Kojima, A., Tamura, T., & Fukunaga, K. (2002) · International Journal of Computer Vision · doi:10.1023/a:1020346032608 · Cited by 338
Xu, R., Xiong, C., Chen, W., & Corso, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9512 · Cited by 298
Using Descriptive Video Services to Create a Large Data Source for Video Annotation ResearchPreprint
Torabi, A., Pal, C., Larochelle, H., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01070 · Cited by 149
The Long-Short Story of Movie Description
Rohrbach, A., Rohrbach, M., & Schiele, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24947-6_17 · Cited by 119
You Lead, We Exceed: Labor-Free Video Concept Learning by Jointly Exploiting Web Videos and Images
Gan, C., Yao, T., Yang, K., Yang, Y., & Mei, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.106 · Cited by 109
Self-citationShetty, R., & Laaksonen, J. T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02949 · Cited by 33
Cited by
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_42 · Cited by 870
Self-citationBoosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 733
Self-citationVideo Captioning With Attention-Based LSTM and Semantic Consistency
Gao, L., Guo, Z., Zhang, H., Xu, X., & Shen, H. T. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2729019 · Cited by 637
End-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605
Shifting More Attention to Video Salient Object Detection
Fan, D.-P., Wang, W., Cheng, M.-M., & Shen, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00875 · Cited by 510
Metadata
- Authors
- Pan, Y., Yao, T., Li, H., & Mei, T.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 984-992
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2556388456
- Cited by
- 364
- References
- 37
- FWCI
- 16.10
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.