The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning
Open accessAbstract
Video captioning, in essential, is a complex natural process, which is affected by various uncertainties stemming from video content, subjective judgment, and so on. In this paper, we build on the recent progress in using encoder-decoder framework for video captioning and address what we find to be a critical deficiency of the existing methods that most of the decoders propagate deterministic hidden states. Such complex uncertainty cannot be modeled efficiently by the deterministic models. In this paper, we propose a generative approach, referred to as multimodal stochastic recurrent neural networks (MS-RNNs), which models the uncertainty observed in the data using latent stochastic variables. Therefore, MS-RNN can improve the performance of video captioning and generate multiple sentences to describe a video considering different random factors. Specifically, a multimodal long short-term memory (LSTM) is first proposed to interact with both visual and textual features to capture a high-level representation. Then, a backward stochastic LSTM is proposed to support uncertainty propagation by introducing latent variables. Experimental results on the challenging data sets, microsoft video description and microsoft research video-to-text, show that our proposed MS-RNN approach outperforms the state-of-the-art video captioning benchmarks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
(2004) · Cited by 9,718
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 3×
“The RNN language model (RNNLM) [36] parameterized the output distribution by applying a softmax function onto the previous hidden state ht−1.”
MethodInfluentialADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
Bleu: a method for automatic evaluation of machine translation
Papineni, K. (2024) · doi:10.57702/ljteqmrb · Cited by 3,672
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3348 · Cited by 1,952
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Fang, C., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.503 · Cited by 1,803
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,204
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-003 · Cited by 1,052
Describing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035
Wang, J., Zhang, T., Song, J., Sebe, N., & Shen, H. T. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699960 · Cited by 971
Self-citationTranslating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., & Saenko, K. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1173 · Cited by 959
Collecting Highly Parallel Data for Paraphrase Evaluation
Chen, D. J., & Dolan, W. B. (2011) · Cited by 904
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 733
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
Serban, I., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A., & Bengio, Y. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10983 · Cited by 697
Video Captioning With Attention-Based LSTM and Semantic Consistency
Gao, L., Guo, Z., Zhang, H., Xu, X., & Shen, H. T. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2729019 · Cited by 637
Self-citationA Recurrent Latent Variable Model for Sequential Data
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02216 · Cited by 635
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.496 · Cited by 564
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.497 · Cited by 563
An overview of methods to evaluate uncertainty of deterministic models in decision support
Uusitalo, L., Lehikoinen, A., Helle, I., & Myrberg, K. (2015) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2014.09.017 · Cited by 538
Semantic Compositional Networks for Visual Captioning
Gan, Z., Gan, C., He, X., Pu, Y., Tran, K., Gao, J., Carin, L., & Deng, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.127 · Cited by 453
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
Pan, P., Xu, Z., Yang, Y., Wu, F., & Zhuang, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.117 · Cited by 412
Unsupervised Deep Hashing with Similarity-Adaptive and Discrete Optimization
Shen, F., Xu, Y., Liu, L., Yang, Y., Huang, Z., & Shen, H. T. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2789887 · Cited by 407
Self-citationTranslating Video Content to Natural Language Descriptions
Rohrbach, M., Qiu, W., Titov, I., Thater, S., Pinkal, M., & Schiele, B. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.61 · Cited by 377
Kojima, A., Tamura, T., & Fukunaga, K. (2002) · International Journal of Computer Vision · doi:10.1023/a:1020346032608 · Cited by 338
Self-Supervised Video Hashing With Hierarchical Binary Auto-Encoder
Song, J., Zhang, H., Li, X., Gao, L., Wang, M., & Hong, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2814344 · Cited by 276
Self-citationTwo-Stream 3-D convNet Fusion for Action Recognition in Videos With Arbitrary Size and Length
Wang, X., Gao, L., Wang, P., Sun, X., & Liu, X. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2749159 · Cited by 263
Self-citationDropout Inference in Bayesian Neural Networks with Alpha-divergences
Li, Y., & Gal, Y. (2017) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1703.02914 · Cited by 236
Discovering Discriminative Graphlets for Aerial Image Categories Recognition
Zhang, L., Han, Y., Yang, Y., Song, M., Yan, S., & Tian, Q. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2278465 · Cited by 233
Sequential Neural Models with Stochastic LayersPreprint
Fraccaro, M., Sønderby, S. K., Paquet, U., & Winther, O. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07571 · Cited by 221
Krishnan, R. G., Shalit, U., & Sontag, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05121 · Cited by 206 · cited 2×
“[49] compared the different variational models, and they guide us to choose an effective variational model.”
BackgroundMethodCai, S., Zuo, W., & Zhang, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.63 · Cited by 201
Quantization-based hashing: a general framework for scalable image and video retrieval
Song, J., Gao, L., Liu, L., Zhu, X., & Sebe, N. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.03.021 · Cited by 182
Self-citationHierarchical LSTM with Adjusted Temporal Attention for Video Captioning
Song, J., Gao, L., Guo, Z., Liu, W., Zhang, D., & Shen, H. T. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/381 · Cited by 165
Self-citationLearning a Recurrent Visual Representation for Image Caption Generation
Chen, X., & Zitnick, C. L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.5654 · Cited by 160
Sequential Neural Models with Stochastic Layers
Fraccaro, M., Sønderby, S. K., Paquet, U., & Winther, O. (2016) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 157
Algorithm-Dependent Generalization Bounds for Multi-Task Learning
Liu, T., Tao, D., Song, M., & Maybank, S. J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2544314 · Cited by 131
Optimized Graph Learning Using Partial Tags and Multiple Features for Image and Video Annotation
Song, J., Gao, L., Nie, F., Shen, H. T., Yan, Y., & Sebe, N. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2601260 · Cited by 130
Self-citationPerson Re-Identification by Dual-Regularized KISS Metric Learning
Tao, D., Guo, Y., Song, M., Li, Y., Yu, Z., & Tang, Y. Y. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2553446 · Cited by 127
Self-citationVideo Captioning with Multi-Faceted Attention
Long, X., Gan, C., & de Melo, G. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00013 · Cited by 104
Hashing with Angular Reconstructive Embeddings
Hu, M., Yang, Y., Shen, F., Xie, N., & Shen, H. T. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2749147 · Cited by 103
Self-citationMulticolumn bidirectional long short-term memory for mobile devices-based human activity recognition
Tao, D., Wen, Y., & Hong, R. (2016) · IEEE Internet of Things Journal · doi:10.1109/jiot.2016.2561962 · Cited by 83
Tensor Rank Preserving Discriminant Analysis for Facial Recognition
Tao, D., Guo, Y., Li, Y., & Gao, X. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2762588 · Cited by 79
Self-citationAttention-based LSTM with Semantic Consistency for Videos Captioning
Guo, Z. Z., Gao, L., Song, J., Xu, X., Shao, J., & Shen, H. T. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2967242 · Cited by 59
Self-citationDeep appearance and motion learning for egocentric activity recognition
Wang, X., Gao, L., Song, J., Zhen, X., Sebe, N., & Shen, H. T. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.08.063 · Cited by 54
Self-citationLearning to Track Multiple Targets
Liu, X., Tao, D., Song, M., Zhang, L., Bu, J., & Chen, C. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2333751 · Cited by 47
SAVE: A framework for semantic annotation of visual events
Lee, M. W., Hakeem, A., Haering, N., et al. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4562954 · Cited by 47
Human Focused Video Description
Khan, M. U. G., Zhang, L., & Gotoh, Y. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130425 · Cited by 40
Diverse image captioning via grouptalk
Wang, Z., Wu, F. Q., Lü, W., Xiao, J., Li, X., Zhang, Z., & Zhuang, Y. (2016) · International Joint Conference on Artificial Intelligence · Cited by 32
Self-citation
Cited by
Task-Adaptive Attention for Image Captioning
Yan, C., Hao, Y., Li, L., Yin, J., Liu, A., Mao, Z., Chen, Z., & Gao, X. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3067449 · Cited by 367
Self-Supervised Video Hashing With Hierarchical Binary Auto-Encoder
Song, J., Zhang, H., Li, X., Gao, L., Wang, M., & Hong, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2814344 · Cited by 276
Self-citationVideo ReCap: Recursive Captioning of Hour-Long Videos
Islam, M. M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., & Bertasius, G. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01723 · Cited by 34
Metadata
- Authors
- Song, J., Guo, Y., Gao, L., Li, X., Hanjalic, A., & Shen, H. T.
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2019-10
- Volume
- 30
- Issue
- 10
- Pages
- 3047-3058
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2962958773
- Cited by
- 259
- References
- 63
- FWCI
- 21.93
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.