Long-term recurrent convolutional networks for visual recognition and description
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or “temporally deep”, are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are “doubly deep” in that they can be compositional in spatial and temporal “layers”. Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages o…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Cameras have been widely used in the HAR applications, however collecting video data presents many issues regarding privacy and computational requirements [10].”
From Human Activity Recognition Using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey (2020) · cites this paper as [10]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“Though RNNs have proven successful on tasks such as speech recognition [43] and text generation [38], it can be difficult to train them to learn long-term dynamics, likely due in part to the vanishing and exploding gradients problem [12] that can result from propagating the gradients down through the many layers of th…”
BackgroundMethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“The net is pre-trained on the 1.2M image ILSVRC-2012 [32] classification training subset of the ImageNet [7] dataset, giving the network a strong initialization to facilitate faster training and prevent over-fitting to the relatively small video datasets.”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2×
“sual and language inputs up to time t. (The visual model ˚ V (x) used in this experiment is the base Caffe [15] reference model, very similar to the wellknown AlexNet [23], pre-trained on ILSVRC-2012 [31] as in Section4.) The third and fourth LSTMs transform the outputs of the LSTM below, and the fourth LSTM’s outputs…”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Research works printed but not linked: 3
- simonyan, 0, Very deep convolutional networks for large-scale image recognition, ICLR 2015
- kiros, 2014, Multimodal neurallanguage models, ICML
- barbu, 2012, Video in sentences out, UAI
Cited by
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
“A natural solution to video classification is to combine the success of CNNs for images and RNNs for sequences [55, 11].”
MethodArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“Architecture modification Layer modification: [143, 268, 269]”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
“Recent advances in deep learning, especially recurrent neural network (RNN) and long short-term memory (LSTM) models [12, 11, 7, 8, 23, 13, 18, 21, 26], provide some useful insights on how to tackle this problem.”
BackgroundGreff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
“structure prediction [11], analysis of audio [12], and video data [13] among others.”
BackgroundVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
“Among those in the second category, a common theme has been to use recurrent neural networks to produce novel captions (Kiros et al. 2014; Mao et al. 2014; Karpathy and Fei-Fei 2015;Vinyals et al. 2015;Chen andLawrenceZitnick 2015; Donahue et al. 2015; Fang et al. 2015).”
BackgroundBottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_2 · Cited by 3,958
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2017) · arXiv · doi:10.48550/arxiv.1711.11248 · Cited by 3,664
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00675 · Cited by 3,516
Social LSTM: Human Trajectory Prediction in Crowded Spaces
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.110 · Cited by 3,483
NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
Shahroudy, A., Liu, J., Ng, T.-T., & Wang, G. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.115 · Cited by 3,148
A survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.213 · Cited by 2,684
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
“Creative and novel applications of using multiple modalities have been among successful recent research directions in machine learning (You et al., 2016; Donahue et al., 2015; Antol et al., 2015; Specia et al., 2016; Tong et al., 2017). opinions from YouTube movie reviews (Zadeh et al., 2016a).”
BackgroundGenerating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061
Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y.-C., & Kim, D. I. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2916583 · Cited by 2,060
TSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,055
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036
“To extract both spatial and temporal information from videos, several architectures have been designed for video feature learning including 2DConvNet-based methods [53], 3DConvNet-based methods [16], and LSTM-based methods [54].”
BackgroundMethodSCA-CNN: Spatial and Channel-Wise Attention in Convolutional Networks for Image Captioning
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., & Chua, T.-S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.667 · Cited by 1,992
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879
“Automatic image captioning has a long history (Ho-dosh et al., 2013; Donahue et al., 2014; Karpa-thy and Fei-Fei, 2015; Kiros et al., 2015).”
BackgroundUnsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Fang, C., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.503 · Cited by 1,803
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Lu, J., Xiong, C., Parikh, D., & Socher, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.345 · Cited by 1,670
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
“Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
BackgroundLong-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Self-citationPredicting residential energy consumption using CNN-LSTM neural networks
Kim, T.-Y., & Cho, S.-B. (2019) · Energy · doi:10.1016/j.energy.2019.05.230 · Cited by 1,469
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification
Xie, S., Sun, C., Huang, J., Tu, Z., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_19 · Cited by 1,415
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Pouyanfar, S., Sadiq, S., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M.-L., Chen, S.-C., & Iyengar, S. S. (2019) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,403
Learning Temporal Regularity in Video Sequences
Hasan, M., Choi, J., Neumann, J., Roy-Chowdhury, A. K., & Davis, L. S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.86 · Cited by 1,397
Deep Recurrent Neural Networks for Hyperspectral Image Classification
Mou, L., Ghamisi, P., & Zhu, X. X. (2017) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2016.2636241 · Cited by 1,383
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
Self-citationSpatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition
Liu, J., Shahroudy, A., Xu, D., & Wang, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_50 · Cited by 1,351
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Meshed-Memory Transformer for Image Captioning
Cornia, M., Stefanini, M., Baraldi, L., & Cucchiara, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01059 · Cited by 1,259
Dense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,251
X3D: Expanding Architectures for Efficient Video Recognition
Feichtenhofer, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00028 · Cited by 1,233
Learning to Track at 100 FPS with Deep Regression Networks
Held, D., Thrun, S., & Savarese, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_45 · Cited by 1,199
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.9 · Cited by 1,174
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.494 · Cited by 1,137
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133
Convolutional neural network: a review of models, methodologies and applications to object detection
Dhillon, A., & Verma, G. K. (2020) · Progress in Artificial Intelligence · doi:10.1007/s13748-019-00203-0 · Cited by 1,119
In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
Li, Y., Chang, M.-C., & Lyu, S. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630787 · Cited by 1,089
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081
DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
Lee, N., Choi, W., Vernaza, P., Choy, C. B., Torr, P. H. S., & Chandraker, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.233 · Cited by 1,067
Temporal Relational Reasoning in Videos
Zhou, B., Andonian, A., Oliva, A., & Torralba, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_49 · Cited by 1,067
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Describing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 968
Self-citationDeep Modular Co-Attention Networks for Visual Question Answering
Yu, Z., Yu, J., Cui, Y., Tao, D., & Tian, Q. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00644 · Cited by 964
Localizing Moments in Video with Natural Language
Hendricks, L. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., & Russell, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.618 · Cited by 958
Self-citationTemporal Segment Networks for Action Recognition in Videos
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Van Gool, L. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2868668 · Cited by 918
Long-Term Temporal Convolutions for Action Recognition
Varol, G., Laptev, I., & Schmid, C. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2712608 · Cited by 915
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.12 · Cited by 908
Self-citationA Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879
“n evaluation metrics. You et al. [156] proposed a semantic attention-based image captioning method. The method providesadetailed,coherentdescriptionofsemanticallyimportantobjects.Thetop-downparadigms [23, 31, 65, 93, 94, 142, 152] are used for extracting visual features first and then convert them into words. In botto…”
MethodTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
End-to-End Learning of Driving Models from Large-Scale Video Datasets
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.376 · Cited by 884
Self-citationShow and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2587640 · Cited by 875
Exploring Visual Relationship for Image Captioning
Yao, T., Pan, Y., Li, Y., & Mei, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_42 · Cited by 870
Recurrent Network Models for Human Dynamics
Fragkiadaki, K., Levine, S., Felsen, P., & Malik, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.494 · Cited by 860
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845
“However, the source of their impressive performance remains poorly understood.”
BackgroundStructured Sequence Modeling with Graph Convolutional Recurrent Networks
Seo, Y., Defferrard, M., Vandergheynst, P., & Bresson, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-04167-0_33 · Cited by 843
TALL: Temporal Activity Localization via Language Query
Gao, J., Sun, C., Yang, Z., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.563 · Cited by 842
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
BackgroundResultSelf-citationVisual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794
A hybrid deep learning based traffic flow prediction method and its understanding
Wu, Y., Tan, H., Qin, L., Ran, B., & Jiang, Z. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2018.03.001 · Cited by 798
Neural Sign Language Translation
Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00812 · Cited by 784
Human Action Recognition and Prediction: A Survey
Kong, Y., & Fu, Y. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01594-9 · Cited by 780
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
“In a related vein, Donahue et al. (2015) propose a combined cnn + lstm architecture (also used by Venugopalan et al., 2015b, 2015a, for video captioning) where the next word is predicted as a function of both previous words and image features.”
BackgroundMethodInfluentialVideos as Space-Time Region Graphs
Wang, X., & Gupta, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_25 · Cited by 765
Learning Deep Representations of Fine-Grained Visual Descriptions
Reed, S., Akata, Z., Lee, H., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.13 · Cited by 760
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.541 · Cited by 760
Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering
Yu, Z., Yu, J., Fan, J., & Tao, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.202 · Cited by 758
Video Summarization with Long Short-Term Memory
Zhang, K., Chao, W.-L., Sha, F., & Grauman, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_47 · Cited by 739
Boosting Image Captioning with Attributes
Yao, T., Pan, Y., Li, Y., Qiu, Z., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.524 · Cited by 733
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_28 · Cited by 727
Self-citationFlow-Guided Feature Aggregation for Video Object Detection
Zhu, X., Wang, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.52 · Cited by 729
DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks
Chen, W., & McDuff, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_22 · Cited by 716
Global Context-Aware Attention LSTM Networks for 3D Action Recognition
Liu, J., Wang, G., Hu, P., Duan, L.-Y., & Kot, A. C. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.391 · Cited by 682
Spatiotemporal Multiplier Networks for Video Action Recognition
Feichtenhofer, C., Pinz, A., & Wildes, R. P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.787 · Cited by 679
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P. S., & Long, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3165153 · Cited by 677
“This architecture is mainly inspired by similar ideas from other well-explored tasks of sequential data, such as neural machine translation [17], [18], speech recognition [19], video action recognition [20], [21], and video captioning [21].”
MethodDeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667
“LRCNs [11] apply CNNs to extract features for each video frame and combine video frame sequences with LSTM [14], which exploits spatio-temporal relationships in video inputs.”
MethodDeep Learning for Safe Autonomous Driving: Current Challenges and Future Directions
Muhammad, K., Ullah, A., Lloret, J., Ser, J. D., & de Albuquerque, V. H. C. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3032227 · Cited by 659
Metadata
- Authors
- Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K.
- Affiliations
- University of California, Berkeley (US)
- International Computer Science Institute (US)
- The University of Texas at Austin (US)
- University of Massachusetts Lowell (US)
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 2625-2634
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1947481528
- Cited by
- 5,228
- References
- 49
- Citations by year
- 2026: 149 · 2025: 246 · 2024: 274 · 2023: 377 · 2022: 412 · 2021: 607 · 2020: 633 · 2019: 758 · 2018: 749 · 2017: 607 · 2016: 305 · 2015: 99
- FWCI
- 238.45
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Funders
- National Science Foundation
- Nvidia
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- recurrent convolutional networks, video recognition, image description generation, image retrieval, video narration, long-range dependencies, temporal dynamics, end-to-end learning, spatiotemporal receptive fields, visual representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.