Long-term recurrent convolutional networks for visual recognition and description
The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or “temporally deep”, are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are “doubly deep” in that they can be compositional in spatial and temporal “layers”. Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages o…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Cameras have been widely used in the HAR applications, however collecting video data presents many issues regarding privacy and computational requirements [10].”
From Human Activity Recognition Using Inertial, Physiological and Environmental Sensors: A Comprehensive Survey (2020) · cites this paper as [10]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“Though RNNs have proven successful on tasks such as speech recognition [43] and text generation [38], it can be difficult to train them to learn long-term dynamics, likely due in part to the vanishing and exploding gradients problem [12] that can result from propagating the gradients down through the many layers of th…”
BackgroundMethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“The net is pre-trained on the 1.2M image ILSVRC-2012 [32] classification training subset of the ImageNet [7] dataset, giving the network a strong initialization to facilitate faster training and prevent over-fitting to the relatively small video datasets.”
BackgroundGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2×
“sual and language inputs up to time t. (The visual model ˚ V (x) used in this experiment is the base Caffe [15] reference model, very similar to the wellknown AlexNet [23], pre-trained on ILSVRC-2012 [31] as in Section4.) The third and fourth LSTMs transform the outputs of the LSTM below, and the fourth LSTM’s outputs…”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Research works printed but not linked: 3
- simonyan, 0, Very deep convolutional networks for large-scale image recognition, ICLR 2015
- kiros, 2014, Multimodal neurallanguage models, ICML
- barbu, 2012, Video in sentences out, UAI
Cited by
Items 201–300 of 374
NoScope: optimizing neural network queries over video at scale
Kang, D., Emmons, J., Abuzaid, F., Bailis, P., & Zaharia, M. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137664 · Cited by 367
“Now, these “deep” models (with millions to billions of parameters) have become not only feasible but also the preferred method for computer vision tasks spanning image classification [82], pedestrian detection [98], and automatic captioning [29].”
MethodSubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition
Camgoz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.332 · Cited by 365
A Hierarchical Approach for Generating Descriptive Image Paragraphs
Krause, J., Johnson, J., Krishna, R., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.356 · Cited by 364
Video Captioning with Transferred Semantic Attributes
Pan, Y., Yao, T., Li, H., & Mei, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.111 · Cited by 364
Research on particle swarm optimization in LSTM neural networks for rainfall-runoff simulation
Xu, Y., Hu, C., Wu, Q., Jian, S., Li, Z., Chen, Y., Zhang, G., Zhang, Z., & Wang, S. (2022) · Journal of Hydrology · doi:10.1016/j.jhydrol.2022.127553 · Cited by 364
Exploring Models and Data for Image Question Answering
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 359
In Defense of Grid Features for Visual Question Answering
Jiang, H., Misra, I., Rohrbach, M., Learned-Miller, E., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01028 · Cited by 358
Self-citationMinimal gated unit for recurrent neural networks
Zhou, G.-B., Wu, J., Zhang, C.-L., & Zhou, Z.-H. (2016) · International Journal of Automation and Computing · doi:10.1007/s11633-016-1006-2 · Cited by 357
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Das, A., Kottur, S., Moura, J. M. F., Lee, S., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.321 · Cited by 356
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources
Wu, Q., Wang, P., Shen, C., Dick, A., & Van Den Hengel, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.500 · Cited by 355
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354
“Recently, image to language generation [14, 22, 4, 33] and attention [36] methods attract some research interests.”
BackgroundMethodInfluentialEvery Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., & Fei-Fei, L. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-017-1013-y · Cited by 354
FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding
Shao, D., Zhao, Y., Dai, B., & Lin, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00269 · Cited by 353
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 349
Computer vision algorithms and hardware implementations: A survey
Feng, X., Jiang, Y., Yang, X., Du, M., & Li, X. (2019) · Integration · doi:10.1016/j.vlsi.2019.07.005 · Cited by 349
Compressed Video Action Recognition
Wu, C.-Y., Zaheer, M., Hu, H., Manmatha, R., Smola, A. J., & Krahenbuhl, P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00631 · Cited by 347
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., & Freeman, W. T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.264 · Cited by 346
A scalable deep learning platform for identifying geologic features from seismic attributes
Huang, L., Dong, X., & Clee, T. E. (2017) · The Leading Edge · doi:10.1190/tle36030249.1 · Cited by 345
SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction
Shi, L., Wang, L., Long, C., Zhou, S., Zhou, M., Niu, Z., & Hua, G. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00888 · Cited by 345
A Joint Sequence Fusion Model for Video Question Answering and Retrieval
Yu, Y., Kim, J., & Kim, G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_29 · Cited by 343
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00645 · Cited by 342
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01101 · Cited by 342
Web traffic anomaly detection using C-LSTM neural networks
Kim, T.-Y., & Cho, S.-B. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.04.004 · Cited by 340
Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment
Trigeorgis, G., Snape, P., Nicolaou, M. A., Antonakos, E., & Zafeiriou, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.453 · Cited by 337
Reconstruction Network for Video Captioning
Wang, B., Ma, L., Zhang, W., & Liu, W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00795 · Cited by 337
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334
Multimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334
“Multimodal models have been used for sentiment analysis [28, 45, 46]; medical purposes, such as detection of suicidal risk, PTSD and depression [29, 34, 35, 43]; emotion recognition [23]; image captioning and media description [8, 41]; question answering [2]; and multimodal translation [31].”
MethodPoTion: Pose MoTion Representation for Action Recognition
Choutas, V., Weinzaepfel, P., Revaud, J., & Schmid, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00734 · Cited by 332
RESOUND: Towards Action Recognition Without Representation Bias
Li, Y., Li, Y., & Vasconcelos, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_32 · Cited by 332
Pairwise Relationship Guided Deep Hashing for Cross-Modal Retrieval
Yang, E., Deng, C., Liu, W., Liu, X., Tao, D., & Gao, X. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10719 · Cited by 331
Deep Reinforcement Learning-Based Image Captioning with Embedding Reward
Ren, Z., Wang, X., Zhang, N., Lv, X., & Li, L.-J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.128 · Cited by 331
Revisiting Video Saliency Prediction in the Deep Learning Era
Wang, W., Shen, J., Xie, J., Cheng, M.-M., Ling, H., & Borji, A. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2924417 · Cited by 330
Inverting the Generator of a Generative Adversarial Network
Creswell, A., & Bharath, A. A. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2875194 · Cited by 330
DeepMon: Mobile GPU-based Deep Learning Framework for Continuous Vision Applications
Huynh, L. N., Lee, Y., & Balan, R. K. (2017) · Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3081333.3081360 · Cited by 329
“There are other models such as Faster-RCNN [22] for object detection and Long-term Recurrent Convolutional Networks (LRCN [19]) for activity recognition – LRCN is the combination of CNN and Long Short Term Memory (LSTM [27]).”
MethodRegularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328
SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
Jin, Y., Dou, Q., Chen, H., Yu, L., Qin, J., Fu, C.-W., & Heng, P.-A. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2787657 · Cited by 316
“HMM, we propose to tap into the temporal dimension of the surgical video data using the LSTM [31], [32], which has been demonstrated as a very powerful tool to model temporal concepts.”
MethodLearning Fashion Compatibility with Bidirectional LSTMs
Han, X., Wu, Z., Jiang, Y.-G., & Davis, L. S. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123394 · Cited by 325
HP-GAN: Probabilistic 3D Human Motion Prediction via GAN
Barsoum, E., Kender, J., & Liu, Z. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00191 · Cited by 322
Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks
Kopuklu, O., Gunduz, A., Kose, N., & Rigoll, G. (2019) · 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) · doi:10.1109/fg.2019.8756576 · Cited by 319
Infrared machine vision and infrared thermography with deep learning: A review
He, Y., Deng, B., Wang, H., Cheng, L., Zhou, K., Cai, S., & Ciampa, F. (2021) · Infrared Physics & Technology · doi:10.1016/j.infrared.2021.103754 · Cited by 319
Learning Video Object Segmentation with Visual Memory
Tokmakov, P., Alahari, K., & Schmid, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.480 · Cited by 316
Learning to detect violent videos using convolutional long short-term memory
Sudhakaran, S., & Lanz, O. (2017) · 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2017.8078468 · Cited by 316
Machine learning for active matter
Cichos, F., Gustavsson, K., Mehlig, B., & Volpe, G. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0146-9 · Cited by 315
Christie, G., Fendley, N., Wilson, J., & Mukherjee, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00646 · Cited by 312
nocaps: novel object captioning at scale
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., & Anderson, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00904 · Cited by 312
Yin and Yang: Balancing and Answering Binary Visual Questions
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., & Parikh, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.542 · Cited by 312
Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals
Zhang, Y., Chen, Y., Wang, J., & Pan, Z. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3102110 · Cited by 312
Recurrent Instance Segmentation
Romera-Paredes, B., & Torr, P. H. S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_19 · Cited by 311
Yadav, S. K., Tiwari, K., Pandey, H. M., & Akbar, S. A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106970 · Cited by 311
On Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks
Zhang, S., Liu, X., & Xiao, J. (2017) · 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2017.24 · Cited by 310
Genetic Algorithm-Optimized Long Short-Term Memory Network for Stock Market Prediction
Chung, H., & Shin, K.-S. (2018) · Sustainability · doi:10.3390/su10103765 · Cited by 310
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
Xu, H., He, K., Plummer, B. A., Sigal, L., Sclaroff, S., & Saenko, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019062 · Cited by 307
Self-citationDeep Learning-Based Gait Recognition Using Smartphones in the Wild
Zou, Q., Wang, Y., Wang, Q., Zhao, Y., & Li, Q. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.2985628 · Cited by 305
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01742 · Cited by 305
Deep Learning-Based Gait Recognition Using Smartphones in the Wild
Zou, Q., Wang, Y., Wang, Q., Zhao, Y., & Li, Q. (2018) · arXiv · doi:10.48550/arxiv.1811.00338 · Cited by 305
Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
Wen, H., Shi, J., Zhang, Y., Lu, K.-H., Cao, J., & Liu, Z. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx268 · Cited by 303
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., & Huang, T. (2018) · arXiv · doi:10.48550/arxiv.1809.03327 · Cited by 301
“[1] pro-pose long-term recurrent convolution network for activity recognition and video captioning.”
BackgroundRohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C., Larochelle, H., Courville, A., & Schiele, B. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0987-1 · Cited by 300
Self-citationReading Scene Text in Deep Convolutional Sequences
He, P., Huang, W., Qiao, Y., Loy, C., & Tang, X. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10465 · Cited by 300
Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016666 · Cited by 299
Action Recognition using Visual Attention
Sharma, S., Kiros, R., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04119 · Cited by 298
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0965-7 · Cited by 298
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A. J., & Zisserman, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06573 · Cited by 291
PassGAN: A Deep Learning Approach for Password Guessing
Hitaj, B., Gasti, P., Ateniese, G., & Perez-Cruz, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-21568-2_11 · Cited by 290
AutoLoc: Weakly-Supervised Temporal Action Localization in Untrimmed Videos
Shou, Z., Gao, H., Zhang, L., Miyazawa, K., & Chang, S.-F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_10 · Cited by 288
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Pigou, L., van den Oord, A., Dieleman, S., Van Herreweghe, M., & Dambre, J. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-016-0957-7 · Cited by 290
“to max-pooling). (e) Architecture with added temporal convolutions and bidirectional recurrence. speech recognition (Hannun et al., 2014), image captioning (Vinyals et al., 2015) and video narration (Donahue et al., 2015). 3 ARCHITECTURES In this section, we briefly describe the different architectures we investigate f…”
MethodImage Captioning with Deep Bidirectional LSTMs
Wang, C., Yang, H., Bartz, C., & Meinel, C. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2964299 · Cited by 287
Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
Hasani, B., & Mahoor, M. H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.282 · Cited by 287
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R., Saenko, K., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.8 · Cited by 286
Self-citationGeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server
Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B., & Xing, E. P. (2016) · Proceedings of the Eleventh European Conference on Computer Systems · doi:10.1145/2901318.2901323 · Cited by 286
Recurrent Fusion Network for Image Captioning
Jiang, W., Ma, L., Jiang, Y.-G., Liu, W., & Zhang, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_31 · Cited by 286
StyleNet: Generating Attractive Visual Captions with Styles
Gan, C., Gan, Z., He, X., Gao, J., & Deng, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.108 · Cited by 283
Zreik, M., van Hamersvelt, R. W., Wolterink, J. M., Leiner, T., Viergever, M. A., & Isgum, I. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2883807 · Cited by 282
Associating neural word embeddings with deep image representations using Fisher Vectors
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299073 · Cited by 281
MARS: Motion-Augmented RGB Stream for Action Recognition
Crasto, N., Weinzaepfel, P., Alahari, K., & Schmid, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00807 · Cited by 281
Learning to Track for Spatio-Temporal Action Localization
Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.362 · Cited by 278
MiCT: Mixed 3D/2D Convolutional Tube for Human Action Recognition
Zhou, Y., Sun, X., Zha, Z.-J., & Zeng, W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00054 · Cited by 278
Self-Supervised Video Hashing With Hierarchical Binary Auto-Encoder
Song, J., Zhang, H., Li, X., Gao, L., Wang, M., & Hong, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2814344 · Cited by 276
Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487478 · Cited by 271
“We consider RNNs in multimodal setting, which is related to the recent use of RNNs in image-captioning [8].”
MethodRecurrent Multimodal Interaction for Referring Image Segmentation
Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., & Yuille, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.143 · Cited by 272
Spatiotemporal Pyramid Network for Video Action Recognition
Wang, Y., Long, M., Wang, J., & Yu, P. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.226 · Cited by 272
Two-Stream 3-D convNet Fusion for Action Recognition in Videos With Arbitrary Size and Length
Wang, X., Gao, L., Wang, P., Sun, X., & Liu, X. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2749159 · Cited by 263
“Moreover, CNN-based [28], [29] and RNN-based [24], [26] models are with great ability to model sequential features, each of which has time relationship with adjacent points.”
BackgroundInfluentialHybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2018) · arXiv · doi:10.48550/arxiv.1805.08298 · Cited by 271
“On both datasets, we compare with four state-of-the-art image captioning models: CNN-RNN [35], LRCN [9], AdaAtt [24], and Att2in [29].”
BackgroundMethodMultimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM
Zhu, G., Zhang, L., Shen, P., & Song, J. (2017) · IEEE Access · doi:10.1109/access.2017.2684186 · Cited by 270
Describing Video With Attention-Based Bidirectional LSTM
Bin, Y., Yang, Y., Shen, F., Xie, N., Shen, H. T., & Li, X. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2831447 · Cited by 270
Revisiting Video Saliency: A Large-Scale Benchmark and a New Model
Wang, W., Shen, J., Guo, F., Cheng, M.-M., & Borji, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00514 · Cited by 268
Comparative study of landslide susceptibility mapping with different recurrent neural networks
Wang, Y., Fang, Z., Wang, M., Peng, L., & Hong, H. (2020) · Computers & Geosciences · doi:10.1016/j.cageo.2020.104445 · Cited by 268
Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey
Mohammadi Foumani, N., Miller, L., Tan, C. W., Webb, G. I., Forestier, G., & Salehi, M. (2024) · ACM Computing Surveys · doi:10.1145/3649448 · Cited by 268
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., & Tao, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019127 · Cited by 281
“These advances stimulated research into bridging the semantic connections between vision and language, such as in visual captioning (Donahue et al. 2015; Xu et al. 2015), visual grounding (Rohrbach et al. 2016; Chen, Kovvuri, and Nevatia 2017; Yu et al. 2018b) and visual question answering…”
BackgroundWhat makes ImageNet good for transfer learning?
Huh, M., Agrawal, P., & Efros, A. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.08614 · Cited by 266
Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs
Koller, O., Zargaran, S., & Ney, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.364 · Cited by 266
Beyond Frame-level CNN: Saliency-Aware 3-D CNN With LSTM for Video Action Recognition
Wang, X., Gao, L., Song, J., & Shen, H. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2611485 · Cited by 265
Listen to Look: Action Recognition by Previewing Audio
Gao, R., Oh, T.-H., Grauman, K., & Torresani, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01047 · Cited by 262
Reddy, B., Kim, Y.-H., Yun, S., Seo, C., & Jang, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.59 · Cited by 260
MMTM: Multimodal Transfer Module for CNN Fusion
Vaezi Joze, H. R., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01330 · Cited by 259
Bagautdinov, T., Alahi, A., Fleuret, F., Fua, P., & Savarese, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.365 · Cited by 258
Huang, X., Sun, J., & Sun, J. (2018) · Transportation Research Part C: Emerging Technologies · doi:10.1016/j.trc.2018.07.022 · Cited by 257
Deep Learning-Based Weather Prediction: A Survey
Ren, X., Li, X., Ren, K., Song, J., Xu, Z., Deng, K., & Wang, X. (2021) · Big Data Research · doi:10.1016/j.bdr.2020.100178 · Cited by 257
RWF-2000: An Open Large Scale Video Database for Violence Detection
Cheng, M., Cai, K., & Li, M. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412502 · Cited by 255
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
Chen, K., Wang, J., Chen, L.-C., Gao, H., Xu, W., & Nevatia, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05960 · Cited by 254
Metadata
- Authors
- Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K.
- Affiliations
- University of California, Berkeley (US)
- International Computer Science Institute (US)
- The University of Texas at Austin (US)
- University of Massachusetts Lowell (US)
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 2625-2634
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1947481528
- Cited by
- 5,228
- References
- 49
- Citations by year
- 2026: 149 · 2025: 246 · 2024: 274 · 2023: 377 · 2022: 412 · 2021: 607 · 2020: 633 · 2019: 758 · 2018: 749 · 2017: 607 · 2016: 305 · 2015: 99
- FWCI
- 238.45
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Funders
- National Science Foundation
- Nvidia
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- recurrent convolutional networks, video recognition, image description generation, image retrieval, video narration, long-range dependencies, temporal dynamics, end-to-end learning, spatiotemporal receptive fields, visual representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.