Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
The DOI matches its Crossref record; 76 of the 78 references it cites were found in a registry or an index.
Abstract
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent are effective for tasks involving sequences, visual and otherwise. We describe a class of recurrent convolutional architectures which is end-to-end trainable and suitable for large-scale visual understanding tasks, and demonstrate the value of these models for activity recognition, image captioning, and video description. In contrast to previous models which assume a fixed visual representation or perform simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep" in that they learn compositional representations in space and time. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Differentiable recurrent models are appealing in that they can directly map variable-length inputs (e.g., videos) to variable-length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent sequence models are directly connected to modern visual convolutional network models and can be jointly trained to learn temporal dynamics and convolutional perceptual representations. Our results show that such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined or optimized.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Cited by
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
A survey of the recent architectures of deep convolutional neural networks
Khan, A., Sohail, A., Zahoora, U., & Qureshi, A. S. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-020-09825-6 · Cited by 3,137
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102
Generating Sentences from a Continuous Space
Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1002 · Cited by 2,061
Deep Facial Expression Recognition: A Survey
Li, S., & Deng, W. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2981446 · Cited by 1,753
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Predicting residential energy consumption using CNN-LSTM neural networks
Kim, T.-Y., & Cho, S.-B. (2019) · Energy · doi:10.1016/j.energy.2019.05.230 · Cited by 1,469
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollar, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298754 · Cited by 1,287
Deepfake Video Detection Using Recurrent Neural Networks
Guera, D., & Delp, E. J. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639163 · Cited by 1,285
Deep Learning in Medical Imaging: General Overview
Lee, J.-G., Jun, S., Cho, Y.-W., Lee, H., Kim, G. B., Seo, J. B., & Kim, N. (2017) · Korean Journal of Radiology · doi:10.3348/kjr.2017.18.4.570 · Cited by 1,276
Learning to Track at 100 FPS with Deep Regression Networks
Held, D., Thrun, S., & Savarese, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_45 · Cited by 1,199
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.494 · Cited by 1,137
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Deepvo: towards end-to-end visual odometry with deep recurrent convolutional neural networks
Wang, S., Clark, R., Wen, H., & Trigoni, N. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989236 · Cited by 970
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 968
Self-citationDelving Deeper into Convolutional Networks for Learning Video RepresentationsPreprint
Ballas, N., Yao, L., Pal, C., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06432 · Cited by 873
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
Learning Deep Structure-Preserving Image-Text Embeddings
Wang, L., Li, Y., & Lazebnik, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.541 · Cited by 760
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_28 · Cited by 727
Self-citationA Brief Review of Facial Emotion Recognition Based on Visual Information
Ko, B. (2018) · Sensors · doi:10.3390/s18020401 · Cited by 724
On the Automatic Generation of Medical Imaging Reports
Jing, B., Xie, P., & Xing, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1240 · Cited by 657
MovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., & Fidler, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.501 · Cited by 639
P-CNN: Pose-Based CNN Features for Action Recognition
Cheron, G., Laptev, I., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.368 · Cited by 633
Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging
Xu, Y., Hosny, A., Zeleznik, R., Parmar, C., Coroller, T., Franco, I., Mak, R. H., & Aerts, H. J. W. L. (2019) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-18-2495 · Cited by 628
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 620
Spatiotemporal Recurrent Convolutional Networks for Traffic Prediction in Transportation Networks
Yu, H., Wu, Z., Wang, S., Wang, Y., & Ma, X. (2017) · Sensors · doi:10.3390/s17071501 · Cited by 618
Patient Subtyping via Time-Aware LSTM Networks
Baytas, I. M., Xiao, C., Zhang, X., Wang, F., Jain, A. K., & Zhou, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3097997 · Cited by 613
End-to-End Learning of Action Detection from Frame Glimpses in Videos
Yeung, S., Russakovsky, O., Mori, G., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.293 · Cited by 585
Dynamic Image Networks for Action Recognition
Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., & Gould, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.331 · Cited by 571
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.496 · Cited by 564
Jointly Modeling Embedding and Translation to Bridge Video and Language
Pan, Y., Mei, T., Yao, T., Li, H., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.497 · Cited by 563
Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM Networks
Liu, J., Wang, G., Duan, L.-Y., Abdiyeva, K., & Kot, A. C. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2785279 · Cited by 537
Learning Two-Branch Neural Networks for Image-Text Matching Tasks
Wang, L., Li, Y., Huang, J., & Lazebnik, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2797921 · Cited by 535
A Hierarchical Deep Temporal Model for Group Activity Recognition
Ibrahim, M. S., Muralidharan, S., Deng, Z., Vahdat, A., & Mori, G. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.217 · Cited by 523
Differential Recurrent Neural Networks for Action Recognition
Veeriah, V., Zhuang, N., & Qi, G.-J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.460 · Cited by 516
A Comprehensive Survey of Vision-Based Human Action Recognition Methods
Zhang, H.-B., Zhang, Y.-X., Zhong, B., Lei, Q., Yang, L., Du, J.-X., & Chen, D.-S. (2019) · Sensors · doi:10.3390/s19051005 · Cited by 503
Where to Look: Focus Regions for Visual Question Answering
Shih, K. J., Singh, S., & Hoiem, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.499 · Cited by 493
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Wang, X., Wu, J., Chen, J., Li, L., Wang, Y.-F., & Wang, W. Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00468 · Cited by 486
Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model
Cornia, M., Baraldi, L., Serra, G., & Cucchiara, R. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2851672 · Cited by 474
A review of deep learning with special emphasis on architectures, applications and recent trends
Sengupta, S., Basak, S., Saikia, P., Paul, S., Tsalavoutis, V., Atiah, F., Ravi, V., & Peters, A. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105596 · Cited by 454
A review of Convolutional-Neural-Network-based action recognition
Yao, G., Lei, T., & Zhong, J. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.05.018 · Cited by 437
A Survey on Multiview Clustering
Chao, G., Sun, S., & Bi, J. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3065894 · Cited by 427
Guiding the Long-Short Term Memory Model for Image Caption Generation
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.277 · Cited by 426
Deep Learning: A Rapid and Efficient Route to Automatic Metasurface Design
Qiu, T., Shi, X., Wang, J., Li, Y., Qu, S., Cheng, Q., Cui, T., & Sui, S. (2019) · Advanced Science · doi:10.1002/advs.201900128 · Cited by 421
Object Hallucination in Image Captioning
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1437 · Cited by 420
Self-citationNúñez, J. C., Cabido, R., Pantrigo, J. J., Montemayor, A. S., & Vélez, J. F. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.033 · Cited by 419
IONet: Learning to Cure the Curse of Drift in Inertial Odometry
Chen, C., Lu, X., Markham, A., & Trigoni, N. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12102 · Cited by 369
Demrozi, F., Pravadelli, G., Bihorac, A., & Rashidi, P. (2020) · IEEE Access · doi:10.1109/access.2020.3037715 · Cited by 368
Relational Graph Attention NetworksPreprint
Busbridge, D., Sherburn, D., Cavallo, P., & Hammerla, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05811 · Cited by 364
Comparison of deep learning approaches to predict COVID-19 infection
Alakus, T. B., & Turkoglu, I. (2020) · Chaos, Solitons & Fractals · doi:10.1016/j.chaos.2020.110120 · Cited by 357
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., & Fei-Fei, L. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-017-1013-y · Cited by 354
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Bernardi, R., Cakici, R., Elliott, D., Erdem, A., Erdem, E., Ikizler-Cinbis, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 349
Computer vision algorithms and hardware implementations: A survey
Feng, X., Jiang, Y., Yang, X., Du, M., & Li, X. (2019) · Integration · doi:10.1016/j.vlsi.2019.07.005 · Cited by 349
Kim, T., & Kim, H. Y. (2019) · PLOS ONE · doi:10.1371/journal.pone.0212320 · Cited by 342
Pointer NetworksPreprint
Vinyals, O., Fortunato, M., & Jaitly, N. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03134 · Cited by 340
Deep Learning for Fall Detection: Three-Dimensional CNN Combined With LSTM on Video Kinematic Data
Lu, N., Wu, Y., Feng, L., & Song, J. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2808281 · Cited by 339
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks
Cho, K., Courville, A., & Bengio, Y. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2477044 · Cited by 338
Pairwise Relationship Guided Deep Hashing for Cross-Modal Retrieval
Yang, E., Deng, C., Liu, W., Liu, X., Tao, D., & Gao, X. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10719 · Cited by 331
Christopher, M., Belghith, A., Bowd, C., Proudfoot, J. A., Goldbaum, M. H., Weinreb, R. N., Girkin, C. A., Liebmann, J. M., & Zangwill, L. M. (2018) · Scientific Reports · doi:10.1038/s41598-018-35044-9 · Cited by 328
L3-Net: Towards Learning Based LiDAR Localization for Autonomous Driving
Lu, W., Zhou, Y., Wan, G., Hou, S., & Song, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00655 · Cited by 311
On Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks
Zhang, S., Liu, X., & Xiao, J. (2017) · 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2017.24 · Cited by 310
Multilevel Language and Vision Integration for Text-to-Clip Retrieval
Xu, H., He, K., Plummer, B. A., Sigal, L., Sclaroff, S., & Saenko, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019062 · Cited by 307
Self-citationNeural Encoding and Decoding with Deep Learning for Dynamic Natural Vision
Wen, H., Shi, J., Zhang, Y., Lu, K.-H., Cao, J., & Liu, Z. (2018) · Cerebral Cortex · doi:10.1093/cercor/bhx268 · Cited by 303
Comparison of Feature Learning Methods for Human Activity Recognition Using Wearable Sensors
Li, F., Shirahama, K., Nisar, M., Köping, L., & Grzegorzek, M. (2018) · Sensors · doi:10.3390/s18020679 · Cited by 302
Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016666 · Cited by 299
Actor-Transformers for Group Activity Recognition
Gavrilyuk, K., Sanford, R., Javan, M., & Snoek, C. G. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00092 · Cited by 297
Hybrid deep neural networks for face emotion recognition
Jain, N., Kumar, S., Kumar, A., Shamsolmoali, P., & Zareapoor, M. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.04.010 · Cited by 294
Gupta, S., & Malik, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04474 · Cited by 289
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
Tewel, Y., Shalev, Y., Schwartz, I., & Wolf, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01739 · Cited by 284
ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., & Tao, D. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019127 · Cited by 281
Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487478 · Cited by 271
Comparative study of landslide susceptibility mapping with different recurrent neural networks
Wang, Y., Fang, Z., Wang, M., Peng, L., & Hong, H. (2020) · Computers & Geosciences · doi:10.1016/j.cageo.2020.104445 · Cited by 268
Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
Duan, L., Liu, J., Yang, W., Huang, T., & Gao, W. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3016485 · Cited by 260
Host–Parasite: Graph LSTM-in-LSTM for Group Activity Recognition
Shu, X., Zhang, L., Sun, Y., & Tang, J. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978942 · Cited by 258
ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
Chen, K., Wang, J., Chen, L.-C., Gao, H., Xu, W., & Nevatia, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05960 · Cited by 254
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis
Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao, L., Lu, J., & Zhou, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00130 · Cited by 253
Learning to Answer Questions from Image Using Convolutional Neural Network
Ma, L., Lu, Z., & Li, H. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10442 · Cited by 253
Visual Madlibs: Fill in the Blank Description Generation and Question Answering
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.283 · Cited by 252
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports
Jing, B., Wang, Z., & Xing, E. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1657 · Cited by 252
Objective comparison of methods to decode anomalous diffusion
Muñoz-Gil, G., Volpe, G., Garcia-March, M. A., Aghion, E., Argun, A., Hong, C. B., Bland, T., Bo, S., Conejero, J. A., Firbas, N., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-26320-w · Cited by 251
Action Recognition with Dynamic Image Networks
Bilen, H., Fernando, B., Gavves, E., & Vedaldi, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2769085 · Cited by 251
Real-time Driver Drowsiness Detection for Android Application Using Deep Neural Networks Techniques
Jabbar, R., Al-Khalifa, K., Kharbeche, M., Alhajyaseen, W., Jafari, M., & Jiang, S. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.04.060 · Cited by 249
Multi-stream CNN: Learning representations based on human-related regions for action recognition
Tu, Z., Xie, W., Qin, Q., Poppe, R., Veltkamp, R. C., Li, B., & Yuan, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.01.020 · Cited by 247
Visual question answering: Datasets, algorithms, and future challenges
Kafle, K., & Kanan, C. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.06.005 · Cited by 245
Review of constraints on vision‐based gesture recognition for human–computer interaction
Chakraborty, B. K., Sarma, D., Bhuyan, M. K., & MacDorman, K. F. (2018) · IET Computer Vision · doi:10.1049/iet-cvi.2017.0052 · Cited by 243
End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks
Wang, S., Clark, R., Wen, H., & Trigoni, N. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917734298 · Cited by 238
Building energy prediction using artificial neural networks: A literature survey
Lu, C., Li, S., & Lu, Z. (2022) · Energy and Buildings · doi:10.1016/j.enbuild.2021.111718 · Cited by 236
Fully Connected Deep Structured Networks
Schwing, A. G., & Urtasun, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.02351 · Cited by 230
Connectionist Temporal Modeling for Weakly Supervised Action Labeling
Huang, D.-A., Fei-Fei, L., & Niebles, J. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_9 · Cited by 226
Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
Shetty, R., Rohrbach, M., Hendricks, L. A., Fritz, M., & Schiele, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.445 · Cited by 222
Self-citationLearnable pooling with Context Gating for video classification
Miech, A., Ivan Laptev, A., & Šivic, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06905 · Cited by 218
Detecting Events and Key Actors in Multi-person Videos
Ramanathan, V., Huang, J., Abu-El-Haija, S., Gorban, A., Murphy, K., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.332 · Cited by 217
Mueller, S. T., Hoffman, R. R., Clancey, W., Emrey, A., & Klein, G. (2019) · arXiv · doi:10.48550/arxiv.1902.01876 · Cited by 215
Knowledge Tracing with Sequential Key-Value Memory Networks
Abdelrahman, G., & Wang, Q. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331195 · Cited by 214
Multimodal Engagement Analysis From Facial Videos in the Classroom
Sümer, Ö., Goldberg, P., D’Mello, S. K., Gerjets, P. H., Trautwein, U., & Kasneci, E. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3127692 · Cited by 214
Learning to predict the cosmological structure formation
He, S., Li, Y., Feng, Y., Ho, S., Ravanbakhsh, S., Chen, W., & Póczos, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1821458116 · Cited by 213
ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
Liu, Y., Shen, C., Jin, L., He, T., Chen, P., Liu, C., & Chen, H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3107437 · Cited by 211
Metadata
- Authors
- Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T.
- Affiliations
- University of California, Berkeley (US)
- International Computer Science Institute (US)
- The University of Texas at Austin (US)
- University of Massachusetts Lowell (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2017-04-01
- Volume
- 39
- Issue
- 4
- Pages
- 677-691
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- OpenAlex
- W2508429489
- Cited by
- 1,547
- References
- 78
- Citations by year
- 2026: 46 · 2025: 79 · 2024: 111 · 2023: 164 · 2022: 202 · 2021: 250 · 2020: 227 · 2019: 225 · 2018: 149 · 2017: 70
- FWCI
- 102.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- DARPA's
- MSEE
- SMISC
- NSF (IIS-1427425, IIS-1212798)
- Berkeley Vision and Learning Center
- NVIDIA
- FITweltweit-Program
- German Academic Exchange Service (DAAD)
- NDSEG
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- long-term recurrent convolutional network, activity recognition, image captioning, audio description, long-range dependencies, temporal dynamics, compositional representations, end-to-end learning, visual description generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.