CoChat

Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

Donahue, Jeff, Hendricks, Lisa Anne, Rohrbach, Marcus, Venugopalan, Subhashini, Guadarrama, Sergio, Saenko, Kate, Darrell, Trevor · 2017 · IEEE Transactions on Pattern Analysis and Machine Intelligence

The DOI matches its Crossref record; 76 of the 78 references it cites were found in a registry or an index.

Abstract

Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent are effective for tasks involving sequences, visual and otherwise. We describe a class of recurrent convolutional architectures which is end-to-end trainable and suitable for large-scale visual understanding tasks, and demonstrate the value of these models for activity recognition, image captioning, and video description. In contrast to previous models which assume a fixed visual representation or perform simple temporal averaging for sequential processing, recurrent convolutional models are "doubly deep" in that they learn compositional representations in space and time. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Differentiable recurrent models are appealing in that they can directly map variable-length inputs (e.g., videos) to variable-length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent sequence models are directly connected to modern visual convolutional network models and can be jointly trained to learn temporal dynamics and convolutional perceptual representations. Our results show that such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined or optimized.

Overview

Integrity signals

Registry

Crossref · 10.1109/tpami.2016.2599174

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 76 references

Cited by

Items 501–600 of 1,001

Metadata

Authors
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T.
Affiliations
  • University of California, Berkeley (US)
  • International Computer Science Institute (US)
  • The University of Texas at Austin (US)
  • University of Massachusetts Lowell (US)
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2017-04-01
Volume
39
Issue
4
Pages
677-691
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
OpenAlex
W2508429489
Cited by
1,547
References
78
Citations by year
2026: 46 · 2025: 79 · 2024: 111 · 2023: 164 · 2022: 202 · 2021: 250 · 2020: 227 · 2019: 225 · 2018: 149 · 2017: 70
FWCI
102.49
Type
journal article
Language
en
Open access
closed
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • DARPA's
  • MSEE
  • SMISC
  • NSF (IIS-1427425, IIS-1212798)
  • Berkeley Vision and Learning Center
  • NVIDIA
  • FITweltweit-Program
  • German Academic Exchange Service (DAAD)
  • NDSEG
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
Keywords
long-term recurrent convolutional network, activity recognition, image captioning, audio description, long-range dependencies, temporal dynamics, compositional representations, end-to-end learning, visual description generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.