CoChat
Sign Up
Cited by 1,133Advanced Graph Neural NetworksHuman Pose and Action RecognitionTraffic Prediction and Management Techniques

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Open access
Jain, Ashesh, Zamir, Amir R., Savarese, Silvio, Saxena, Ashutosh·2016·2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2016.573
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatiotemporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks (RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes Structural-RNN (S-RNN), a generic method that transforms an arbitrary spatio-temporal graph into a feedforward, jointly trainable mixture of RNNs via factor decomposition and factor sharing. Across human motion forecasting, human-object activity detection/anticipation, and driver maneuver anticipation, S-RNN reports improvements over unstructured RNNs and non-deep structured baselines.

Key findings

  • S-RNN casts any st-graph \(G=(V,E_S,E_T)\) into a rich, scalable RNN mixture by representing node factors with nodeRNNs and edge factors with edgeRNNs, connected as a bipartite graph when factors are neighbors.
  • On H3.6m human motion forecasting, S-RNN improves short-term angle error over LSTM-3LR and ERD; e.g., walking at 80 ms: S-RNN 1.08 vs LSTM-3LR 1.18 and ERD 1.30; eating at 80 ms: 1.35 vs 1.36 and 1.66; smoking at 80 ms: 1.90 vs 2.05 and 2.34; discussion at 80 ms: 1.67 vs 2.25 and 2.67.
  • On CAD-120 activity detection/anticipation, S-RNN reports higher F1-scores than Koppula et al.; multi-task S-RNN reaches 91.1% affordance detection and 80.9% object-affordance anticipation, versus 81.5% and 36.7% for Koppula et al.
  • On 1100 miles of real-world driving data, S-RNN improves precision/recall over AIO-HMM across reported maneuver categories; e.g., lane change precision 92.7% vs 83.8% and recall 84.4% vs 79.2%.

Methods

  • The method takes an st-graph, semantically partitions nodes and edges, derives node and edge factors, represents each factor with an RNN, and connects edgeRNNs to nodeRNNs to form a feedforward, fully differentiable architecture trained jointly.
  • Evaluations cover three tasks: human motion modeling/forecasting on H3.6m mocap (downsampled by two, trained on 6 subjects, tested on subject S5, 50 seed frames, 100 forecast frames, walking/eating/smoking/discussion); human activity detection/anticipation on CAD-120 using node/edge features from Koppula et al.; and driver maneuver anticipation using features from Jain et al.
  • Architecture details include motion edgeRNNs of FC(256)-FC(256)-LSTM(512) and nodeRNNs of LSTM(512)-FC(256)-FC(100)-FC(•); activity edgeRNNs of a single LSTM(128) and nodeRNNs of LSTM(256)-softmax(•).

Limitations

  • The paper states that long-term motion forecasts (>500 ms) can significantly differ from ground truth due to stochasticity in human motion, so long-term forecast numbers are not a fair representative of algorithms’ modeling capabilities.
  • The paper states S-RNN is not probabilistic and is not meant to model the joint distribution over outputs; it learns dependencies via structural sharing of RNNs instead.
  • Evident from the text: ERD and LSTM-3LR were reproduced because the authors’ implementation was unavailable at submission, and no statistical significance tests are reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/cvpr.2016.573

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Findingnone given

    S-RNN significantly outperforms its unstructured (plain-vanilla) RNN counterparts.

    “in defence of structured approaches, we show S-RNN significantly outperforms its unstructured (plain-vanilla) RNN counterparts”
    Quote in the paperSubstantial claim
  • Findingown data

    S-RNN outperforms non-deep learning based structured counterparts on several diverse spatio-temporal problems.

    “in defence of RNNs, we show on several diverse spatio-temporal problems that modeling structure with S-RNN outperforms the non-deep learning based structured counterparts.”
    Quote in the paperSubstantial claim
  • Findingown data

    S-RNN shows improvement over the state-of-the-art with a large margin on a diverse set of problems ranging from human motion to object interactions.

    “The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin.”
    Quote in the paperSubstantial claim
  • Findingown data

    On aperiodic activities, ERD and LSTM-3LR struggle to model human motion.

    “On aperiodic activities, ERD and LSTM-3LR struggle to model human motion.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Without edgeRNNs the motion freezes to some mean standing position.

    “Without (w/o) edgeRNNs the motion freezes to some mean standing position.”
    Quote in the paperSubstantial claim
  • May combine two claims. 15
  • The quote only partly states this claim. 2
  • Classified as probably not a substantial claim. 7

  • May combine two claims.

    Findingown data

    By visualizing memory cells, S-RNN learns certain semantic sub-motions, and its modularity was demonstrated by generating novel human motion.

    “By visualizing the memory cells we showed that S-RNN learns certain semantic sub-motions, and demonstrated its modularity by generating novel human motion.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The cell fires positive on forward movement of the leg and negative on backward movement.

    “The cell fires positive (red color) on the forward movement of the leg and negative (blue color) on its backward movement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    S-RNN mimics the ground truth in the short-term and generates human-like motion in the long term.

    “S-RNN therefore mimics the ground truth in the short-term and generates human like motion in the long term.”
    • Also stated in: § 5. Conclusion · plain
      “S-RNN, on the other hand, mimics the ground truth in the short-term and generates human like motion in the long term.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    On anticipating object affordance S-RNN F1-score is 44% more than [31], and 7% more on detection.

    “On anticipating object affordance S-RNN F1-score is 44% more than [31], and 7% more on detection.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [31]

      Not checked in the free run.

  • The quote only partly states this claim.

    Findingown data

    S-RNN learns semantic concepts, as revealed by visualizing its memory cells.

    “We also study complexity and convergence properties of S-RNN and provide further experimental insights by visualizing its memory cells that reveals some cells interestingly represent certain semantic operations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Discussion is one of the most challenging aperiodic activity for all algorithms.

    “We also observe that discussion is one of the most challenging aperiodic activity for all algorithms.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both S-RNN and ERD converge to similar training error, however S-RNN generalizes better with a smaller test error for next step prediction.

    “Both S-RNN and ERD converge to similar training error, however S-RNN generalizes better with a smaller test error for next step prediction.”
    • Also stated in: § Visualization of memory cells. · plain
      “S-RNN generalizes better than ERD with a smaller test error.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    S-RNN significantly improves over Koppula et al. on both anticipation and detection.

    “S-RNN significantly improves over Koppula et al. on both anticipation [31] and detection [29].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [31]

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    S-RNN stays close to the ground-truth in the short-term and generates human-like motion in the long-term on the 'eating' activity.

    “S-RNN stays close to the ground-truth in the short-term and generates human like motion in the long-term.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Spatio-temporal graphs are a popular general tool for representing high-level spatio-temporal structures.

    “Spatio-temporal graphs (st-graphs) are a popular [39,37,4,11,32,65,22] general tool for representing such high-level spatio-temporal structures.”
    Quote in the paperNot a substantial claim0 of 7 sources checked
    • cites [39]

      Not checked in the free run.

    • cites [37]

      Not checked in the free run.

    • cites [4]

      Not checked in the free run.

    • cites [11] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [65] source

      Not checked in the free run.

    • cites [22]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Since the number of objects varies with environment, factor sharing between the object nodes and the human-object edges becomes crucial.

    “Since the number of objects varies with environment, factor sharing between the object nodes and the human-object edges becomes crucial.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The number of objects varies with environment.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The overall S-RNN architecture is compact because edgeRNNs are shared between nodeRNNs, and the number of semantic categories are usually few in context-rich applications.

    “The overall S-RNN architecture is compact because the edgeRNNs are shared between the nodeRNNs, and the number of semantic categories are usually few in context-rich applications.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The number of semantic categories is usually few in context-rich applications.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    S-RNN does not have any Markov assumptions like spatio-temporal CRF, and therefore, it better models the long-time dependencies needed for anticipation.

    “S-RNN does not have any Markov assumptions like spatio-temporal CRF, and therefore, it better models the long-time dependencies needed for anticipation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The number of RNNs depends on the number of semantically similar nodes in the st-graph.

    “The number of RNNs depends on the number of semantically similar nodes in the st-graph.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Such modularity is of interest and has been explored to generate diverse motion styles.

    “Such modularity is of interest and has been explored to generate diverse motion styles [55].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In S-RNN, R_V2 and R_E1 handle all object nodes and human-object edges respectively, allowing a fixed S-RNN architecture to handle varying size st-graphs.

    “In S-RNN, R V2 and R E1 handles all the object nodes and the human-object edges respectively.This allows our fixed S-RNN architecture to handle varying size st-graphs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    S-RNN is scalable and applicable to any problem expressed over st-graphs.

    “We make use of factor graph, and factor sharing in order to obtain an RNN mixture that is scalable and applicable to any problem expressed over st-graphs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    S-RNN architecture is modular and enables generating hybrid human motions.

    “We now present several insights into S-RNN architecture and demonstrate the modularity of the architecture which enables it to generate hybrid human motions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    S-RNN is not probabilistic and is not meant to model the joint distribution over the outputs.

    “Unlike CRF, our approach is not probabilistic and is not meant to model the joint distribution over the outputs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The importance of edgeRNNs is signified and the design is validated.

    “This signifies the importance of edgeRNNs and also validates our design.”
    • Also stated in: § Turns · plain
      “The table also shows the importance of edgeRNNs in handling spatio-temporal components.”
    Quote in the paperSubstantial claim
20 of 20

  • The paper relies on this without support.

    RNNs lack a high-level and intuitive spatio-temporal structure.

    “However, they lack a high-level and intuitive spatio-temporal structure though they have been shown to be successful at modeling long sequences [49,43,52].”

  • The paper relies on this without support.

    In human-object interaction, the number of object nodes varies with the environment.

    “For example, in human-object interaction the number of object nodes vary with the environment.”

  • The paper relies on this without support.

    In the H3.6m dataset, significant parts of eating, smoking, and discussion activities are aperiodic, while walking activity is mostly periodic.

    “In H3.6m data set, significant parts of eating, smoking, and discussion activities are aperiodic, while walking activity is mostly periodic.”

  • The paper relies on this without support.

    Human motion is stochastic such that long-term forecasts (>500ms) can significantly differ from ground truth while still being human-like.

    “Due to stochasticity in human motion, longterm forecasts (> 500ms) can significantly differ from the ground truth but still depict human-like motion.”

  • The paper relies on this without support.

    The number of objects varies with environment.

    “Since the number of objects varies with environment”

  • The paper relies on this without support.

    The number of semantic categories is usually few in context-rich applications.

    “the number of semantic categories are usually few in context-rich applications.”
6 of 6

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. koppula, 2014, Anticipatory planning for humanrobot teams, ISER
  2. jain, 2013, Beyond geometric path planning: Learning context-driven user preferences via suboptimal feedback, ISRR
  3. sutton, 2011, An introduction to conditional random fields, Machine Learning, 4
  4. quattoni, 2004, Conditional random fields for object recognition, NIPS
  5. socher, 2011, Parsing Natural Scenes and Natural Language with Recursive Neural Networks, ICML

Cited by

Sorted by citations

Related

Authors
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A.
Affiliations
  • Cornell University (US)
  • Stanford University (US)
ORCID
Venue
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2016-06
Pages
5308-5317
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2016.573
OpenAlex
W2176353499
Cited by
1,133
References
68
Citations by year
2026: 37 · 2025: 76 · 2024: 122 · 2023: 130 · 2022: 132 · 2021: 203 · 2020: 173 · 2019: 125 · 2018: 85 · 2017: 42 · 2016: 6 · 2015: 1 · 2013: 1
FWCI
46.34
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Building and Construction › Traffic Prediction and Management Techniques
Keywords
spatio-temporal graph, recurrent neural networks, sequence modeling, human motion modeling, object interaction modeling, graph neural networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review