CoChat
Sign Up
PreprintCited by 1,281Multimodal Machine Learning ApplicationsLogic, Reasoning, and KnowledgeTopic Modeling

A simple neural network module for relational reasoning

Open access
Santoro, Adam, Raposo, David, Barrett, David G. T., Malinowski, Mateusz, Pascanu, Razvan, Battaglia, Peter, Lillicrap, Timothy·2017·arXiv
DOI 10.48550/arxiv.1706.01427arXiv 1706.01427
Open this paper in CoChat — free accountOpen PDF

Abstract

Relational reasoning is a central component of generally intelligent behavior, but has proven difficult for neural networks to learn. In this paper we describe how to use Relation Networks (RNs) as a simple plug-and-play module to solve problems that fundamentally hinge on relational reasoning. We tested RN-augmented networks on three tasks: visual question answering using a challenging dataset called CLEVR, on which we achieve state-of-the-art, super-human performance; text-based question answering using the bAbI suite of tasks; and complex reasoning about dynamic physical systems. Then, using a curated dataset called Sort-of-CLEVR we show that powerful convolutional networks do not have a general capacity to solve relational questions, but can gain this capacity when augmented with RNs. Our work shows how a deep learning architecture equipped with an RN module can implicitly discover and learn to reason about entities and their relations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.01427

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Viewing SimplE as a single-vector-per-entity model makes it easily integrable (or compatible) with other embedding models in knowledge graph completion, computer vision and natural language processing.

    “Viewing SimplE as a single-vector-per-entity model makes it easily integrable (or compatible) with other embedding models (in knowledge graph completion, computer vision and natural language processing) such as [35, 47, 36].”

    From SimplE Embedding for Link Prediction in Knowledge Graphs (2018) · cites this paper as [35]

    The Visual question answering section (p. 10) says the paper’s Relation Network module can readily be combined with CNNs and LSTMs, supporting the citation as an example of an integrable model used with vision and language inputs. It does not discuss SimplE, single-vector-per-entity representations, or compatibility with knowledge-graph-completion embedding models. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 43

  1. [1][1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV , 2015. -
  2. [2][2] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In NIPS , 2016. -
  3. [3][3] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP , 2013. -
  4. [4][4] Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. Chains of reasoning over entities, relations, and text using recurrent neural networks. arXiv:1607.01426 , 2016. - · arXiv
  5. [5][5] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv:1606.01847 , 2016. - · arXiv
  6. [6][6] Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In NIPS , 2015. -
  7. [7][7] Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv:1609.05518 , 2016. - · arXiv
  8. [8][8] Dave Golland, Percy Liang, and Dan Klein. A game-theoretic approach to generating spatial descriptions. In EMNLP , 2010. -
  9. [10][10] Sergio Guadarrama, Lorenzo Riano, Dave Golland, Daniel Gouhring, Yangqing Jia, Dan Klein, Pieter Abbeel, and Trevor Darrell. Grounding spatial relations for human-robot interaction. In IROS , 2013. -
  10. [11][11] Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena , 42(1-3):335-346, 1990. -
  11. [12][12] Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arXiv:1506.05163 , 2015. - · arXiv
  12. [13][13] Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. In ICLR , 2017. -
  13. [14][14] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. arXiv:1704.05526 , 2017. - · arXiv
  14. [15][15] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR , 2017. -
  15. [16][16] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. arXiv:1705.03633 , 2017. - · arXiv
  16. [17][17] Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. arXiv:1703.09684 , 2017. - · arXiv
  17. [18][18] Charles Kemp and Joshua B Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences , 105(31):10687-10692, 2008. -
  18. [19][19] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv:1609.02907 , 2016. - · arXiv
  19. [20][20] Jayant Krishnamurthy and Thomas Kollar. Jointly learning to parse and perceive: Connecting natural language to the physical world. TACL , 2013. -
  20. [22][22] Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. Inducing probabilistic ccg grammars from logical form with higher-order unification. In EMNLP , 2010. -
  21. [23][23] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. arXiv:1604.00289 , 2016. - · arXiv
  22. [24][24] Tian Lan, Weilong Yang, Yang Wang, and Greg Mori. Image retrieval with structured object queries using latent ranking svm. In ECCV , 2012. -
  23. [25][25] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015. -
  24. [26][26] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. ICLR , 2016. -
  25. [29][29] Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS , 2014. -
  26. [30][30] Mateusz Malinowski and Mario Fritz. A pooling approach to modelling spatial relations for image retrieval and annotation. arXiv:1411.5190 , 2014. - · arXiv
  27. [31][31] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A deep learning approach to visual question answering. arXiv:1605.02697 , 2016. - · arXiv
  28. [32][32] Allen Newell. Physical symbol systems. Cognitive science , 4(2):135-183, 1980. -
  29. [33][33] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In ICML , 2016. -
  30. [34][34] Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Tim Lillicrap. Scaling memory-augmented neural networks with sparse reads and writes. In NIPS , 2016. -
  31. [35][35] David Raposo, Adam Santoro, David Barrett, Razvan Pascanu, Timothy Lillicrap, and Peter Battaglia. Discovering objects and their relations from entangled scene representations. arXiv:1702.05068 , 2017. - · arXiv
  32. [36][36] Mengye Ren, Ryan Kiros, and Richard Zemel. Image question answering: A visual semantic embedding model and a new dataset. In NIPS , 2015. -
  33. [37][37] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 2009. -
  34. [38][38] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI , 2011. -
  35. [40][40] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS , 2012. -
  36. [41][41] Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698 , 2015. - · arXiv
  37. [42][42] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In ICLR , 2015. -
  38. [43][43] Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In ICML , 2016. -
  39. [44][44] Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV , 2016. -
  40. [45][45] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML , 2015. -
  41. [46][46] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR , 2016. -
  42. [47][47] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnab´ as P´ oczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. arXiv:1703.06114 , 2017. - · arXiv
  43. [48][48] John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming. In AAAI , 1996.

Cited by

Sorted by citations

Authors
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T.
Affiliations
  • Google (United States) (US)
  • Cold Spring Harbor Laboratory (US)
  • Max Planck Society (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1706.01427
arXiv
1706.01427
OpenAlex
W2624614404
Cited by
1,281
References
0
Citations by year
2026: 1 · 2025: 40 · 2024: 83 · 2023: 115 · 2022: 112 · 2021: 253 · 2020: 282 · 2019: 255 · 2018: 121 · 2017: 19
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
relational reasoning, relation network, visual question answering, CLEVR, bAbI, question answering, physical reasoning, entity relationships, convolutional neural networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review