Iterative Visual Reasoning Beyond Convolutions
Open accessAbstract
We present a novel framework for iterative visual reasoning. Our framework goes beyond current recognition systems that lack the capability to reason beyond stack of convolutions. The framework consists of two core modules: a local module that uses spatial memory [4] to store previous beliefs with parallel updates; and a global graph-reasoning module. Our graph module has three components: a) a knowledge graph where we represent classes as nodes and build edges to encode different types of semantic relationships between them; b) a region graph of the current image where regions in the image are nodes and spatial relationships between these regions are edges; c) an assignment graph that assigns regions to classes. Both the local module and the global module roll-out iteratively and cross-feed predictions to each other to refine estimates. The final predictions are made by combining the best of both modules with an attention mechanism. We show strong performance over plain ConvNets, e.g. achieving an 8.4% absolute improvement on ADE [55] measured by per-class average precision. Analysis also shows that the framework is resilient to missing regions for reasoning.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Self-citationFeature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · see citing works
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582 · see citing works
Item-based collaborative filtering recommendation algorithms
Sarwar, B., Karypis, G., Konstan, J., & Riedl, J. (2001) · Proceedings of the 10th international conference on World Wide Web · doi:10.1145/371920.372071 · Cited by 9,135 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Self-citationHarnad, S. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90087-6 · Cited by 4,235 · cited 1× · see citing works
“However, symbols need to be grounded [15] before such systems are practically useful.”
BackgroundThe link‐prediction problem for social networks
Liben‐Nowell, D., & Kleinberg, J. M. (2007) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20591 · Cited by 3,063 · see citing works
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029 · see citing works
Self-citationWei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966 · see citing works
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908 · see citing works
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · see citing works
Self-citationTorralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206772 · Cited by 1,917 · see citing works
Semantic Understanding of Scenes Through the ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., & Torralba, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1140-0 · Cited by 1,831 · see citing works
Attention to Scale: Scale-Aware Semantic Image Segmentation
Chen, L.-C., Yang, Y., Wang, J., Xu, W., & Yuille, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.396 · Cited by 1,462 · see citing works
Newell, A. (1980) · Cognitive Science · doi:10.1207/s15516709cog0402_2 · Cited by 1,458 · see citing works
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271 · see citing works
Self-citationDeep Convolutional Networks on Graph-Structured Data
Henaff, M., Bruna, J., & LeCun, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.05163 · Cited by 1,255 · see citing works
Hobbs, J. R., Stickel, M. E., Appelt, D. E., & Martin, P. (1988) · Artificial Intelligence · doi:10.1016/0004-3702(93)90015-4 · Cited by 1,222 · see citing works
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,118 · see citing works
Self-citationImage retrieval using scene graphs
Johnson, J., Krishna, R., Stark, M., Li, L.-J., Shamma, D. A., Bernstein, M. S., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298990 · Cited by 1,101 · see citing works
Self-citationScene perception: Detecting and judging objects undergoing relational violations
Biederman, I., Mezzanotte, R. J., & Rabinowitz, J. C. (1982) · Cognitive Psychology · doi:10.1016/0010-0285(82)90007-x · Cited by 1,059 · see citing works
Parikh, D., & Grauman, K. (2011) · International Conference on Computer Vision · doi:10.1109/iccv.2011.6126281 · Cited by 901 · see citing works
FastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887 · see citing works
Context-based vision system for place and object recognition
Torralba, Murphy, M., Freeman, & Рубин (2003) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238354 · Cited by 867 · see citing works
Human Pose Estimation with Iterative Error Feedback
Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.512 · Cited by 805 · see citing works
Dynamic Memory Networks for Visual and Textual Question AnsweringPreprint
Xiong, C., Merity, S., & Socher, R. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1603.01417 · Cited by 770 · see citing works
Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation
Tu, Z., & Bai, X. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.186 · Cited by 578 · see citing works
Diagnosing Error in Object Detectors
Hoiem, D., Chodpathumwan, Y., & Dai, Q. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33712-3_25 · Cited by 520 · see citing works
An empirical study of context in object detection
Divvala, S., Hoiem, D., Hays, J. H., Efros, A. A., & Hebert, M. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206532 · Cited by 501 · see citing works
Random Walk Inference and Learning in A Large Scale Knowledge Base
Lao, N., Mitchell, T. M., & Cohen, W. W. (2018) · Figshare · doi:10.1184/r1/6476267 · Cited by 479 · see citing works
NEIL: Extracting Visual Knowledge from Web Data
Chen, X., Shrivastava, A., & Gupta, A. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.178 · Cited by 408 · see citing works
Self-citationLarge-Scale Object Classification Using Label Relation Graphs
Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K. P., Samy Bengio, A., Li, Y., Neven, H., & Adam, H. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_4 · Cited by 371 · see citing works
Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources
Wu, Q., Wang, P., Shen, C., Dick, A., & van den Hengel, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.500 · Cited by 362 · see citing works
The More You Know: Using Knowledge Graphs for Image Classification
Marino, K., Salakhutdinov, R., & Gupta, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.10 · Cited by 331 · see citing works
Self-citationBeyond Skip Connections: Top-Down Modulation for Object DetectionPreprint
Shrivastava, A., Sukthankar, R., Malik, J., & Gupta, A. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06851 · Cited by 315 · see citing works
Self-citationLearning Everything about Anything: Webly-Supervised Visual Concept Learning
Divvala, S. K., Farhadi, A., & Guestrin, C. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.412 · Cited by 267 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsPreprint
Krishna, R., Zhu, Y., Groth, O., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07332 · Cited by 248 · see citing works
Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks
Das, R., Neelakantan, A., Belanger, D., & McCallum, A. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1013 · Cited by 239 · see citing works
Discriminative Models for Multi-Class Object Layout
Desai, C., Ramanan, D., & Fowlkes, C. C. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0439-x · Cited by 208 · see citing works
Semantic Understanding of Scenes through the ADE20K DatasetPreprint
Zhou, B., Zhao, H., Puig, X., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.05442 · Cited by 190 · see citing works
Spatial Memory for Context Reasoning in Object Detection
Chen, X., & Gupta, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.440 · Cited by 172 · see citing works
Self-citationDeep Neural Networks with Massive Learned Knowledge
Hu, Z., Yang, Z., Salakhutdinov, R., et al. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1173 · Cited by 83 · see citing works
Top-Down Learning for Structured Labeling with Convolutional Pseudoprior
Xie, S., Huang, X., & Tu, Z. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_19 · Cited by 41 · see citing works
Building a Large-scale Multimodal Knowledge Base System for Answering Visual QueriesPreprint
Zhu, Y., Zhang, C., Ré, C., & Fei-Fei, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.05670 · Cited by 36 · see citing works
Self-citationChen, T., Wang, Z., Li, L., Li, D., Li, Z., Chang, X., Bian, P., Liang, G., Wang, Q., & Xie, T. (2025) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.ecoop.2025.6
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A Comprehensive Survey on Graph Neural Networks
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978386 · Cited by 11,000 · see citing works
Graph neural networks: A review of methods and applications
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020) · AI Open · doi:10.1016/j.aiopen.2021.01.001 · Cited by 6,880 · see citing works
MUREL: Multimodal Relational Reasoning for Visual Question Answering
Cadene, R., Ben-younes, H., Cord, M., & Thome, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00209 · Cited by 330 · see citing works
Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification
Xu, A., Yao, A., Li, A., Liang, A., & Zhang, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00675 · Cited by 228 · see citing works
Visual Semantic Information Pursuit: A Survey
Liu, D., Bober, M., & Kittler, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2950025 · Cited by 39 · see citing works
Multi-Modal Dynamic Graph Transformer for Visual Grounding
Chen, S., & Li, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01509 · Cited by 28 · see citing works
Wu (2021) · doi:10.1523/eneuro.0267-20.2020 · Cited by 6 · see citing works
- Authors
- Chen, X., Li, L.-J., Fei-Fei, L., & Gupta, A.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 7239-7248
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2964094751
- Semantic Scholar
- 57c36f13b188816051a27478e2f56bb284f4fb13
- Cited by
- 236
- References
- 56
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 15 · 2023: 30 · 2022: 31 · 2021: 48 · 2020: 55 · 2019: 38 · 2018: 8
- FWCI
- 11.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Multidisciplinary University Research Initiative
- Office of Naval Research (N00014-16-1-2007)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- spatial memory, graph neural networks, knowledge graphs, region graph, attention mechanism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review