The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
IQA: Visual Question Answering in Interactive Environments
Open accessAbstract
We introduce Interactive Question Answering (IQA), the task of answering questions that require an autonomous agent to interact with a dynamic visual environment. IQA presents the agent with a scene and a question, like: "Are there any apples in the fridge?" The agent must navigate around the scene, acquire visual understanding of scene elements, interact with objects (e.g. open refrigerators) and plan for a series of actions conditioned on the question. Popular reinforcement learning approaches with a single controller perform poorly on IQA owing to the large and diverse state space. We propose the Hierarchical Interactive Memory Network (HIMN), consisting of a factorized set of controllers, allowing the system to operate at multiple levels of temporal abstraction. To evaluate HIMN, we introduce IQUAD V1, a new dataset built upon AI2-THOR [35], a simulated photo-realistic environment of configurable indoor scenes with interactive objects. IQUAD V1 has 75,000 questions, each paired with a unique scene configuration. Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1. For sample questions and results, please view our video: https://youtu.be/pXd3C-1jr98.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
Self-citationAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2463671 · Cited by 6,778
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving
Fikes, R. E., & Nilsson, N. J. (1971) · Artificial Intelligence · doi:10.1016/0004-3702(71)90010-5 · Cited by 4,882
LSD-SLAM: Large-scale direct monocular SLAM
Engel, J., Schöps, T., & Cremers, D. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10605-2_54 · Cited by 3,851
ORB-SLAM: A Versatile and Accurate Monocular SLAM System
Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2015) · Zaguan (Universidad de Zaragoza) · Cited by 3,153
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
The vector field histogram-fast obstacle avoidance for mobile robots
Borenstein, J., & Koren, Y. (1991) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.88137 · Cited by 2,301
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Deeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 1,982
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,815
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Real-time simultaneous localisation and mapping with a single camera
Davison (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238654 · Cited by 1,733
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533
Self-citationHierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
Real-time obstacle avoidance for fast mobile robots
Borenstein, J., & Koren, Y. (1989) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.44033 · Cited by 1,258
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.12 · Cited by 908
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
MovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., & Fidler, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.501 · Cited by 639
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.9 · Cited by 631
Reinforcement Learning with Hierarchies of Machines
Parr, R., & Russell, S. (1997) · Cited by 618
AI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578
Self-citationVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570
Policy gradient reinforcement learning for fast quadrupedal locomotion
Kohl, N., & Stone, P. (2004) · IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 · doi:10.1109/robot.2004.1307456 · Cited by 564
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaskowski, W. (2016) · 2016 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 541
Learning to Reason: End-to-End Module Networks for Visual Question Answering
Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 517
Cognitive Mapping and Planning for Visual Navigation
Gupta, S., Davidson, J., Levine, S., Sukthankar, R., & Malik, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.769 · Cited by 500
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.149 · Cited by 488
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487
Hierarchical task and motion planning in the now
Kaelbling, L. P., & Lozano-Perez, T. (2011) · 2011 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980391 · Cited by 467
Inferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 467
Combined task and motion planning through an extensible planner-independent interface layer
Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S., & Abbeel, P. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6906922 · Cited by 459
Visual question answering: A survey of methods and datasets
Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A., & van den Hengel, A. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.05.001 · Cited by 438
High speed obstacle avoidance using monocular vision and reinforcement learning
Michels, J., Saxena, A., & Ng, A. Y. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102426 · Cited by 394
A Deep Hierarchical Approach to Lifelong Learning in Minecraft
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., & Mannor, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10744 · Cited by 299
Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning
Wu, J., Yildirim, I., Lim, J. J., Freeman, W. T., & Tenenbaum, J. B. (2016) · Cited by 264
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244
Self-citationCognitive Mapping and Planning for Visual Navigation
Gupta, S., Tolani, V., Davidson, J., Levine, S., Sukthankar, R., & Malik, J. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01236-7 · Cited by 225
Playing for Data: Ground Truth from Computer GamesPreprint
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.02192 · Cited by 188
Learning appearance in virtual scenarios for pedestrian detection
Marin, J., Vazquez, D., Geronimo, D., & Lopez, A. M. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540218 · Cited by 168
DeepStory: Video Story QA by Deep Embedded Memory Networks
Kim, K.-M., Heo, M.-O., Choi, S.-H., & Zhang, B. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/280 · Cited by 167
Trajectory Optimization using Reinforcement Learning for Map Exploration
Kollar, T., & Roy, N. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364907087426 · Cited by 138
Gated-Attention Architectures for Task-Oriented Language Grounding
Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., & Salakhutdinov, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11832 · Cited by 135
Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images
Mottaghi, R., Bagherinezhad, H., Rastegari, M., & Farhadi, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.383 · Cited by 131
Self-citationVisual Semantic Planning Using Deep Successor Representations
Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L., Gupta, A., Mottaghi, R., & Farhadi, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.60 · Cited by 129
Self-citationPlay and Learn: Using Video Games to Train Computer Vision Models
Shafaei, A., Little, J. J., & Schmidt, M. (2016) · British Machine Vision Conference (BMVC) · doi:10.5244/c.30.26 · Cited by 114
Reactive navigation in outdoor environments using potential fields
Haddad, H., Khatib, M., Lacroix, S., & Chatila, R. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1998.677268 · Cited by 108
Sim, R., & Little, J. J. (2006) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2006.282485 · Cited by 103
A Diagram Is Worth A Dozen ImagesPreprint
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.07396 · Cited by 98
Self-citationMarioQA: Answering Questions by Watching Gameplay Videos
Mun, J., Seo, P. H., Jung, I., & Han, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.312 · Cited by 95
On-line map building and navigation for autonomous mobile robots
Oriolo, G., Vendittelli, M., & Ulivi, G. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1995.525695 · Cited by 88
Gated-Attention Architectures for Task-Oriented Language Grounding
Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., & Salakhutdinov, R. (2017) · arXiv · doi:10.48550/arxiv.1706.07230 · Cited by 82
Vision-based 3-D trajectory tracking for unknown environments
Saeedi, P., Lawrence, P. D., & Lowe, D. G. (2006) · IEEE Transactions on Robotics · doi:10.1109/tro.2005.858856 · Cited by 80
“What Happens If...” Learning to Predict the Effect of Forces in Images
Mottaghi, R., Rastegari, M., Gupta, A., & Farhadi, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_17 · Cited by 79
Self-citationIntegrating symbolic and geometric planning for mobile manipulation
Dornhege, C., Gissler, M., Teschner, M., & Nebel, B. (2009) · IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) · doi:10.1109/ssrr.2009.5424160 · Cited by 76
Made to measure: Bespoke landmarks for 24-hour, all-weather localisation with a camera
Linegar, C., Churchill, W., & Newman, P. M. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487208 · Cited by 72
Semantic Pose Using Deep Networks Trained on Synthetic RGB-D
Papon, J. A., & Schoeler, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.95 · Cited by 58
3-D Object Map Building Using Dense Object Models with SIFT-based Recognition Features
Tomono, M. (2006) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2006.282312 · Cited by 48
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2016) · arXiv · doi:10.48550/arxiv.1612.06890 · Cited by 40
Fast, robust, continuous monocular egomotion computation
Jaegle, A., Phillips, S., & Daniilidis, K. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487206 · Cited by 31
A guide to vision-based map building
Wooden, D. (2006) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2006.1638021 · Cited by 29
Using Classical Planners for Tasks with Continuous Operators in Robotics
Srivastava, S., Riano, L., Russell, S., & Abbeel, P. (2013) · Cited by 27
Semantic Pose using Deep Networks Trained on Synthetic RGB-DPreprint
Papon, J. A., & Schoeler, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.00835 · Cited by 27
Understanding Grounded Language Learning AgentsPreprint
Hill, F., Hermann, K. M., Blunsom, P., & Clark, S. C. (2017) · arXiv (Cornell University) · Cited by 24
Communicating Hierarchical Neural Controllers for Learning Zero-shot Task Generalization
Oh, J., Singh, S. P., Lee, H., & Kohli, P. (2017) · Cited by 1
Cited by
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
Self-citationA Metaverse: Taxonomy, Components, Applications, and Open Challenges
Park, S.-M., & Kim, Y.-G. (2022) · IEEE Access · doi:10.1109/access.2021.3140175 · Cited by 1,824
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00331 · Cited by 703
Self-citationAI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578
Self-citation
Metadata
- Authors
- Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., & Farhadi, A.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 4089-4098
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2772262724
- Cited by
- 352
- References
- 71
- FWCI
- 16.00
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.