Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Many animals, and an increasing number of artificial agents, display sophisticated capabilities to perceive and manipulate objects. But human beings remain distinctive in their capacity for flexible, creative tool use—using objects in new ways to act on the world, achieve a goal, or solve a problem. To study this type of general physical problem solving, we introduce the Virtual Tools game. In this game, people solve a large range of challenging physical puzzles in just a handful of attempts. We propose that the flexibility of human physical problem solving rests on an ability to imagine the effects of hypothesized actions, while the efficiency of human search arises from rich action priors which are updated via observations of the world. We instantiate these components in the “sample, simulate, update” (SSUP) model and show that it captures human performance across 30 levels of the Virtual Tools game. More broadly, this model provides a mechanism for explaining how people condense general physical knowledge into actionable, task-specific plans to achieve flexible and efficient physical problem solving.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
The Cultural Origins of Human Cognition
Tomasello, M. (2009) · doi:10.2307/j.ctvjsf4jc · Cited by 6,368
An internal model for sensorimotor integration
Wolpert, D. M., Ghahramani, Z., & Jordan, M. I. (1995) · Science · doi:10.1126/science.7569931 · Cited by 3,553
Gick, M. L., & Holyoak, K. J. (1980) · Cognitive Psychology · doi:10.1016/0010-0285(80)90013-4 · Cited by 2,552 · cited 4×
“Retrieved 2019-05- 20, from http://arxiv.org/abs/1612.00341 (arXiv: 1612.00341) Chu, Y., & MacGregor, J. N. (2011, February).”
BackgroundInfluentialThe Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Object perception as Bayesian inference
Kersten, D., Mamassian, P., & Yuille, A. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142005 · Cited by 1,269
Interaction Networks for Learning about Objects, Relations and Physics
Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1612.00222 · Cited by 969 · cited 1×
“over continuous spaces used here. Further, 16 we imagine that improvements from the machine learning community on predictive model learning may be able to learn these simulators (Chang et al., 2016; Battaglia et al., 2016), and hope that the Tools game will spur further progress in this direction. 6.2.3 Updating and p…”
BackgroundInfluentialSimple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 2×
“SSUP is inspired by the theory of “problem solving as search” (Newell & Simon, 1972), as well as Dyna and other model-based policy optimization methods (Deisenroth, Neumann, Peters, et al., 2013; Sutton, 1991).”
MethodQuantifying Generalization in Reinforcement Learning
Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.02341 · Cited by 769
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708
Simulation as an engine of physical scene understanding
Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1306572110 · Cited by 681
Self-citationPrefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596 · cited 1×
“model-free settings (Finn, Abbeel, & Levine, 2017; Rakelly, Zhou, Finn, Levine, & Quillen, 2019; Gupta, Mendonca, Liu, Abbeel, & Levine, 2018; Schmidhuber, Zhao, & Schraudolph, 1998; Wang et al., 2018) . While such approaches seem promising for the Tools game, they have mostly been analyzed in settings…”
BackgroundEmergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424 · cited 1×
“Others learn from thousands of years of simulated experience (Baker et al., 2019), which is significantly longer than required for people.”
BackgroundRetrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217
Gotta Learn Fast: A New Benchmark for Generalization in RL
Nichol, A., Pfau, V., Hesse, C., Klimov, O., & Schulman, J. (2018) · arXiv · doi:10.48550/arxiv.1804.03720 · Cited by 204
Reconciling intuitive physics and Newtonian mechanics for colliding objects
Sanborn, A. N., Mansinghka, V. K., & Griffiths, T. L. (2013) · Psychological Review · doi:10.1037/a0031912 · Cited by 189
PHYRE: A New Benchmark for Physical Reasoning
Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., & Girshick, R. (2019) · arXiv · doi:10.48550/arxiv.1908.05656 · Cited by 37
Cited by
Emergent Tool Use From Multi-Agent Autocurricula
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019) · arXiv · doi:10.48550/arxiv.1909.07528 · Cited by 424
Tool Learning with Foundation Models
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z., Huang, Y., Xiao, C., Han, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.08354 · Cited by 184
(2024) · doi:10.1145/3704435 · Cited by 184
PHYRE: A New Benchmark for Physical Reasoning
Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L., & Girshick, R. (2019) · arXiv · doi:10.48550/arxiv.1908.05656 · Cited by 37
IntPhys 2019: A Benchmark for Visual Intuitive Physics Understanding
Riochet, R., Castro, M. Y., Bernard, M., Lerer, A., Fergus, R., Izard, V., & Dupoux, E. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3083839 · Cited by 19
Metadata
- Authors
- Allen, K. R., Smith, K. A., & Tenenbaum, J. B.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2020-11-23
- Volume
- 117
- Issue
- 47
- Pages
- 29302-29310
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W3037966979
- Cited by
- 119
- References
- 20
- FWCI
- 5.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.