Interplay of approximate planning strategies
Open accessAbstract
Humans routinely formulate plans in domains so complex that even the most powerful computers are taxed. To do so, they seem to avail themselves of many strategies and heuristics that efficiently simplify, approximate, and hierarchically decompose hard tasks into simpler subtasks. Theoretical and cognitive research has revealed several such strategies; however, little is known about their establishment, interaction, and efficiency. Here, we use model-based behavioral analysis to provide a detailed examination of the performance of human subjects in a moderately deep planning task. We find that subjects exploit the structure of the domain to establish subgoals in a way that achieves a nearly maximal reduction in the cost of computing values of choices, but then combine partial searches with greedy local steps to solve subtasks, and maladaptively prune the decision trees of subtasks in a reflexive manner upon encountering salient losses. Subjects come idiosyncratically to favor particular sequences of actions to achieve subgoals, creating novel complex actions or "options."
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Progress is being made in both neuroscience and machine learning on finding potential mechanisms for hierarchical planning and goal-seeking.
“Progress is being made in both neuroscience and machine learning on finding potential mechanisms for this type of hierarchical planning and goal-seeking.”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Huys et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
(1998) · The Journal of Clinical Psychiatry · Cited by 19,139 · see citing works
Kass, R. E., & Raftery, A. E. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476572 · Cited by 12,558 · see citing works
(1998) · PubMed · Cited by 10,216 · see citing works
The mini international neuropsychiatric interview
Lecrubier, Y., Sheehan, D. V., Hergueta, T., & Weiller, E. (1998) · European Psychiatry · doi:10.1016/s0924-9338(99)80239-9 · Cited by 5,882 · see citing works
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,406 · see citing works
Node centrality in weighted networks: Generalizing degree and shortest paths
Opsahl, T., Agneessens, F., & Skvoretz, J. V. (2010) · Social Networks · doi:10.1016/j.socnet.2010.03.006 · Cited by 3,734 · see citing works
Hierarchical Dirichlet Processes
Teh, Y. W., Jordan, M. I., Beal, M. J., & Blei, D. M. (2006) · Journal of the American Statistical Association · doi:10.1198/016214506000000302 · Cited by 3,445 · see citing works
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Self-citationThe Architecture of Cognitive Control in the Human Prefrontal Cortex
Koechlin, E., Ody, C., & Kouneiher, F. (2003) · Science · doi:10.1126/science.1088545 · Cited by 1,922 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Self-citationHierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,464 · see citing works
Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes
Badre, D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.004 · Cited by 980 · see citing works
Building neural representations of habits.
Jog, M. S., Kubota, Y., Connolly, C. I., Hillegaart, V., & Graybiel, A. M. (1999) · Science · doi:10.1126/science.286.5445.1745 · Cited by 834 · see citing works
Evidence of hierarchies in cognitive maps
Hirtle, S. C., & Jonides, J. (1985) · Memory & Cognition · doi:10.3758/bf03197683 · Cited by 655 · see citing works
Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646 · see citing works
Proceedings of the Eighteenth International Conference on Machine Learning
Brodley, C. E., & Danyluk, A. (2001) · Cited by 566 · see citing works
“Memo” Functions and Machine Learning
Michie, D. (1968) · Nature · doi:10.1038/218019a0 · Cited by 557 · see citing works
Neural representations of events arise from temporal community structure
Schapiro, A. C., Rogers, T. T., Cordova, N. I., Turk-Browne, N. B., & Botvinick, M. M. (2013) · Nature Neuroscience · doi:10.1038/nn.3331 · Cited by 556 · see citing works
DYNAMIC RESPONSE-BY-RESPONSE MODELS OF MATCHING BEHAVIOR IN RHESUS MONKEYS
Lau, B., & Glimcher, P. W. (2005) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.2005.110-04 · Cited by 485 · see citing works
Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002410 · Cited by 439 · see citing works
Self-citationAutomatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
McGovern, A., & Barto, A. G. (2001) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 416 · see citing works
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297 · see citing works
Rutledge, R. B., Lazzaro, S. C., Lau, B., Myers, C. E., Gluck, M. A., & Glimcher, P. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.3524-09.2009 · Cited by 255 · see citing works
Serotonin Selectively Modulates Reward Value in Human Decision-Making
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P., & Dolan, R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.0053-12.2012 · Cited by 252 · see citing works
Self-citationIdentifying useful subgoals in reinforcement learning by local graph partitioning
Şimşek, Ö., Wolfe, A. P., & Barto, A. G. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102454 · Cited by 238 · see citing works
Dezfouli, A., & Balleine, B. W. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003364 · Cited by 221 · see citing works
Productivity and Reuse in Language
O’Donnell, T. J. (2015) · The MIT Press eBooks · doi:10.7551/mitpress/9780262028844.001.0001 · Cited by 220 · see citing works
'Fine-to-Coarse' Route Planning and Navigation in Regionalized Environments
Wiener, J. M., & Mallot, H. A. (2003) · Spatial Cognition & Computation · doi:10.1207/s15427633scc0304_5 · Cited by 197 · see citing works
Cognitive Illusions of Authorship Reveal Hierarchical Error Detection in Skilled Typists
Logan, G. D., & Crump, M. J. C. (2010) · Science · doi:10.1126/science.1190483 · Cited by 150 · see citing works
Dynamic Signals Related to Choices and Outcomes in the Dorsolateral Prefrontal Cortex
Seo, H., Barraclough, D. J., & Lee, D. (2007) · Cerebral Cortex · doi:10.1093/cercor/bhm064 · Cited by 143 · see citing works
Camerer, C., & Ho, T. (1998) · Journal of Mathematical Psychology · doi:10.1006/jmps.1998.1217 · Cited by 117 · see citing works
Ostlund, S. B., Winterbauer, N. E., & Balleine, B. W. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1176-09.2009 · Cited by 103 · see citing works
Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage
O’Donnell, T. J. (2015) · Project Muse (Johns Hopkins University) · Cited by 100 · see citing works
Huys, Q. J. M., Guitart‐Masip, M., Dolan, R. J., & Dayan, P. (2015) · Clinical Psychological Science · doi:10.1177/2167702614562040 · Cited by 81 · see citing works
Self-citationGoal neglect and knowledge chunking in the construction of novel behaviour
Bhandari, A., & Duncan, J. (2013) · Cognition · doi:10.1016/j.cognition.2013.08.013 · Cited by 70 · see citing works
Fragment Grammars: Exploring Computation and Reuse in Language
O’Donnell, T. J., Tenenbaum, J. B., & Goodman, N. D. (2009) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 46 · see citing works
Wingate, D., Diuk, C., O’Donnell, T. J., Tenenbaum, J. B., & Gershman, S. J. (2013) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 14 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
What Is a Cognitive Map? Organizing Knowledge for Flexible Behavior
Behrens, T. E. J., Muller, T. H., Whittington, J. C. R., Mark, S., Baram, A. B., Stachenfeld, K. L., & Kurth-Nelson, Z. (2018) · Neuron · doi:10.1016/j.neuron.2018.10.002 · Cited by 1,055 · see citing works
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002 · see citing works
Self-citationLieder, F., & Griffiths, T. L. (2020) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1900061x · Cited by 892 · see citing works
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Self-citationComputational psychiatry: towards a mathematically informed understanding of mental illness
Adams, R. A., Huys, Q. J. M., & Roiser, J. P. (2015) · Journal of Neurology, Neurosurgery & Psychiatry · doi:10.1136/jnnp-2015-310737 · Cited by 285 · see citing works
Self-citationHippocampal and prefrontal processing of network topology to simulate the future
Javadi, A.-H., Emo, B., Howard, L. R., Zisch, F. E., Yu, Y., Knight, R., Pinelo Silva, J., & Spiers, H. J. (2017) · Nature Communications · doi:10.1038/ncomms14652 · Cited by 207 · see citing works
Petzschner, F. H., Weber, L. A. E., Gard, T., & Stephan, K. E. (2017) · Biological Psychiatry · doi:10.1016/j.biopsych.2017.05.012 · Cited by 181 · see citing works
Neural Mechanisms of Hierarchical Planning in a Virtual Subway Network
Balaguer, J., Spiers, H., Hassabis, D., & Summerfield, C. (2016) · Neuron · doi:10.1016/j.neuron.2016.03.037 · Cited by 163 · see citing works
Rational use of cognitive resources in human planning
Callaway, F., van Opheusden, B., Gul, S., Das, P., Krueger, P. M., Griffiths, T. L., & Lieder, F. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01332-8 · Cited by 121 · see citing works
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119 · see citing works
Toward the neural implementation of structure learning
Tervo, D. G. R., Tenenbaum, J. B., & Gershman, S. J. (2016) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2016.01.014 · Cited by 106 · see citing works
Self-citationOpen Problems in Cooperative AI
Dafoe, A., Hughes, E., Bachrach, Y., Collins, T., McKee, K. R., Leibo, J. Z., Larson, K., & Graepel, T. (2020) · arXiv · doi:10.48550/arxiv.2012.08630 · Cited by 93 · see citing works
Sebold, M., Schad, D. J., Nebe, S., Garbusow, M., Jünger, E., Kroemer, N. B., Kathmann, N., Zimmermann, U. S., Smolka, M. N., Rapp, M. A., et al. (2016) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00945 · Cited by 41 · see citing works
Self-citationHybrid social learning in human-algorithm cultural transmission
Brinkmann, L., Gezerli, D., Kleist, K. V., Müller, T. F., Rahwan, I., & Pescetelli, N. (2022) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2020.0426 · Cited by 24 · see citing works
Rational Simplification and Rigidity in Human Planning
Ho, M. K., Cohen, J. D., & Griffiths, T. L. (2023) · Psychological Science · doi:10.1177/09567976231200547 · Cited by 11 · see citing works
- Authors
- Huys, Q. J. M., Lally, N., Faulkner, P., Eshel, N., Seifritz, E., Gershman, S. J., Dayan, P., & Roiser, J. P.
- Affiliations
- University of Zurich (CH)
- National Institutes of Health (US)
- University of California, Los Angeles (US)
- Harvard University (US)
- Massachusetts Institute of Technology (US)
- Gatsby Computational Neuroscience Unit (GB)
- University College London (GB)
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2015-02-09
- Volume
- 112
- Issue
- 10
- Pages
- 3098-3103
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4364207
- OpenAlex
- W2046194382
- Cited by
- 207
- References
- 41
- Citations by year
- 2026: 9 · 2025: 26 · 2024: 12 · 2023: 19 · 2022: 22 · 2021: 23 · 2020: 17 · 2019: 16 · 2018: 16 · 2017: 26 · 2016: 12 · 2015: 8
- FWCI
- 15.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Zurich Open Repository and Archive (University of Zurich) (acceptedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- UCL Discovery (University College London) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion)
- Zurich Open Repository and Archive (University of Zurich)
- Funders
- Wellcome Trust (091593)
- National Institute of Mental Health (F30MH100729)
- National Institute of General Medical Sciences (T32GM007753)
- Topics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- planning strategies, approximate planning, hierarchical task decomposition, decision tree pruning, greedy local search, options, human planning, heuristics
- MeSH terms
- Humans, Intelligence, Planning Techniques, Stochastic Processes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
