Structure Learning in Human Sequential Decision-Making
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Studies of sequential decision-making in humans frequently find suboptimal performance relative to an ideal actor that has perfect knowledge of the model of how rewards and events are generated in the environment. Rather than being suboptimal, we argue that the learning problem humans face is more complex, in that it also involves learning the structure of reward generation in the environment. We formulate the problem of structure learning in sequential decision tasks using Bayesian reinforcement learning, and show that learning the generative model for rewards qualitatively changes the behavior of an optimal learning agent. To test whether people exhibit structure learning, we performed experiments involving a mixture of one-armed and two-armed bandit reward models, where structure learning produces many of the qualitative behaviors deemed suboptimal in previous studies. Our results demonstrate humans can perform structure learning in a near-optimal manner.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · cited 1×
“Using model-based (Bayesian) reinforcement learning”
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Theory-based Bayesian models of inductive learning and reasoning
Tenenbaum, J. B., Griffiths, T. L., & Kemp, C. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.05.009 · Cited by 880 · cited 2×
“The idea of modeling sequential decision making under uncertainty as a structure learning problem is a natural extension of previous work on structure learning in models of cognition [13,16] (also see [32]), animal learning [33] and motor control (e.g., see [34]).”
BackgroundMethodLearning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326
Braun, D. A., Mehring, C., & Wolpert, D. M. (2010) · Behavioural Brain Research · doi:10.1016/j.bbr.2009.08.031 · Cited by 210 · cited 1×
“The idea of modeling sequential decision making under uncertainty as a structure learning problem is a natural extension of previous work on structure learning in models of cognition [13,16] (also see [32]), animal learning [33] and motor control (e.g., see [34]).”
Background
Cited by
Prefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596
Cognitive control over learning: Creating, clustering, and generalizing task-set structure.
Collins, A. G. E., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0030852 · Cited by 426
Learning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326
Deconstructing the human algorithms for exploration
Gershman, S. J. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.014 · Cited by 267
Attention, learning, and the value of information
Gottlieb, J. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.034 · Cited by 241
“Converging evidence shows that humans indeed infer hidden models of complex tasks (Acuña and Schrater, 2010; Braun et al., 2010; Daw et al., 2011; Gershman and Niv, 2010; Yakushijin and Jacobs, 2011), and indirect evidence from tasks involving schemas or contextual associations suggests that lower”
Background
Metadata
- Authors
- Acuña, D. E., & Schrater, P.
- Venue
- PLoS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-12-02
- Volume
- 6
- Issue
- 12
- Pages
- e1001003
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W1970966707
- Cited by
- 60
- References
- 5
- FWCI
- 4.36
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Structure Learning in Human Sequential Decision-Making. https://app.cochat.ai/papers/10.1371/journal.pcbi.1001003
- The paper
- Acuña, D. E., & Schrater, P. (2010). Structure Learning in Human Sequential Decision-Making. PLoS Computational Biology. https://doi.org/10.1371/journal.pcbi.1001003
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.