Novelty and Inductive Generalization in Human Reinforcement Learning
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
In reinforcement learning (RL), a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model and temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Predictive reward signal of dopamine neurons
Schultz, W. (1998) · Journal of Neurophysiology · doi:10.1152/jn.1998.80.1.1 · Cited by 4,564
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009
A new one-trial test for neurobiological studies of memory in rats. 1: behavioral data
Ennaceur, A., & Delacour, J. (1988) · Behavioural Brain Research · doi:10.1016/0166-4328(88)90157-x · Cited by 3,223 · cited 1×
“…a lever that periodically delivers a visual stimulus (Reed et al., 1996), will display a preference for environments in which novel objects have appeared (Bardo and Bevins, 2000) and will interact more with novel objects placed in a familiar environment (Sheldon, 1969; Ennaceur and Delacour, 1988).”
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 2×
“…and animals employ a form of the temporal difference (TD) learning algorithm, which uses prediction errors (the discrepancy between received and expected reward) to update reward predictions (Houk et al., 1995; Montague et al., 1996; Schultz et al., 1997); for a recent review, see Niv (2009).”
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Midbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 1×
“Despite this remarkable correspondence, the prediction error interpretation of dopamine has been challenged by the observation that dopamine neurons also respond to the appearance of novel stimuli (Horvitz et al., 1997; Schultz, 1998), a finding not predicted by classical RL theories.”
Mere Exposure: A Gateway to the Subliminal
Zajonc, R. B. (2001) · Current Directions in Psychological Science · doi:10.1111/1467-8721.00154 · Cited by 1,299 · cited 2×
“Another well-studied form of neophobia is known as the mere exposure effect: Simply presenting an object repeatedly is sufficient to enhance preference for that object relative to a novel object (Zajonc, 2001).”
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
The adaptive nature of human categorization.
Anderson, J. R. (1991) · Psychological Review · doi:10.1037/0033-295x.98.3.409 · Cited by 1,069
Cited by
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
Self-citationIntrinsic motivation, curiosity, and learning
Oudeyer, P.-Y., Gottlieb, J., & Lopes, M. (2016) · Progress in Brain Research · doi:10.1016/bs.pbr.2016.05.005 · Cited by 387
Deconstructing the human algorithms for exploration
Gershman, S. J. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.014 · Cited by 267
Self-citationMenegas, W., Babayan, B. M., Uchida, N., & Watabe-Uchida, M. (2017) · eLife · doi:10.7554/elife.21886 · Cited by 265
Hippocampal remapping as hidden state inference
Sanders, H., Wilson, M. A., & Gershman, S. J. (2020) · eLife · doi:10.7554/elife.51140 · Cited by 163
Self-citation
Metadata
- Authors
- Gershman, S. J., & Niv, Y.
- Venue
- Topics in Cognitive Science
- Publisher
- Wiley
- Published
- 2015-03-23
- Volume
- 7
- Issue
- 3
- Pages
- 391-415
- ISSN
- 1756-8757 (ISSN-L)
- 1756-8765
- OpenAlex
- W2044320213
- Cited by
- 98
- References
- 23
- FWCI
- 7.19
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Novelty and Inductive Generalization in Human Reinforcement Learning. https://app.cochat.ai/papers/10.1111/tops.12138
- The paper
- Gershman, S. J., & Niv, Y. (2015). Novelty and Inductive Generalization in Human Reinforcement Learning. Topics in Cognitive Science. https://doi.org/10.1111/tops.12138
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.