The algorithmic anatomy of model-based evaluation
Open accessAbstract
Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More recent formulations emphasize cooperation, rather than only competition, between behavioural controllers.
“More recent formulations emphasize cooperation, rather than only competition (Daw and Dayan, 2014, Dolan and Dayan, 2013, Keramati et al., 2011, O’Doherty et al., 2015, Pezzulo et al., 2013; see also Daw et al., 2011, Lee et al., 2014) for relevant empirical evidence that supports an integrative viewpoint.”
From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Daw and Dayan, 2014
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Functional architecture of basal ganglia circuits: neural substrates of parallel processing
Alexander, G. E., & Crutcher, M. D. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90107-l · Cited by 4,131 · see citing works
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · see citing works
Two-process learning theory: Relationships between Pavlovian conditioning and instrumental learning.
Rescorla, R. A., & Solomon, R. L. (1967) · Psychological Review · doi:10.1037/h0024475 · Cited by 1,487 · cited 1× · see citing works
“Ventral areas of striatum may be implicated in the performance, if not the acquisition, of instrumental behaviour, consistent with an important role of Pavlovian processes in motivation and regulation of behavioural vigour in Pavlovian to instrumental transfer paradigms [5,82–85].”
BackgroundHuys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002410 · Cited by 439 · see citing works
Self-citationProspective coding for objects in primate prefrontal cortex
Rainer, G., Rao, S. C., & Miller, E. K. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-13-05493.1999 · Cited by 432 · cited 1× · see citing works
“Unfortunately, we know rather little about the nature and evolution of cortical representations of states that underpin MF (and MB) predictions, although it is widely believed that these representations do capture and represent explicitly statistical regularities and predictabilities in their input [95–97].”
BackgroundBanishing the homunculus: Making working memory work
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2006) · Neuroscience · doi:10.1016/j.neuroscience.2005.04.067 · Cited by 385 · see citing works
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380 · see citing works
Self-citationSpeed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297 · see citing works
O'Reilly, R. C. (2002) · Cerebral Cortex · doi:10.1093/cercor/12.3.246 · Cited by 241 · see citing works
Hart, G., Leung, B. K., & Balleine, B. W. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2013.11.003 · Cited by 166 · see citing works
Pezzulo, G., Rigoli, F., & Chersi, F. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00092 · Cited by 163 · see citing works
Evidence for model-based action planning in a sequential finger movement task
Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010) · Journal of Motor Behavior · doi:10.1080/00222895.2010.526467 · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · see citing works
“emphasize cooperation, rather than only competition (Daw and Dayan, 2014; Dolan and Dayan, 2013; Keramati et al., 2011; O’Doherty et al., 2015; Pezzulo et al., 2013; see also Daw et al., 2011; Lee et al., 2014) for relevant empirical evidence that supports an integrative viewpoint.”
Cognitive computational neuroscience
Kriegeskorte, N., & Douglas, P. K. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0210-5 · Cited by 535 · see citing works
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · see citing works
Self-citationDeconstructing the human algorithms for exploration
Gershman, S. J. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.014 · Cited by 267 · see citing works
Anxiety, Depression, and Decision Making: A Computational Perspective
Bishop, S. J., & Gagne, C. (2018) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-080317-062007 · Cited by 240 · see citing works
Depression: a decision-theoretic analysis
Huys, Q. J. M., Daw, N. D., & Dayan, P. (2015) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-071714-033928 · Cited by 221 · see citing works
Self-citationPetzschner, F. H., Weber, L. A. E., Gard, T., & Stephan, K. E. (2017) · Biological Psychiatry · doi:10.1016/j.biopsych.2017.05.012 · Cited by 181 · see citing works
Learning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity
Coddington, L. T., & Dudman, J. T. (2019) · Neuron · doi:10.1016/j.neuron.2019.08.036 · Cited by 150 · see citing works
Dark control: The default mode network as a reinforcement learning agent
Dohmatob, E., Dumas, G., & Bzdok, D. (2020) · Human Brain Mapping · doi:10.1002/hbm.25019 · Cited by 121 · see citing works
The why, what, where, when and how of goal-directed choice: neuronal and computational principles
Verschure, P. F. M. J., Pennartz, C. M. A., & Pezzulo, G. (2014) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2013.0483 · Cited by 131 · see citing works
“Third, the strict distinction between retrospective model-free and prospective model-based decision-making is misleading in the sense that in both cases, the state space on which decision-making operates is defined based on models albeit of varying spatio-temporal complexity [31].”
BackgroundIndicators and criteria of consciousness in animals and intelligent machines: an inside-out approach
Pennartz, C. M. A., Farisco, M., & Evers, K. (2019) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2019.00025 · Cited by 121 · see citing works
Consciousness, Representation, Action: The Importance of Being Goal-Directed
Pennartz, C. M. A. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.10.006 · Cited by 49 · see citing works
The principles of goal-directed decision-making: from neural mechanisms to computation and robotics
Pezzulo, G., Verschure, P. F. M. J., Balkenius, C., & Pennartz, C. M. A. (2014) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2013.0470 · Cited by 35 · see citing works
- Authors
- Daw, N. D., & Dayan, P.
- Affiliations
- New York University (US)
- Gatsby Computational Neuroscience Unit (GB)
- Corresponding author
- Dayan, Peter
- Venue
- Philosophical Transactions of the Royal Society B: Biological Sciences
- Publisher
- The Royal Society
- Published
- 2014-11-05
- Volume
- 369
- Issue
- 1655
- Pages
- 20130478
- ISSN
- 0962-8436 (ISSN-L)
- 1471-2970
- PMC
- PMC4186231
- OpenAlex
- W2153894094
- Cited by
- 195
- References
- 14
- Citations by year
- 2026: 12 · 2025: 24 · 2024: 16 · 2023: 11 · 2022: 16 · 2021: 14 · 2020: 21 · 2019: 15 · 2018: 20 · 2017: 21 · 2016: 12 · 2015: 10 · 2014: 3
- FWCI
- 1.69
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Philosophical Transactions of the Royal Society B Biological Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- model-based reasoning, model-free learning, cognitive modeling, reinforcement learning, prediction and control, animal cognition
- MeSH terms
- Animals, Brain, Humans, Learning, Markov Chains, Models, Neurological
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
