Reinforcement-Based Decision Making in Corticostriatal Circuits: Mutual Constraints by Neurocomputational and Diffusion Models
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
In this letter, we examine the computational mechanisms of reinforce-ment-based decision making. We bridge the gap across multiple levels of analysis, from neural models of corticostriatal circuits—the basal ganglia (BG) model (Frank, 2005 , 2006 ) to simpler but mathematically tractable diffusion models of two-choice decision making. Specifically, we generated simulated data from the BG model and fit the diffusion model (Ratcliff, 1978 ) to it. The standard diffusion model fits underestimated response times under conditions of high response and reinforcement conflict. Follow-up fits showed good fits to the data both by increasing nondecision time and by raising decision thresholds as a function of conflict and by allowing this threshold to collapse with time. This profile captures the role and dynamics of the subthalamic nucleus in BG circuitry, and as such, parametric modulations of projection strengths from this nucleus were associated with parametric increases in decision boundary and its modulation by conflict. We then present data from a human reinforcement learning experiment involving decisions with low- and high-reinforcement conflict. Again, the standard model failed to fit the data, but we found that two variants similar to those that fit the BG model data fit the experimental data, thereby providing a convergence of theoretical accounts of complex interactive decision-making mechanisms consistent with available data. This work also demonstrates how to make modest modifications to diffusion models to summarize core computations of the BG model. The result is a better fit and understanding of reinforcement-based choice data than that which would have occurred with either model alone.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Conflict monitoring and cognitive control
Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.624 · Cited by 7,351
Effects of noise letters upon the identification of a target letter in a nonsearch task
Eriksen, B. A., & Eriksen, C. W. (1974) · Perception & Psychophysics · doi:10.3758/bf03203267 · Cited by 7,230 · cited 2×
“For example, in the Eriksen paradigm (Eriksen & Eriksen, 1974 ) subjects must identify a target item that is surrounded (flanked) by items that, on conflict trials, suggest the other response to the target.”
MethodRatcliff, R. (1978) · Psychological Review · doi:10.1037/0033-295x.85.2.59 · Cited by 4,513 · cited 1×
“The diffusion model (Ratcliff, 1978) is a model of the cognitive processes involved in making simple two-choice decisions (see Figure 2).”
Self-citationThe Neural Basis of Decision Making
Gold, J. I., & Shadlen, M. N. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.29.051605.113038 · Cited by 3,764 · cited 1×
“…a tight connection with the behavior of populations of single cells that appear to implement the processes of evidence accumulation to criterion (Gold & Shadlen, 2001, 2007; Mazurek, Roitman, Ditterich, & Shadlen, 2003; Ratcliff, Cherian, & Segraves, 2003; Ratcliff, Hasegawa, Hasegawa, Smith, &…”
BackgroundThe Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks
Ratcliff, R., & McKoon, G. (2008) · Neural Computation · doi:10.1162/neco.2008.12-06-420 · Cited by 3,059 · cited 3×
“…one accumulator or two accumulators) were successful in accounting for the qualitative patterns of results found in two-choice tasks (see Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006; Ratcliff, 2006; Ratcliff & McKoon, 2008; Ratcliff, Thapar, Smith, & McKoon, 2005; Usher & McClelland, 2001).”
BackgroundMethodSelf-citationThe time course of perceptual choice: The leaky, competing accumulator model.
Usher, M., & McClelland, J. L. (2001) · Psychological Review · doi:10.1037/0033-295x.108.3.550 · Cited by 2,152 · cited 4×
“…one accumulator or two accumulators) were successful in accounting for the qualitative patterns of results found in two-choice tasks (see Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006; Ratcliff, 2006; Ratcliff & McKoon, 2008; Ratcliff, Thapar, Smith, & McKoon, 2005; Usher & McClelland, 2001).”
BackgroundMethodInfluentialBy Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964
Self-citationVisual fixations and the computation and comparison of value in simple choice
Krajbich, I., Armel, C., & Rangel, A. (2010) · Nature Neuroscience · doi:10.1038/nn.2635 · Cited by 1,408 · cited 1×
“(This process is analogous to recently reported data with the diffusion model in which good fits to reward-based decision making were obtained by assuming that the drift process is biased toward the decision option to which the subject is currently fixating; Krajbich, Armel, & Rangel, 2010 ).”
ResultPsychology and neurobiology of simple decisions
Smith, P. L., & Ratcliff, R. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.01.006 · Cited by 1,051
Self-citationAron, A. R., Behrens, T. E., Smith, S., Frank, M. J., & Poldrack, R. A. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0519-07.2007 · Cited by 982
Self-citation
Cited by
Diffusion decision model: Current issues and history
Ratcliff, R., Smith, P. L., Brown, S. D., & McKoon, G. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.01.007 · Cited by 1,630
“The conflict conditions were modeled with collapsing decision boundaries that accounted for the shifts in RT distributions [69].”
BackgroundMethodSelf-citationNeuronal Reward and Decision Signals: From Theories to Data
Schultz, W. (2015) · Physiological Reviews · doi:10.1152/physrev.00023.2014 · Cited by 1,126
“Particularly good fits are obtained with a diffusion model of movement selection (449).”
BackgroundHDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python
Wiecki, T. V., Sofer, I., & Frank, M. J. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00014 · Cited by 1,047
“The details of this mechanism are described in other modeling papers (e.g., Ratcliff and Frank, 2012).”
Self-citationMetacognition in human decision-making: confidence and error monitoring
Yeung, N., & Summerfield, C. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2011.0416 · Cited by 667
Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold
Cavanagh, J. F., Wiecki, T. V., Cohen, M. X., Figueroa, C. M., Samanta, J., Sherman, S. J., & Frank, M. J. (2011) · Nature Neuroscience · doi:10.1038/nn.2925 · Cited by 645
Self-citationCognitive control over learning: Creating, clustering, and generalizing task-set structure.
Collins, A. G. E., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0030852 · Cited by 426
“This functionality provides a dynamic regulation of the model's decision threshold as a function of response conflict (Ratcliff & Frank, 2012), such that more time is taken to accumulate evidence among noisy corticostriatal signals to prevent impulsive responding and to settle on a more optimal…”
Self-citationA computational model of inhibitory control in frontal cortex and basal ganglia.
Wiecki, T. V., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0031542 · Cited by 404
Self-citationCollins, A. G. E., & Frank, M. J. (2014) · Psychological Review · doi:10.1037/a0037015 · Cited by 403
Self-citationSelf-Control as Value-Based Choice
Berkman, E. T., Hutcherson, C. A., Livingston, J. L., Kahn, L. E., & Inzlicht, M. (2017) · Current Directions in Psychological Science · doi:10.1177/0963721417704394 · Cited by 337
Reward Pays the Cost of Noise Reduction in Motor and Cognitive Control
Manohar, S. G., Chong, T. T.-J., Apps, M. A. J., Batla, A., Stamelou, M., Jarman, P. R., Bhatia, K. P., & Husain, M. (2015) · Current Biology · doi:10.1016/j.cub.2015.05.038 · Cited by 317
Eye tracking and pupillometry are indicators of dissociable latent decision processes.
Cavanagh, J. F., Wiecki, T. V., Kochar, A., & Frank, M. J. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0035813 · Cited by 289
Self-citationThe drift diffusion model as the choice rule in reinforcement learning
Pedersen, M. L., Frank, M. J., & Biele, G. (2017) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1199-y · Cited by 283
Self-citationRevisiting the Evidence for Collapsing Boundaries and Urgency Signals in Perceptual Decision-Making
Hawkins, G. E., Forstmann, B. U., Wagenmakers, E.-J., Ratcliff, R., & Brown, S. D. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2410-14.2015 · Cited by 279
Self-citationDeconstructing the human algorithms for exploration
Gershman, S. J. (2018) · Cognition · doi:10.1016/j.cognition.2017.12.014 · Cited by 267
“…an analogy to perceptual decision making, have asserted that responses are generated when an evidence accumulation process crosses a decision threshold (Milosavljevic, Malmaud, Huth, Koch, & Rangel, 2010; Ratcliff & Frank, 2012; Tajima, Drugowitsch, & Pouget, 2016; Summerfield & Tsetsos, 2012).”
BackgroundSummerfield, C., & Tsetsos, K. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00070 · Cited by 167
Nonspecific Inhibition of the Motor System during Response Preparation
Greenhouse, I., Sias, A., Labruna, L., & Ivry, R. B. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.1436-15.2015 · Cited by 159
“Moreover, preparatory inhibition does not appear to reflect the setting of a response decision threshold (Ratcliff and Frank, 2012).”
BackgroundAuditory information enhances post-sensory visual evidence during rapid multisensory decision-making
Franzen, L., Delis, I., De Sousa, G., Kayser, C., & Philiastides, M. G. (2020) · Nature Communications · doi:10.1038/s41467-020-19306-7 · Cited by 57
Striatal activation reflects urgency in perceptual decision making
van Maanen, L., Fontanesi, L., Hawkins, G. E., & Forstmann, B. U. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.06.045 · Cited by 36
“literature suggest that decision-makers regulate response selection as a function of time (Bowman et al., 2012; Cisek et al., 2009; Ditterich, 2006a, 2006b; Drugowitsch et al., 2012; Ratcliff and Frank, 2012; Thura et al., 2012).”
Metadata
- Authors
- Ratcliff, R., & Frank, M. J.
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2012-05
- Volume
- 24
- Issue
- 5
- Pages
- 1186-1229
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W1997904803
- Cited by
- 207
- References
- 22
- FWCI
- 8.60
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.