Transition from ‘model-based’ to ‘model-free’ behavioral control in addiction: Involvement of the orbitofrontal cortex and dorsolateral striatum
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Cocaine addiction is a complex and multidimensional process involving a number of behavioral and neural forms of plasticity. The behavioral transition from voluntary drug use to compulsive drug taking may be explained at the neural level by drug-induced changes in function or interaction between a flexible planning system, associated with prefrontal cortical regions, and a rigid habit system, associated with the striatum. The dichotomy between these two systems is operationalized in computational theory by positing model-based and model-free learning mechanisms, the former relying on an “internal model” of the environment and the latter on pre-computed or cached values to control behavior. In this review, we will suggest that model-free and model-based learning mechanisms appear to be differentially affected, at least in the case of psychostimulants such as cocaine, with the former being enhanced while the latter are disrupted. As a result, the behavior of long-term drug users becomes less flexible and responsive to the desirability of expected outcomes and more habitual, based on the long history of reinforcement. To support our specific proposal, we will review recent neural and behavioral evidence on the effect of psychostimulant exposure on orbitofrontal and dorsolateral striatum structure and function.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neural systems of reinforcement for drug addiction: from actions to habits to compulsion
Everitt, B. J., & Robbins, T. W. (2005) · Nature Neuroscience · doi:10.1038/nn1579 · Cited by 3,785 · cited 2×
“Interaction between these pavlovian and instrumental processes lead to the transition from voluntary drug use to compulsive drug taking (Everitt and Robbins, 2005).”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 1×
“The dichotomy between these two systems is captured in computational neuroscience by the concepts of model-based and model-free reinforcement learning (Daw et al., 2005, Dayan and Daw, 2008).”
BackgroundA framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · cited 1×
“…learning underlies habitual behavior, and the dorsolateral striatum, already demonstrated to be involved in automatic behavior (Balleine et al., 2007), seems to be the most promising candidate as a neural correlate of exclusively model-free signals (Montague et al., 1996, Wunderlich et al., 2012).”
BackgroundDissecting components of reward: ‘liking’, ‘wanting’, and learning
Berridge, K. C., Robinson, T. E., & Aldridge, J. W. (2009) · Current Opinion in Pharmacology · doi:10.1016/j.coph.2008.12.014 · Cited by 1,942
Jentsch, J. D., & Taylor, J. R. (1999) · Psychopharmacology · doi:10.1007/pl00005483 · Cited by 1,545 · cited 1×
“…use is mediated by the positive reinforcing effects of the drug, but the transition to addiction may be linked to a reduced behavioral control with a concomitant development of drug-seeking habits (Jentsch and Taylor, 1999, Volkow and Fowler, 2000, Everitt and Robbins, 2005, Berridge et al., 2009).”
BackgroundGoal-directed instrumental action: contingency and incentive learning and their cortical substrates
Balleine, B. W., & Dickinson, A. (1998) · Neuropharmacology · doi:10.1016/s0028-3908(98)00033-1 · Cited by 1,541 · cited 1×
“…in instrumental contexts, overtraining makes test performance insensitive to reinforcer devaluation: these animals no longer change learned responding as a result of reinforcer devaluation (Holman, 1975, Colwill and Rescorla, 1988) (Balleine and Dickinson, 1998, Killcross and Coutureau, 2003).”
BackgroundTemporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · cited 1×
“A rich body of work using these computational tools has already broadened our understanding of the microstructure of behavior and the neural circuits underlying valuebased decision-making (O'Doherty et al., 2003, Seymour et al., 2004).”
BackgroundHow does stress increase risk of drug abuse and relapse?
Sinha, R. (2001) · Psychopharmacology · doi:10.1007/s002130100917 · Cited by 1,505 · cited 1×
“…between cocaine and stress effects on model-free and model-based learning mechanisms is extremely intriguing, especially because stress is a critical risk factor for the development of addiction as well as for relapse to drug seeking after abstinence (Piazza and Le Moal, 1996, Sinha, 2001).”
BackgroundThe Role of the Dorsal Striatum in Reward and Decision-Making: Figure 1.
Balleine, B. W., Delgado, M. R., & Hikosaka, O. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.1554-07.2007 · Cited by 1,390 · cited 2×
“…that model-free learning underlies habitual behavior, and the dorsolateral striatum, already demonstrated to be involved in automatic behavior (Balleine et al., 2007), seems to be the most promising candidate as a neural correlate of exclusively model-free signals (Montague et al., 1996,…”
BackgroundRelative reward preference in primate orbitofrontal cortex
Tremblay, L., & Schultz, W. (1999) · Nature · doi:10.1038/19525 · Cited by 1,314 · cited 1×
“Furthermore anticipatory neuronal activity in OFC changes when the value of an outcome is reduced by satiety (Critchley and Rolls, 1996, Tremblay and Schultz, 1999) and functional studies demonstrate that the blood oxygen level-dependent activity in OFC correlates with changes in the value of…”
BackgroundAddiction, a disease of compulsion and drive: involvement of the orbitofrontal cortex
Volkow, N. D. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.318 · Cited by 1,240
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226 · cited 3×
“, 2009a) and that the DLS is involved in this deficit (Yin et al., 2004).”
BackgroundEncoding Predictive Reward Value in Human Amygdala and Orbitofrontal Cortex
Gottfried, J. A., O'Doherty, J., & Dolan, R. J. (2003) · Science · doi:10.1126/science.1087919 · Cited by 1,165
Frontal cortex and reward-guided learning and decision-making
Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011) · Neuron · doi:10.1016/j.neuron.2011.05.014 · Cited by 1,111
The role of the dorsomedial striatum in instrumental conditioning
Yin, H. H., Ostlund, S. B., Knowlton, B. J., & Balleine, B. W. (2005) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2005.04218.x · Cited by 1,099
Reinstatement of cocaine-reinforced responding in the rat
de Wit, H., & Stewart, J. (1981) · Psychopharmacology · doi:10.1007/bf00432175 · Cited by 928 · cited 1×
“These deficits are appropriately positioned for a critical role in drug relapse, in which drug cues trigger excessive and compulsive drugseeking behavior even after long periods of abstinence (de Wit and Stewart, 1981, Robbins and Everitt, 1999).”
BackgroundThe orbitofrontal cortex: Neuronal activity in the behaving monkey
Thorpe, S. J., Rolls, E. T., & Maddison, S. (1983) · Experimental Brain Research · doi:10.1007/bf00235545 · Cited by 836 · cited 1×
“While the above deficits might reflect processing changes in other areas, single unit recording shows that psychostimulant exposure affects signaling of outcome expectancies by OFC (Thorpe et al., 1983, Schoenbaum et al., 1998).”
BackgroundBuilding neural representations of habits.
Jog, M. S., Kubota, Y., Connolly, C. I., Hillegaart, V., & Graybiel, A. M. (1999) · Science · doi:10.1126/science.286.5445.1745 · Cited by 834
A specific role for posterior dorsolateral striatum in human habit learning
Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06796.x · Cited by 769
Diamond, D. M., Campbell, A. M., Park, C. R., Halonen, J., & Zoladz, P. R. (2007) · Neural Plasticity · doi:10.1155/2007/60803 · Cited by 762 · cited 1×
“…-PA Author M anuscript shown that stress disrupts synaptic plasticity and cognitive control processing in prefrontal cortex (Alexander et al., 2007, Diamond et al., 2007, Steinhauser et al., 2007), although it facilitates striatum-dependent processes (Schwabe et al., 2007, Quirarte et al., 2009).”
BackgroundCoordination of Actions and Habits in the Medial Prefrontal Cortex of Rats
Killcross, S. (2003) · Cerebral Cortex · doi:10.1093/cercor/13.4.400 · Cited by 748
Piazza, P. V., & Le Moal, M. (1996) · Annual Review of Pharmacology and Toxicology · doi:10.1146/annurev.pa.36.040196.002043 · Cited by 690
Impulsivity and rapid discounting of delayed hypothetical rewards in cocaine-dependent individuals.
Coffey, S. F., Gudleski, G. D., Saladin, M. E., & Brady, K. T. (2003) · Experimental and Clinical Psychopharmacology · doi:10.1037/1064-1297.11.1.18 · Cited by 625 · cited 1×
“Moreover, delay discounting tasks have shown that psychostimulant exposure enhances impulsive choices in humans and rats (Coffey et al., 2003, Kirby and Petry, 2004, Heil et al., 2006, Roesch et al., 2007b, Simon et al., 2007).”
BackgroundTemporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605 · cited 1×
“A rich body of work using these computational tools has already broadened our understanding of the microstructure of behavior and the neural circuits underlying valuebased decision-making (O'Doherty et al., 2003, Seymour et al., 2004).”
BackgroundDrug addiction: bad habits add up
Robbins, T. W., & Everitt, B. J. (1999) · Nature · doi:10.1038/19208 · Cited by 599 · cited 1×
“These deficits are appropriately positioned for a critical role in drug relapse, in which drug cues trigger excessive and compulsive drugseeking behavior even after long periods of abstinence (de Wit and Stewart, 1981, Robbins and Everitt, 1999).”
BackgroundOrbitofrontal Cortex and Representation of Incentive Value in Associative Learning
Gallagher, M., McMahan, R. W., & Schoenbaum, G. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-15-06610.1999 · Cited by 593 · cited 1×
“For example, normal performance in both Pavlovian and instrumental versions of the reinforcer devaluation task depends on the OFC (Gallagher et al., 1999, Pickens et al., 2003, Izquierdo et al., 2004, Pickens et al., 2005) (Gremel C.M. et al., Soc.”
BackgroundSelf-citationIzquierdo, A., Suda, R. K., & Murray, E. A. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.1921-04.2004 · Cited by 588
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Schoenbaum, G., Chiba, A. A., & Gallagher, M. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-05-01876.1999 · Cited by 579 · cited 1×
“Neurons were recorded as rats learned and reversed odor discrimination problems in which unique odor cues were paired with a sucrose reward and a quinine punishment (Schoenbaum et al., 1999).”
BackgroundSelf-citationDetermining the neural substrates of goal-directed learning in the human brain
Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.0564-07.2007 · Cited by 529
A unified framework for addiction: Vulnerabilities in the decision process
Redish, A. D., Jensen, S., & Johnson, A. (2008) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0800472x · Cited by 501 · cited 1×
“Addiction is characterized by a failure of the decision-making system (Redish et al., 2008, Lucantonio et al., 2012).”
BackgroundHampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1010-06.2006 · Cited by 498
Sensory-specific satiety-related olfactory activation of the human orbitofrontal cortex
OʼDoherty, J., Rolls, E. T., Francis, S., Bowtell, R., McGlone, F., Kobal, G., Renner, B., & Ahne, G. (2000) · NeuroReport · doi:10.1097/00001756-200002070-00035 · Cited by 482
Critchley, H. D., & Rolls, E. T. (1996) · Journal of Neurophysiology · doi:10.1152/jn.1996.75.4.1673 · Cited by 468
Amphetamine Exposure Enhances Habit Formation
Nelson, A., & Killcross, S. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.4305-05.2006 · Cited by 445
Wyvell, C. L., & Berridge, K. C. (2001) · The Journal of Neuroscience · doi:10.1523/jneurosci.21-19-07831.2001 · Cited by 436 · cited 2×
“(Wyvell and Berridge, 2001, Shiflett, 2012).”
The ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Orbitofrontal Cortex Mediates Outcome Encoding in Pavlovian But Not Instrumental Conditioning
Ostlund, S. B., & Balleine, B. W. (2007) · The Journal of Neuroscience · doi:10.1523/jneurosci.5443-06.2007 · Cited by 379 · cited 1×
“…Murray, 2008, Rygula et al., 2010), differential outcome expectancy (Overmier et al., 1971), outcome-specific Pavlovian-to-instrumental transfer (Ostlund and Balleine, 2007), outcomespecific conditioned reinforcement (Burke et al., 2008), and most recently in sensory preconditioning (Jones et…”
BackgroundSensory-specific satiety-related olfactory activation of the human orbitofrontal cortex
OʼDoherty, J., Rolls, E. T., Francis, S., Bowtell, R., McGlone, F., Kobal, G., Renner, B., & Ahne, G. (2000) · NeuroReport · doi:10.1097/00001756-200003200-00046 · Cited by 361
Mapping value based planning and extensively trained choice in the human brain
Wunderlich, K., Dayan, P., & Dolan, R. J. (2012) · Nature Neuroscience · doi:10.1038/nn.3068 · Cited by 310
Beta-adrenergic Modulation of Cognitive Flexibility during Stress
Alexander, J. K., Hillier, A., Smith, R. M., Tivarus, M. E., & Beversdorf, D. Q. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.3.468 · Cited by 306 · cited 1×
“…Author M anuscript N IH -PA Author M anuscript shown that stress disrupts synaptic plasticity and cognitive control processing in prefrontal cortex (Alexander et al., 2007, Diamond et al., 2007, Steinhauser et al., 2007), although it facilitates striatum-dependent processes (Schwabe et al., 2007,…”
BackgroundShift from goal-directed to habitual cocaine seeking after prolonged experience in rats
Zapata, A., Minney, V. L., & Shippenberg, T. S. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.4072-10.2010 · Cited by 296 · cited 1×
“While this might reflect dysfunction in other model-based areas, in a recent study, Zapata and colleagues (Zapata et al., 2010) showed that such insensitivity to outcome devaluation can be reversed by inactivation of DLS in cocaine trained animals.”
Owen, A. M. (1997) · Progress in Neurobiology · doi:10.1016/s0301-0082(97)00042-7 · Cited by 251
Contributions of the ventromedial prefrontal cortex to goal-directed action selection
O’Doherty, J. P. (2011) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2011.06290.x · Cited by 225
Stress-induced cognitive dysfunction: hormone-neurotransmitter interactions in the prefrontal cortex
Shansky, R. M., & Lipps, J. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00123 · Cited by 205 · cited 1×
“Interestingly, similar to what we have shown with cocaine exposure, stress seems to modulate prefrontal cortex mechanisms and dorsolateral striatum processes in a way that biases towards model-free strategies (Shansky and Lipps, 2013).”
Backgroundde Wit, S., Corlett, P. R., Aitken, M. R., Dickinson, A., & Fletcher, P. C. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.1639-09.2009 · Cited by 200
Rudebeck, P. H., & Murray, E. A. (2008) · The Journal of Neuroscience · doi:10.1523/jneurosci.2272-08.2008 · Cited by 148 · cited 1×
“…for calculating the current value of a particular outcome, such as reversal learning (Schoenbaum et al., 2003, Ghods-Sharifi et al., 2008, Rudebeck and Murray, 2008, Rygula et al., 2010), differential outcome expectancy (Overmier et al., 1971), outcome-specific Pavlovian-to-instrumental…”
BackgroundRygula, R., Walker, S. C., Clarke, H. F., Robbins, T. W., & Roberts, A. C. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.2631-10.2010 · Cited by 136 · cited 1×
“…current value of a particular outcome, such as reversal learning (Schoenbaum et al., 2003, Ghods-Sharifi et al., 2008, Rudebeck and Murray, 2008, Rygula et al., 2010), differential outcome expectancy (Overmier et al., 1971), outcome-specific Pavlovian-to-instrumental transfer (Ostlund and…”
BackgroundBurke, K. A., Takahashi, Y. K., Correll, J., Leon Brown, P., & Schoenbaum, G. (2009) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2009.06992.x · Cited by 76 · cited 1×
“By contrast, rats tested on simple extinction exhibited normal performance in response to reward omission after OFC lesions or inactivation (Burke et al., 2009, Takahashi et al., 2009).”
BackgroundSelf-citation
Cited by
Story, G. W., Vlaev, I., Seymour, B., Darzi, A., & Dolan, R. J. (2014) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2014.00076 · Cited by 286
Miller, K. J., Shenhav, A., & Ludvig, E. A. (2019) · Psychological Review · doi:10.1037/rev0000120 · Cited by 280
Going and stopping: dichotomies in behavioral control by the prefrontal cortex
Gourley, S. L., & Taylor, J. R. (2016) · Nature Neuroscience · doi:10.1038/nn.4275 · Cited by 195
Pathological overeating: emerging evidence for a compulsivity construct
Moore, C. F., Sabino, V., Koob, G. F., & Cottone, P. (2017) · Neuropsychopharmacology · doi:10.1038/npp.2016.269 · Cited by 126
“It is hypothesized that compulsive behavior may reflect a maladaptive stimulus– response habit that was once a flexible and voluntary goal-directed behavior (Everitt and Robbins, 2005; Halbout et al, 2016; Lucantonio et al, 2014; Ostlund and Balleine, 2008).”
Learning to lose control: A process-based account of behavioral addiction
Perales, J. C., King, D. L., Navas, J. F., Schimmenti, A., Sescousse, G., Starcevic, V., van Holst, R. J., & Billieux, J. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.12.025 · Cited by 101
Metadata
- Authors
- Lucantonio, F., Caprioli, D., & Schoenbaum, G.
- Venue
- Neuropharmacology
- Publisher
- Elsevier BV
- Published
- 2014-01
- Volume
- 76
- Pages
- 407-415
- ISSN
- 0028-3908 (ISSN-L)
- OpenAlex
- W1975489711
- Cited by
- 78
- References
- 49
- FWCI
- 3.79
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.