Habits as action sequences: hierarchical action control and changes in outcome value
Open accessAbstract
Goal-directed action involves making high-level choices that are implemented using previously acquired action sequences to attain desired goals. Such a hierarchical schema is necessary for goal-directed actions to be scalable to real-life situations, but results in decision-making that is less flexible than when action sequences are unfolded and the decision-maker deliberates step-by-step over the outcome of each individual action. In particular, from this perspective, the offline revaluation of any outcomes that fall within action sequence boundaries will be invisible to the high-level planner resulting in decisions that are insensitive to such changes. Here, within the context of a two-stage decision-making task, we demonstrate that this property can explain the emergence of habits. Next, we show how this hierarchical account explains the insensitivity of over-trained actions to changes in outcome value. Finally, we provide new data that show that, under extended extinction conditions, habitual behaviour can revert to goal-directed control, presumably as a consequence of decomposing action sequences into single actions. This hierarchical view suggests that the development of action sequences and the insensitivity of actions to changes in outcome value are essentially two sides of the same coin, explaining why these two aspects of automatic behaviour involve a shared neural structure.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The arbitrator selects between model-based and model-free systems according to criteria such as accuracy or prediction error or prevailing rate of reinforcement.
“Given the tradeoff, the arbitrator selects between the two systems according to various criteria, such as the accuracy or prediction error connected with each (Daw et al., 2005; Lee et al., 2014) or the prevailing rate of reinforcement (Dezfouli et al., 2014).”
From Context, attention, and the switch between habit and goal-direction in behavior (2021) · cites this paper as Dezfouli et al., 2014
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
James, W. (1891) · Henry Holt and Company eBooks · doi:10.1037/11059-000 · Cited by 25,799 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641 · see citing works
Reason, J. (1990) · doi:10.1017/cbo9781139062367 · Cited by 5,108 · see citing works
Plans and the structure of behavior.
Miller, G. A., Galanter, E., & Pribram, K. H. (1960) · doi:10.1037/10039-000 · Cited by 4,630 · see citing works
Brown, R. W. (1960) · Language · doi:10.2307/411065 · Cited by 3,525 · see citing works
Attentional requirements of learning: Evidence from performance measures
Nissen, M. J., & Bullemer, P. (1987) · Cognitive Psychology · doi:10.1016/0010-0285(87)90002-8 · Cited by 2,966 · see citing works
The problem of serial order in behavior
Lashley, K. S. (1951) · MPG.PuRe (Max Planck Society) · Cited by 2,734 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
A framework for mesencephalic dopamine systems based on predictive Hebbian learning
Montague, P., Dayan, P., & Sejnowski, T. (1996) · The Journal of Neuroscience · doi:10.1523/jneurosci.16-05-01936.1996 · Cited by 2,066 · see citing works
Model-based influences on humans’ choices and striatal prediction errors
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011) · Neuron · doi:10.1016/j.neuron.2011.02.027 · Cited by 1,779 · see citing works
Balleine, B. W., & O'Doherty, J. P. (2010) · Neuropsychopharmacology · doi:10.1038/npp.2009.131 · Cited by 1,775 · see citing works
Self-citationCategorization of action slips
Norman, D. A. (1981) · Psychological Review · doi:10.1037/0033-295x.88.1.1 · Cited by 1,736 · see citing works
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,464 · see citing works
Movement control in skilled motor performance.
Keele, S. W. (1968) · Psychological Bulletin · doi:10.1037/h0026739 · Cited by 1,232 · see citing works
Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2004.03095.x · Cited by 1,226 · see citing works
Self-citationTonic dopamine: Opportunity costs and the control of response vigor
Niv, Y., Daw, N. D., Joel, D., & Dayan, P. (2007) · Psychopharmacology · doi:10.1007/s00213-006-0502-4 · Cited by 1,133 · see citing works
Cerebral mechanisms in behavior.
indicated, N. A. (1952) · Journal of Consulting Psychology · doi:10.1037/h0051344 · Cited by 918 · see citing works
GPS, A PROGRAM THAT SIMULATES HUMAN THOUGHT
Newell, A., & Simon, H. A. (1988) · Elsevier eBooks · doi:10.1016/b978-1-4832-1446-7.50040-6 · Cited by 819 · see citing works
Rosenbaum, D. Α. (1990) · doi:10.1016/b978-0-12-374226-1.x0001-0 · Cited by 684 · see citing works
Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
Botvinick, M. M., Niv, Y., & Barto, A. G. (2009) · Cognition · doi:10.1016/j.cognition.2008.08.011 · Cited by 646 · see citing works
Neural computations underlying arbitration between model-based and model-free learning
Lee, S. W., Shimojo, S., & O’Doherty, J. P. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.028 · Cited by 642 · see citing works
Variations in the Sensitivity of Instrumental Responding to Reinforcer Devaluation
Adams, C. D. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400878 · Cited by 635 · see citing works
Start/stop signals emerge in nigrostriatal circuits during sequence learning
Jin, X., & Costa, R. M. (2010) · Nature · doi:10.1038/nature09263 · Cited by 611 · see citing works
Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories
Barnes, T. D., Kubota, Y., Hu, D., Jin, D. Z., & Graybiel, A. M. (2005) · Nature · doi:10.1038/nature04053 · Cited by 595 · see citing works
Acquisition of a memory skill.
Ericsson, K. A., Chase, W. G., & Faloon, S. (1980) · Science · doi:10.1126/science.7375930 · Cited by 579 · see citing works
YIN, H., KNOWLTON, B., & BALLEINE, B. (2006) · Behavioural Brain Research · doi:10.1016/j.bbr.2005.07.012 · Cited by 532 · see citing works
Self-citationDISTURBANCE OF SEQUENTIAL MOVEMENTS IN PATIENTS WITH PARKINSON'S DISEASE
Benecke, R., Rothwell, J. C., Dick, J., Day, B. L., & MARSDEN, C. D. (1987) · Brain · doi:10.1093/brain/110.2.361 · Cited by 514 · see citing works
Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences
Jin, X., Tecuapetla, F., & Costa, R. M. (2014) · Nature Neuroscience · doi:10.1038/nn.3632 · Cited by 499 · see citing works
Cortical and basal ganglia contributions to habit learning and automaticity
Ashby, F. G., Turner, B. O., & Horvitz, J. C. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.02.001 · Cited by 481 · see citing works
Thorn, C. A., Atallah, H. E., Howe, M. W., & Graybiel, A. M. (2010) · Neuron · doi:10.1016/j.neuron.2010.04.036 · Cited by 412 · see citing works
Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation
Faure, A., Haberland, U., Condé, F., & Massioui, N. E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3894-04.2005 · Cited by 402 · see citing works
Postconditioning devaluation of a reinforcer affects instrumental responding.
Colwill, R., & Rescorla, R. (1985) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.11.1.120 · Cited by 375 · see citing works
Differential roles of monkey striatum in learning of sequential hand movement
Miyachi, S., Hikosaka, O., Miyashita, K., Kárádi, Z., & Rand, M. K. (1997) · Experimental Brain Research · doi:10.1007/pl00005669 · Cited by 374 · see citing works
Speed/accuracy trade-off between the habitual and the goal-directed processes
Keramati, M., Dezfouli, A., & Piray, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002055 · Cited by 346 · see citing works
Neurobiology of Sensation and Reward
Gottfried, J. A. (2011) · CRC Press eBooks · doi:10.1201/b10776 · Cited by 331 · see citing works
Expert Chess Memory: Revisiting the Chunking Hypothesis
Gobet, F., & Simon, H. A. (1998) · Memory · doi:10.1080/741942359 · Cited by 320 · see citing works
Habits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297 · see citing works
Self-citationPostconditioning devaluation of a reinforcer affects instrumental responding.
Colwill, R. M., & Rescorla, R. A. (1985) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.11.1.120 · Cited by 273 · see citing works
Miyachi, S., Hikosaka, O., & Lu, X. (2002) · Experimental Brain Research · doi:10.1007/s00221-002-1213-7 · Cited by 262 · see citing works
Wymbs, N. F., Bassett, D. S., Mucha, P. J., Porter, M. A., & Grafton, S. T. (2012) · Neuron · doi:10.1016/j.neuron.2012.03.038 · Cited by 231 · see citing works
Dezfouli, A., & Balleine, B. W. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003364 · Cited by 221 · see citing works
Self-citationRhodes, B. J., Bullock, D. H., Verwey, W. B., Averbeck, B. B., & Page, M. P. A. (2004) · Human Movement Science · doi:10.1016/j.humov.2004.10.008 · Cited by 219 · see citing works
Nigrostriatal Dopamine System in Learning to Perform Sequential Motor Tasks in a Predictive Manner
Matsumoto, N., Hanakawa, T., Maki, S., Graybiel, A. M., & Kimura, M. (1999) · Journal of Neurophysiology · doi:10.1152/jn.1999.82.2.978 · Cited by 181 · see citing works
Acquisition of hierarchical control over the temporal organization of a skill.
Pew, R. W. (1966) · Journal of Experimental Psychology · doi:10.1037/h0023100 · Cited by 174 · see citing works
Control of automated behavior: insights from the discrete sequence production task
Abrahamse, E. L., Ruitenberg, M. F. L., de Kleine, E., & Verwey, W. B. (2013) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2013.00082 · Cited by 164 · see citing works
Motor Sequences and the Basal Ganglia: Kinematics, Not Habits
Desmurget, M., & Turner, R. S. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.0163-10.2010 · Cited by 157 · see citing works
Grossberg, S., & Pearson, L. R. (2008) · Psychological Review · doi:10.1037/a0012618 · Cited by 131 · see citing works
NMDA Receptors in Dopaminergic Neurons Are Crucial for Habit Learning
Wang, L. P., Li, F. R., Wang, D., Xie, K., Wang, D., Shen, X., & Tsien, J. Z. (2011) · Neuron · doi:10.1016/j.neuron.2011.10.019 · Cited by 122 · see citing works
Carr, H. A., & Watson, J. B. (1908) · Journal of Comparative Neurology and Psychology · doi:10.1002/cne.920180103 · Cited by 110 · see citing works
Choice and contingency in the development of behavioral autonomy during instrumental conditioning.
Kosaki, Y., & Dickinson, A. (2010) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/a0016887 · Cited by 101 · see citing works
Tremblay, P.-L., Bédard, M., Langlois, D., Blanchet, P. J., Lemay, M., & Parent, M. J. (2010) · Experimental Brain Research · doi:10.1007/s00221-010-2372-6 · Cited by 92 · see citing works
Omission Learning after Instrumental Pretraining
Dickinson, A. (2010) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/713932679 · Cited by 88 · see citing works
The role of response-reinforcer associations increases throughout extended instrumental training
Colwill, R. M., & Rescorla, R. A. (1988) · Animal Learning & Behavior · doi:10.3758/bf03209051 · Cited by 68 · see citing works
(1947) · BMJ · doi:10.1136/bmj.1.4511.890 · Cited by 55 · see citing works
Tremblay, P.-L., Bédard, M., Lévesque, M., Chebli, M., Parent, M. J., Courtemanche, R., & Blanchet, P. J. (2008) · Behavioural Brain Research · doi:10.1016/j.bbr.2008.11.002 · Cited by 48 · see citing works
Lévesque, M., Bédard, M., Courtemanche, R., Tremblay, P.-L., Scherzer, P. B., & Blanchet, P. J. (2007) · Experimental Brain Research · doi:10.1007/s00221-007-1010-4 · Cited by 32 · see citing works
Environmental statistics and the trade-off between model-based and TD learning in humans
Simon, D. A., & Daw, N. D. (2011) · neural information processing systems · Cited by 23 · see citing works
Sensation, Incentive Learning, and the Motivational Control of Goal-Directed Action
Gottfried, J. A. (2011) · CRC Press eBooks · doi:10.1201/b10776-21 · Cited by 21 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [3]Dickinson A. 1994. Instrumental conditioning. In Animal cognition and learning (ed. Mackintosh NJ.), pp. 4–79. London, UK: Academic Press. [Google Scholar]
- [26]Reason J. 1990. Human error. Cambridge, UK: Cambridge University Press. [Google Scholar]
Cited by
Sorted by citations
Wood, W., & Rünger, D. (2016) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033417 · Cited by 1,445 · see citing works
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · see citing works
“This process is often described as the acquisition of sets of associations (Dickinson, 2012), including (bidirectional) actionoutcome associations.”
BackgroundWhen Does Model-Based Control Pay Off?
Kool, W., Cushman, F. A., & Gershman, S. J. (2016) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005090 · Cited by 211 · see citing works
Humans primarily use model-based inference in the two-stage task
Feher da Silva, C., & Hare, T. A. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0905-y · Cited by 153 · see citing works
Context, attention, and the switch between habit and goal-direction in behavior
Bouton, M. E. (2021) · Learning & Behavior · doi:10.3758/s13420-021-00488-z · Cited by 100 · see citing works
Regulation of habit formation in the dorsal striatum
Malvaez, M., & Wassum, K. M. (2018) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.11.005 · Cited by 96 · see citing works
Corticostriatal foundations of habits
Seger, C. A. (2018) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.01.006 · Cited by 38 · see citing works
The principles of goal-directed decision-making: from neural mechanisms to computation and robotics
Pezzulo, G., Verschure, P. F. M. J., Balkenius, C., & Pennartz, C. M. A. (2014) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2013.0470 · Cited by 35 · see citing works
Action sequence learning, habits, and automaticity in obsessive-compulsive disorder
Banca, P., Herrojo Ruiz, M., Gonzalez-Zalba, M. F., Biria, M., Marzuki, A. A., Piercy, T., Sule, A., Fineberg, N. A., & Robbins, T. W. (2024) · eLife · doi:10.7554/elife.87346 · Cited by 9 · see citing works
- Authors
- Dezfouli, A., Lingawi, N. W., & Balleine, B. W.
- Affiliations
- The University of Sydney (AU)
- Venue
- Philosophical Transactions of the Royal Society B: Biological Sciences
- Publisher
- The Royal Society
- Published
- 2014-11-05
- Volume
- 369
- Issue
- 1655
- Pages
- 20130482
- ISSN
- 0962-8436 (ISSN-L)
- 1471-2970
- PMC
- PMC4186235
- OpenAlex
- W2120101916
- Cited by
- 139
- References
- 63
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 14 · 2023: 15 · 2022: 8 · 2021: 10 · 2020: 15 · 2019: 16 · 2018: 16 · 2017: 11 · 2016: 10 · 2015: 6 · 2014: 4
- FWCI
- 5.77
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Philosophical Transactions of the Royal Society B Biological Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute of Mental Health (MH56446)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Social Sciences › Psychology › Applied Psychology › Behavioral Health and Interventions
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- habit, action sequences, goal-directed action, outcome value, overtraining, extinction, decision-making, automatism
- MeSH terms
- Animals, Decision Making, Goals, Habits, Humans, Learning, Rodentia
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
