Context, attention, and the switch between habit and goal-direction in behavior

In the paper

“However, instead of erasing R-O, S-R exists along with it, coded by a separate circuit in the brain (e.g., Balleine, 2019; Balleine & O'Doherty, 2010; Gruber & McDonald, 2012; Robbins et al., 2019).”

What the sources say

• Not checked yetcites Balleine, 2019
• Supported

  The “Neural Theories of Action Selection” section says goal-directed and habitual learning processes are concurrently engaged and can be maintained in parallel, supporting the claim that S-R learning does not erase R-O learning. The paper also identifies distinct corticostriatal substrates for the two forms of control in its sections on goal-directed and habit learning. (LLM, conf=0.94)

  cites Balleine & O'Doherty, 2010 source
• Supported

  The Conclusion (page unavailable) says learning and memory systems function in parallel even when one is not the dominant response strategy, supporting coexistence rather than erasure. The striatum sections (pages unavailable) associate R-O learning with the dorsomedial striatum and S-R habits with the dorsolateral striatum. (LLM, conf=0.88)

  cites Gruber & McDonald, 2012 source

In the paper

“Habit develops when the cue always predicts the response will be reinforced, but not when the cue is associated with a reinforcer unpredictably on a random half of the trials.”

What the sources say

• Not checked yetcites Thrailkill et al., 2018
• Not checked yetcites Thrailkill et al., 2021

In the paper

“Additional experiments found that the interleaved training of R2 could be reinforced by a different reinforcer (sucrose when grain had been trained or grain when sucrose had been trained). And it turned out that R2 was not necessary; all we needed to do was present the reinforcers freely (not contingent on any response) during the sessions in Context B.”

What the sources say

• Not checked yetcites Trask et al., 2020 source

In the paper

“In a discriminated operant procedure, habit developed when the reinforcer was predictably delivered in every S, but not when it was unpredictably delivered in only half of them.”

What the sources say

• Not checked yetcites Thrailkill et al., 2018, Experiment 4

In the paper

“Sixty-six sessions was a lot of training by most standards – it involved roughly 1,054 pairings of the response and reinforcer, whereas free-operant methods can create habit in about 360”

What the sources say

• Not checked yetcites Thrailkill & Bouton, 2015

In the paper

“We found that inactivation of the prelimbic cortex suppressed moderately trained (but not extensively trained) instrumental behavior, and that inactivation of infralimbic cortex suppressed the extensively trained (but not moderately trained) response.”

What the sources say

• Not checked yetcites Shipman et al., 2018 source

In the paper

“When R1 was trained the same way but without the interleaved R2 training, or when R2 training began after R1 training had finished, reinforcer devaluation tests determined that R1 was indeed a habit.”

What the sources say

• Not checked yetcites Trask et al., 2020 source

In the paper

“When we trained rats with a longer 8-min, rather than a 30-s, tone stimulus setting the occasion for lever pressing on the RI-30 schedule, habit did develop after extended training; rats that subsequently acquired a taste aversion to the pellet continued to lever press during the test despite their aversion to the reinforcer”

What the sources say

• Not checked yetcites Thrailkill et al., 2018, Experiments 2 and 3

In the paper

“But there was a rub: The effects of reinforcer devaluation suggested that both the moderately trained R2 and extensively trained R1 were goal-directed actions.”

What the sources say

• Not checked yetcites Shipman et al., 2018 source

In the paper

“As Thrailkill et al. (2018) had found with stimuli that were five times as long and a reinforcement schedule that was 1/10 as rich, the CRF procedure produced habit, whereas PRF maintained goal-direction.”

What the sources say

• Not checked yetcites Thrailkill et al., 2021

In the paper

“With a lever-insertion stimulus, which may be more salient, habit does develop even when the reinforcer is uncertain, perhaps because the stimulus (and the strength of its presumed prediction of the reinforcer) is less decremented by the non-reinforced trials”

What the sources say

• Not checked yetcites Thrailkill et al., 2021

In the paper

“Thus, with as many as 66 sessions, the behavior was still a goal-directed action.”

What the sources say

• Not checked yetcites Thrailkill et al., 2018

In the paper

“With a standard tone stimulus, habit develops if the tone predicts that a reinforcer will occur whenever it is presented, but not when reinforcer delivery is uncertain”

What the sources say

• Not checked yetcites Thrailkill et al., 2018
• Not checked yetcites Thrailkill et al., 2021

In the paper

“In either case, responding returned (was “renewed”), and by comparing the groups for which the pellets had been devalued or not, we could see that the response that had been trained as an action renewed as action and the response trained as a habit renewed as a habit.”

What the sources say

• Not checked yetcites Steinfeld & Bouton, 2020

In the paper

“it is also not clear whether existing reinforcement learning theories can accommodate the evidence that, with an equal number of response-reinforcer pairings, habit develops when the reinforcers are predictable, and not when they are not (Thrailkill et al., 2018, 2021).”

What the sources say

• Not checked yetcites Thrailkill et al., 2018, 2021

In the paper

“habits develop more rapidly with behaviors reinforced on interval as opposed to ratio schedules”

What the sources say

• Not checked yetcites Dickinson et al., 1983 source

In the paper

“Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change.”

In the paper

“We and others have shown that extinction weakens instrumental responding, but does not erase the original learning. Instead, extinction involves a form of context-specific inhibition (rather than unlearning) of the response”

What the sources say

• Not checked yetcites Bouton, 1993 source
• Not checked yetcites Bouton, 2019 source
• Not checked yetcites Bouton, Maren, & McNally, 2021b

In the paper

“according to at least one venerable theory of associative learning, the Pearce-Hall model of Pavlovian learning (Pearce & Hall, 1980), attention paid to cues that predict a reinforcer should decline as the reinforcer becomes better and better predicted during extended conditioning.”

What the sources say

• Not checked yetcites Pearce & Hall, 1980 source

In the paper

“On the other hand, the attentional perspective, which holds that a behavior becomes a habit once you aren’t attending to it, suggests more flexibility: A behavior’s status as habit or action might in principle change as attention changes.”

In the paper

“Thus, when the reinforcer is imperfectly predicted by S, we might pay more attention to both the predictor and the behavior itself. That may be the reason why a discriminated operant that is reinforced during only some of the S presentations may keep the status of goal-directed action.”

In the paper

“The results of experiments in rats (e.g., Kaye & Pearce, 1984; Wilson et al., 1992) and humans (Hogarth et al., 2008) suggest that a visual cue that has an uncertain (50%) relation to an outcome maintains orienting behavior, whereas a cue that has a perfect (100%) relation to an outcome does not.”

What the sources say

• Not checked yetcites Kaye & Pearce, 1984
• Not checked yetcites Wilson et al., 1992
• Not checked yetcites Hogarth et al., 2008 source

In the paper

“We know that the “context” (e.g., the Skinner box itself) can be the stimulus (S) that controls habits learned with free-operant methods (Thrailkill & Bouton, 2015)– when we test lever pressing in a different context, the habit decreases in strength.”

What the sources say

• Not checked yetcites Thrailkill & Bouton, 2015

In the paper

“Although a little S-R learning may occur early during instrumental training (e.g., Thrailkill & Bouton, 2015), the action➔habit transition seems to fit a general organizing principle of associative learning.”

What the sources say

• Not checked yetcites Thrailkill & Bouton, 2015

In the paper

“The picture that emerges in our studies of action➔habit conversion is one of flexibility rather than rigidity. Habits are not really fixed or confining.”

In the paper

“The simplest interpretation of these results after action➔habit conversion is that goal-direction transfers well across contexts (i.e., from A to C), whereas habit does not transfer as effectively from B to C.”

In the paper

“Given the tradeoff, the arbitrator selects between the two systems according to various criteria, such as the accuracy or prediction error connected with each (Daw et al., 2005; Lee et al., 2014) or the prevailing rate of reinforcement (Dezfouli et al., 2014).”

What the sources say

• Not checked yetcites Daw et al., 2005 source
• Not checked yetcites Lee et al., 2014 source
• Not checked yetcites Dezfouli et al., 2014 source

In the paper

“Although actions seem less affected by context change than habits are, the action➔habit conversion and context-dependent interference by habit allows the organism to switch between the two with changes in context (see below).”

In the paper

“In our animals and with our methods, habit learning does not seem permanent, but like extinction, interferes with goal-direction in a context-specific way.”

In the paper

“Habits are context-specific, but actions are not (see also Thrailkill & Bouton, 2015).”

What the sources say

• Not checked yetcites Thrailkill & Bouton, 2015

In the paper

“Other frameworks that emphasize multiple parallel memory systems in the brain (e.g., Gruber & McDonald, 2012; McDonald & White, 1993; White, 1996) also allow behavioral flexibility.”

What the sources say

• Not checked yetcites Gruber & McDonald, 2012 source
• Not checked yetcites McDonald & White, 1993 source
• Not checked yetcites White, 1996 source

In the paper

“When a behavior is in habit mode, it exploits the tried and true; when it is in action mode, it can presumably explore new opportunities or connections (cf. Beeseley et al., 2015)”

What the sources say

• Not checked yetcites Beeseley et al., 2015 source

In the paper

“Approaches to action and habit in the reinforcement learning tradition (e.g., Sutton & Barto, 1998) have also often accepted the coexistence of action and habit systems and allowed one or the other to be selected by a hypothetical “arbitrator” for behavioral control (e.g.,Daw et al., 2005 ; Lee et al., 2014).”

What the sources say

• Not checked yetcites Sutton & Barto, 1998
• Not checked yetcites Daw et al., 2005 source
• Not checked yetcites Lee et al., 2014 source

In the paper

“The parallel with habit learning is apparent: In the action➔habit conversion, like the conditioning➔extinction (or conditioning➔punishment) conversion, the first-thing learned (goal-directed action) remains at least partially intact after second learning. It also transfers relatively well across contexts. In contrast, the second-thing learned (habit) does not.”

In the paper

“Actions or behaviors based on a “model-based” system (which encodes representations of all the behavior’s possible consequences) are considered rich but computationally expensive, while habits or behaviors based on a “model-free” system (which encodes only the behavior’s “cached” or average value) may be efficient but inflexible.”

In the paper

“It should be noted that the dual-process or “associative-cybernetic” approaches of Dickinson and Balleine and colleagues (e.g., Balleine & Ostlund, 2007; de Wit & Dickinson, 2009; Dickinson, 2012) do posit separate and co-existing action and habit memories.”

What the sources say

• Not checked yetcites Balleine & Ostlund, 2007 source
• Not checked yetcites de Wit & Dickinson, 2009 source
• Not checked yetcites Dickinson, 2012 source

In the paper

“As noted earlier, the development of a habit and automaticity is theoretically functional because it preserves limited space in working memory for processing other things.”

In the paper

“On this account, the switch back to R-O after unexpected reinforcers or context change (Bouton et al., 2020; Steinfeld & Bouton, 2021; Trask et al., 2020) would be created by merely increasing attention to R (and the consequent associative retrieval of O).”

What the sources say

• Not checked yetcites Bouton et al., 2020 source
• Not checked yetcites Steinfeld & Bouton, 2021 source
• Not checked yetcites Trask et al., 2020 source

In the paper

“The attention explanation given above is based on a well-known theory of attention and its role in Pavlovian learning (Pearce & Hall, 1980).”

What the sources say

• Not checked yetcites Pearce & Hall, 1980 source

In the paper

“The other idea that the reinforcer must be predictable in order to create habit (Thrailkill et al., 2018, 2021) seems to lie outside the scope of existing habit theories.”

What the sources say

• Not checked yetcites Thrailkill et al., 2018, 2021

In the paper

“Mackintosh’s attention rule is consistent with a number of results (e.g., Le Pelley et al., 2016; Pearce & Mackintosh, 2010), and contemporary approaches to attention and learning now suppose that both the Mackintosh and Pearce-Hall rules can play a role in any situation (e.g., Le Pelley, 2004; Pearce & Mackintosh, 2010; see Chao et al., 2021, for recent discussion).”

What the sources say

• Not checked yetcites Le Pelley et al., 2016 source
• Not checked yetcites Pearce & Mackintosh, 2010
• Not checked yetcites Le Pelley, 2004 source

In the paper

“Perhaps the habit context mostly modulates attention to the response instead of switching between S-R and R-O.”

In the paper

“To explain why the presentation of surprising reinforcers also seems to activate action after habit learning (Bouton et al., 2020; Trask et al., 2020), we could thus note that they may provide a form of context change (e.g., Bouton et al., 2021b).”

What the sources say

• Not checked yetcites Bouton et al., 2020 source
• Not checked yetcites Trask et al., 2020 source
• Not checked yetcites Bouton et al., 2021b

In the paper

“The result of this simple mechanism is that in the habit context, the habit association is active and action is suppressed, whereas outside the habit context, S-R is not activated, and goal-direction(R-O) is expressed.”

In the paper

“In our experiments (Thrailkill et al., 2018, 2021), attention to the response was theoretically maintained by making the reinforcer uncertain from trial to trial; this maintained goal-direction and prevented habit.”

What the sources say

• Not checked yetcites Thrailkill et al., 2018, 2021

In the paper

“In a different account of attention and learning, Mackintosh (1975) argued that attention increases (rather than decreases) to the best predictor of a reinforcer.”

What the sources say

• Not checked yetcites Mackintosh (1975) source

In the paper

“Extended practice will thus normally decrease attention to the response; when this occurs, habit is created, and the response can be elicited by S without the organism processing O.”

In the paper

“The Pearce-Hall rule predicts that attention to the response will decline – and thus incorrectly implies habit.”

In the paper

“Outcome uncertainty maintains attention to behavior and therefore action, whereas outcome predictability reduces it and creates habit.6”

In the paper

“Habit is not a fixed endpoint; its acquisition does not erase R-O and make it permanently unavailable.”

In the paper

“But the research reviewed here emphasizes that goal-direction is not erased during habit learning, and that it can be recovered under the right conditions.”

In the paper

“It is worth noting that the range of cues that can constitute context is arguably very wide, potentially including time, drug states, deprivation and stress states, and other behaviors among other things (e.g., Bouton, 2019; Bouton et al., 2021a, b).”

What the sources say

• Not checked yetcites Bouton, 2019 source
• Not checked yetcites Bouton et al., 2021a, b

In the paper

“For example, the Rescorla-Wagner model (Rescorla & Wagner, 1972) and its extensions by Wagner (e.g., Wagner, 1978, 1981, 2008; Wagner & Brandon, 1989) emphasize that the reinforcer becomes less surprising and less processed as classical conditioning approaches asymptote.”

What the sources say

• Not checked yetcites Rescorla & Wagner, 1972
• Not checked yetcites Wagner, 1978, 1981, 2008
• Not checked yetcites Wagner & Brandon, 1989

In the paper

“Crucially, the activation of S-R is context-dependent, either because the context itself provides the S associated with R, or because (as we have seen in other retroactive interference paradigms) the context controls the second thing learned (e.g., Bouton, 1993; Nelson, 2002).”

What the sources say

• Not checked yetcites Bouton, 1993 source
• Not checked yetcites Nelson, 2002 source

In the paper

“With a ratio schedule, the higher the rate of responding, the higher the rate of reinforcement, and response rate is thus a good predictor of the reinforcer.”

In the paper

“If habits do play a role in addiction, they may do so because the habit-action switch becomes sticky (see also Balleine et al., 2009; Lee et al., 2014). As a consequence, once a habit is learned, it may be difficult to switch it back to goal-directed status.”

What the sources say

• Not checked yetcites Balleine et al., 2009 source
• Not checked yetcites Lee et al., 2014 source

In the paper

“Thus, if there is a good predictor, the organism will tune it in rather than tune it out. Crucially, the organism will also tune out contemporaneous predictors that are not as good as (or no better than) the best one.”

What the sources say

• Not checked yetcites Mackintosh (1975) source

In the paper

“Although things work this way in instrumental learning (i.e., habit is eventually observed), classically conditioned responding is still suppressed by reinforcer devaluation even after extensive training (e.g., Holland, 1990, 2005).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites Holland, 1990, 2005

In the paper

“If an instrumental reinforcer were likewise less processed after asymptotic training, it could provide a different reason why reinforcer devaluation becomes ineffective: Reinforcer devaluation will not affect instrumental responding if the animal is not processing the reinforcer.”

In the paper

“The two conclusions are consistent with what we know about interference and attention in associative learning, which until now have had too little contact with the science of goal-directed actions and habits.”