Rethinking motor learning and savings in adaptation paradigms: model-free memory for successful actions combines with internal models

May combine two claims.

In Adp+Rep−, hand directions at probe targets were clockwise and gradually converged to naïve performance, predicted by the state-space model.

“As expected for generalization of adaptation, hand directions in Adp+Rep− were clockwise and gradually converged to naïve performance and this was predicted well by the state-space model (Fig. 2E).”

May combine two claims.

In Adp+Rep+, hand directions at probe targets crossed and showed increasing bias away from naive directions as probe directions moved further from the repeated direction.

“Adp+Rep+ crossed and began to show an increasing bias away from naive directions as the probe directions moved further away from the repeated direction in hand space (Fig. 2F); this is the opposite of the expectation for adaptation but entirely”

May combine two claims.

Mean movement direction at the end of training was 76.0±2.1° for Adp+Rep− and 71.6±1.3° for Adp+Rep+.

“The mean movement direction at the end of training across subjects was 76.0±2.1° (mean±s.d.) for Adp+Rep− (supplementary Fig. S1D) and the mean movement direction at the end of training was 71.6±1.3° (mean±s.d.) for Adp+Rep+.”

May combine two claims.

No savings was observed for the repetition-only group Adp−Rep+; its learning rate was not significantly different from naïve training in Adp−Rep− (0.16±0.04 trial-1 v.s. 0.13±0.02 trial-1, two-tailed t-test, t(10)=0.594, p=0.565).

“In contrast, no savings was observed for the repetition-only group, Adp−Rep+ (Fig. 4B); indeed the learning rate was not significantly different from naïve training in Adp−Rep− (0.16±0.04 trial-1 v.s. 0.13±0.02 trial-1, two-tailed t-test, t(10)=0.594, p=0.565) (Fig. 4C).”

May combine two claims.

The bias in Adp+Rep+ was comparable for trained and untrained targets, with increasing absolute size of bias for farther away targets in both directions.

“To summarize Experiment 1, adaptation to a target sequence that led to movements distributed around the repeated direction in hand space led to a bias towards the repeated direction that was comparable for trained and untrained targets, with increasing absolute size of bias for farther away targets in both directions.”

May combine two claims.

An adaptation protocol with movement repetition led to clear savings whereas neither adaptation alone nor repetition alone led to any savings.

“To summarize Experiment 2, an adaptation protocol with movement repetition led to clear savings whereas neither adaptation alone nor repetition alone led to any savings.”

May combine two claims.

The hand direction versus target direction relationship was well described with a single linear fit (0 < slope ≪ 1) that had a close fit to both training and probe targets.

“The hand direction versus target direction relationship was well described with a single linear fit (0 < slope ≪ 1) that had a close fit to both training and probe targets.”

May combine two claims.

Subjects in Adp+Rep− rotated their hand movement by an average of -13.97±1.41°, or about 70% adaptation on average for all targets.

“subjects rotated their hand movement by an average of --13.97±1.41° (mean±s.d.) (the vertical displacement from the naïve line in Fig. 2C), or about 70% adaptation on average for all targets.”

May combine two claims.

Adp+Rep− (0.12±0.02 trial-1) were no faster than the naïve training control and showed no savings (t(14)=-0.39, p=0.70).

“In contrast, Adp+Rep− (0.12±0.02 trial-1) were no faster than the naïve training control and showed no savings (t(14)=-0.39, p=0.70)(Fig. 4A,C).”

May combine two claims.

The bias generated during Adp+Rep+ had a slope of 0.32±0.03 for trained targets, comparable to 0.42±0.04 for probe targets.

“Interestingly, the bias generated during Adp+Rep+, which can be plotted as the dependent relationship between displayed targets and hand movement direction, was also apparent during learning, with a slope of 0.32 (±0.03) for the trained targets that was comparable to the slope for the probe targets (0.42±0.04).”

May combine two claims.

SAME-SOLNhand showed savings for the -30° rotation after training on +30°; the relearning rate during test (0.23±0.03 trial-1) was significantly faster than initial learning (paired one-tailed t(5)=-2.371, p=0.03).

“Consistent with the prediction of operant reinforcement, SAME-SOLNhand showed savings for the -30° rotation after training on +30° (Fig. 6A); the relearning rate during test (0.23±0.03 trial-1) was significantly faster than initial learning (Fig. 6C) (paired one-tailed t(5)=-2.371, p=0.03).”

May combine two claims.

There was marked savings for a +25° rotation for Adp+Rep+, but no savings at all for Adp+Rep−.

“Thus we saw a marked savings effect for a +25° rotation for Adp+Rep+, but no savings at all for Adp+Rep−.”

May combine two claims.

SAME-SOLNvisual showed no savings; its relearning rate during test was 0.11±0.02 trial-1 (paired one-tailed t(5)=0.238, p=0.411).

“In contrast, no savings were seen for SAME-SOLNvisual which had a relearning rate of 0.11±0.02 trial-1 during test (Fig. 6B) (paired one-tailed t(5)=0.238, p=0.411).”

May combine two claims.

We found evidence for operant reinforcement in the form of savings - subjects showed faster relearning when rotations of either sign required an adapted solution that coincided with a previously successful hand movement direction.

“We found evidence for operant reinforcement in the form of savings – subjects showed faster relearning when rotations of either sign (clockwise or counter-clockwise) required an adapted solution that coincided with a previously successful hand movement direction.”

May combine two claims.

For Adp+Rep+, the adaptation model predicted hand directions relatively well in the early phase of training (r2=0.753) but then failed as subjects developed a directional bias beyond what was expected from adaptation alone (r2=-0.502).

“For Adp+Rep+, the adaptation model was able to predict hand directions relatively well in the early phase of training (r2=0.753) but then began to fail as subjects developed a directional bias beyond what was expected from adaptation alone (r2=-0.502) (Fig. 2D, asymptotic training).”

May combine two claims.

Adp+Rep+ showed marked savings whereas Adp+Rep− showed no savings, even though they adapted to the same mean rotation.

“Adp+Rep+ showed marked savings whereas Adp+Rep−showed no savings, even though they adapted to the same mean rotation.”

May combine two claims.

Neither repetition alone nor adaptation alone led to savings; the association of the two is critical.

“Neither repetition alone nor adaptation alone led to savings, which suggests that it is the association of the two that is critical.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

May combine two claims.

Use-dependent plasticity and operant reinforcement both occur along with adaptation.

“We have shown that use-dependent plasticity and operant reinforcement both occur along with adaptation.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

Probably not a substantial claim: may not pass S3 (load-bearing).

The state space model (SSM) fits trial-to-trial perturbation data extremely well.

“Despite its apparent simplicity, the SSM framework fits trial-to-trial perturbation data extremely well (Ethier et al., 2008; Huang and Shadmehr, 2007; Scheidt et al., 2001; Smith et al., 2006; Tanaka et al., 2009).”

• cites Ethier et al., 2008 sourceNot checked yet

  Not checked in the free run.

• cites Huang and Shadmehr, 2007Not checked yet

  Not checked in the free run.

• cites Scheidt et al., 2001 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The association of movement repetition with successful adaptation is necessary and sufficient for savings.

“These results suggest that the association of movement repetition with successful adaptation is necessary and sufficient for savings.”

Classified as probably not a substantial claim.

A basic single-timescale SSM predicts that learning should always occur at the same rate, regardless of past experience.

“In particular, relearning of a given perturbation for a second time is faster than initial learning – a phenomenon known as savings (Ebbinghaus, 1913; Kojima et al., 2004; Krakauer et al., 2005; Smith et al., 2006; Zarahn et al., 2008) – whereas a basic single-timescale SSM predicts that learning should always occur at the same rate, regardless of past experience (Zarahn et al., 2008).”

• cites Zarahn et al., 2008 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

SSM variants that include multiple timescales of learning explain savings over short timescales but fail to predict savings following a prolonged period of washout.

“Although SSM variants that include multiple timescales of learning (Kording et al., 2007; Smith et al., 2006) are able to explain savings over short timescales, this approach fails to predict the fact that savings still occurs following a prolonged period of washout of initial learning (Krakauer et al., 2005; Zarahn et al., 2008).”

• cites Kording et al., 2007 sourceNot checked yet

  Not checked in the free run.

• cites Smith et al., 2006 sourceNot checked yet

  Not checked in the free run.

• cites Krakauer et al., 2005 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Results of Experiment 3 suggest that savings is attributable to a model-free operant memory for actions and not to faster re-learning or re-expression of a previously learned internal model.

“In summary, the results of Experiment 3 suggest that savings is attributable a model-free operant memory for actions and not to faster re-learning or re-expression of a previously learned internal model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Biases can be observed in parallel with acquisition of an internal model along the task-irrelevant dimension in a redundant task.

“More recently it has been shown that biases can be observed in parallel with acquisition of an internal model along the task-irrelevant dimension in a redundant task (Diedrichsen et al., 2010).”

• cites Diedrichsen et al., 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

A model-free process based on repeated actions is in operation in Adp+Rep+ but not Adp+Rep−.

“These results are opposite of what would be predicted if the observed behavior were solely due to adaptation of an internal model and show that a model-free process based on repeated actions is in operation in Adp+Rep+ but not Adp+Rep−.”

Classified as probably not a substantial claim.

The core idea in all these models is that savings results from either faster re-acquisition or re-expression of a previously learned internal model.

“The core idea in all of these models is that savings is the result of either faster re-acquisition or re-expression of a previously learned internal model; i.e., they all explain savings within a model-based learning framework.”

May combine two claims.

Errors in Adp+Rep+ were first reduced through adaptation but then further reduced through mechanisms other than adaptation.

“This suggests that errors were first reduced through adaptation but then were further reduced through mechanisms other than adaptation.”

May combine two claims.

The process underlying savings is model-free but distinct from use-dependent plasticity.

“Here we will argue that the process underlying savings is also model-free but distinct from use-dependent plasticity.”

May combine two claims.

The study had power of 0.8, so the negative results are likely true negatives.

“We had power of 0.8 (see Methods) and thus the negative results are likely true negatives.”

May combine two claims.

Adaptation itself can act like a channel but in the task-relevant dimension - it not only reduces visual error but also guides subjects' hand towards a new path in hand space.

“The critical difference between our study and that by Diedrichsen and colleagues is that we reasoned that adaptation itself can act like a channel but in the task-relevant dimension – it not only reduces visual error but also guides subjects' hand towards a new path in hand space.”

May combine two claims.

The biases observed in the setting of adaptation appear larger compared to those induced by repetition alone, which suggests that repetition in the context of reducing errors in response to a perturbation may in itself generate reward that modulates use-dependent plasticity.

“Interestingly, the biases we observed here in the setting of adaptation appear to be larger compared to those induced by repetition alone, which suggests that repetition in the context of reducing errors in response to a perturbation may in itself generate reward that modulates use-dependent plasticity.”

May combine two claims.

The reward landscape is not flat during adaptation but rather is increasingly rewarding as the prediction error decreases; the hand movement induced by the adaptation process will be reinforced through increasing success.

“We reasoned that the reward landscape is not flat during adaptation but rather is increasingly rewarding as the prediction error decreases, i.e., the hand movement that is induced by the adaptation process will be reinforced through increasing success (decreasing error).”

May combine two claims.

Additional learning mechanisms must be present which are model-free in the sense that they are associated with a memory for action independently of an internal model and are likely to be driven directly by task success (i.e. reward).

“Instead, additional learning mechanisms must be present which are model-free in the sense that they are associated with a memory for action independently of an internal model and are likely to be driven directly by task success (i.e. reward).”

• AssumptionRests on an assumption: Any task implicitly carries some notion of success or failure that can be encapsulated mathematically through a cost (reward) function.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the theory of reinforcement learning, the general problem to be solved is to use experience to identify a suitable control policy in an unknown or changing environment.

“In the theory of reinforcement learning, the general problem to be solved is to use experience to identify a suitable control policy in an unknown or changing environment (Sutton and Barto, 1998).”

• cites Sutton and Barto, 1998Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Savings is a form of procedural and motor memory that manifests as faster relearning compared with initial learning.

“Savings is a form of procedural and motor memory that manifests as faster relearning compared with initial learning (Ebbinghaus, 1913; Smith et al., 2006; Zarahn et al., 2008).”

• cites Ebbinghaus, 1913Not checked yet

  Not checked in the free run.

• cites Smith et al., 2006 sourceNot checked yet

  Not checked in the free run.

• cites Zarahn et al., 2008 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Adaptation is the model-based component, while model-free components include use-dependent plasticity and operant reinforcement.

“Adaptation is the model-based component, while model-free components include use-dependent plasticity and operant reinforcement.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Each approach can be advantageous in different circumstances.

“Therefore each approach can be advantageous in different circumstances.”

May combine two claims.

Patients with PD show a selective savings deficit in an error-based motor learning paradigm: they adapt to initial rotation as well as control subjects but do not show savings.

“This is exactly what has been found: patients with PD were able to adapt to initial rotation as well as control subjects but they did not show savings (Bedard and Sanes, 2011; Marinelli et al., 2009).”

• cites Bedard and Sanes, 2011Not checked yet

  Not checked in the free run.

• cites Marinelli et al., 2009 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Adaptation is indifferent to reward outcomes on individual trials, and is purely sensitive to errors in the predicted state of the hand or cursor.

“On the contrary, we believe that adaptation is indifferent to reward outcomes on individual trials, and is purely sensitive to errors in the predicted state of the hand or cursor.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

May combine two claims.

Adaptation is an automatic, obligate, perhaps reward-indifferent, cerebellar-based learning process.

“It appears, therefore, that adaptation is “automatic” – it is an obligate, perhaps reward-indifferent (Mazzoni and Krakauer, 2006), cerebellar-based (Martin et al., 1996a, b; Smith and Shadmehr, 2005; Tseng et al., 2007) learning process that will attempt to”

• cites Mazzoni and Krakauer, 2006 sourceNot checked yet

  Not checked in the free run.

• cites Martin et al., 1996a, bNot checked yet

  Not checked in the free run.

• cites Smith and Shadmehr, 2005 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

A reinforcement process was necessary and sufficient for savings, and use-dependent plasticity is not sufficient for savings.

“We conclude from Experiment 2 that a reinforcement process was necessary and sufficient for savings, and that use-dependent plasticity is not sufficient for savings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Competition between model-based and model-free mechanisms enables the benefits of each to be combined to maximum effect.

“Theoretical treatments have argued that competition between these two mechanisms enables the benefits of each to be combined to maximum effect (Daw et al., 2005).”

• cites Daw et al., 2005 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Subjects were randomly assigned to groups in each experiment.

“They were randomly assigned to groups in each experiment.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

If motor learning studied with error-based paradigms results from the combination of model-free and model-based learning processes, then these paradigms may be well suited to study how the brain modularly assembles complex motor abilities.

“If we are right in our assertion that motor learning studied with error-based paradigms results from the combination of model-free and model-based learning processes then these paradigms may be well suited to study how the brain modularly assembles complex motor abilities.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

Probably not a substantial claim: may not pass S4 (based).

In continuous and high-dimensional action spaces, pure model-free learning is unfeasible, especially if a detailed feedback control policy must be acquired.

“In continuous and high-dimensional action spaces, pure model-free learning is unfeasible, especially if a detailed feedback control policy must be acquired.”

• AssumptionRests on an assumption: Any task implicitly carries some notion of success or failure that can be encapsulated mathematically through a cost (reward) function.

Probably not a substantial claim: may not pass S3 (load-bearing).

The presence of dopamine receptors on cells in primary motor cortex could provide a candidate mechanism for reward-based modulation of use-dependent plasticity.

“The presence of dopamine receptors on cells in primary motor cortex (Huntley et al., 1992; Luft and Schwarz, 2009; Ziemann et al., 1997) could provide a candidate mechanism for reward-based modulation of such use-dependent plasticity (Hosp et al., 2011).”

• cites Huntley et al., 1992 sourceNot checked yet

  Not checked in the free run.

• cites Luft and Schwarz, 2009Not checked yet

  Not checked in the free run.

• cites Ziemann et al., 1997 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

There is a precedent for unconscious reward-based learning in the perceptual learning literature, and the reward can be internal - it does not need to be explicitly provided by the experimenter.

“There is a precedent for such unconscious reward-based learning in the perceptual learning literature, and the reward can be internal – it does not need to be explicitly provided by the experimenter (Seitz et al., 2009).”

• cites Seitz et al., 2009 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Operant reinforcement may require dopaminergic projections to the striatum.

“Finally, operant reinforcement may require dopaminergic projections to the striatum (Wachter et al., 2010).”

• cites Wachter et al., 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The workspace was calibrated so that the image of the cursor indicating hand position fell exactly on the unseen tip of the middle finger's location (i.e., veridical display).

“The workspace was calibrated so that the image of the cursor indicating hand position fell exactly on the unseen tip of the middle finger's location (i.e., veridical display) (Mazzoni et al., 2007).”

• cites Mazzoni et al., 2007 sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Reinforcement algorithms assign value to near-misses even when actual reinforcement is not given on such trials.

“This idea of the value of “near-misses” has been argued for in reinforcement algorithms that assign value to near-misses even when actual reinforcement is not given on such trials (MacLin et al., 2007).”

• cites MacLin et al., 2007 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A recent motor learning model has been conceptualized in terms of the existence of fast and slow error-based processes.

“A recent motor learning model has been conceptualized in terms of the existence of fast and slow error-based processes (Kording et al., 2007; Smith et al., 2006).”

• cites Kording et al., 2007 sourceNot checked yet

  Not checked in the free run.

• cites Smith et al., 2006 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The existence of separate learning processes may indicate an underlying anatomical separation.

“The existence of separate learning processes may indicate an underlying anatomical separation.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

Probably not a substantial claim: may not pass S3 (load-bearing).

The minimum sample size for an independent samples t-test using a two-tailed alpha of 0.05 and power of 0.8, assuming an effect size d = 1.9375, is 6 subjects per group.

“For an independent samples t-test using a two-tailed alpha of 0.05 and power of 0.8, and assuming an effect size d = 1.9375 (computed based on previously reported group means and standard deviation; time constant = 0.47 for savings and 0.16 for naïve, with s.d. = 0.16), the minimum sample size is 6 subjects per group.”

• cites Zarahn et al., 2008 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

All subjects were naïve to the purpose of the study.

“All subjects were naïve to the purpose of the study and gave informed consent in compliance to guidelines set forth by the Columbia University Medical Center Institutional Review Board.”

May combine two claims.

Heretofore unexplained, or perhaps erroneously explained, phenomena in adaptation experiments result from the fact that most such experiments inadvertently lie somewhere between our adaptation-only protocol and our adaptation-plus-repetition protocol, with the result that three distinct forms of learning - adaptation, use-dependent plasticity, and operant reinforcement - are unintentionally lumped

“Based on our results, we argue that heretofore unexplained, or perhaps erroneously explained, phenomena in adaptation experiments result from the fact that most such experiments inadvertently lie somewhere between our adaptation-only protocol and our adaptation-plus-repetition protocol, with the result that three distinct forms of learning – adaptation, use-dependent plasticity, and operant”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

May combine two claims.

Re-experiencing the same task with the same or even opposite rotational perturbations induces the learner to initially reduce error through pure adaptation but when their movements come within range of the attractor, savings occurs.

“Re-experiencing the same task with the same or even opposite rotational perturbations induces the learner to initially reduce error through pure adaptation but when their movements come within range of the attractor, savings occurs.”

• AssumptionThe authors state this as an assumption: In motor control, the reward structure is unambiguous: success is achieved by the cursor reaching the target.

Probably not a substantial claim: may not pass S3 (load-bearing).

Near misses have a rewarding/motivating nature in gambling, where they increase the desire to play.

“The rewarding/motivating nature of “near misses” has been reported for gambling where they increase the desire to play (Clark et al., 2009; Daw et al., 2006; Kakade and Dayan, 2002).”

• cites Clark et al., 2009 sourceNot checked yet

  Not checked in the free run.

• cites Daw et al., 2006 sourceNot checked yet

  Not checked in the free run.

• cites Kakade and Dayan, 2002Not checked yet

  Not checked in the free run.