Divergent Research Methods Limit Understanding of Working Memory Training

May combine two claims.

Across 57 experiments from 51 studies, more than 120 different transfer tasks were used, and transfer tasks were rarely consistent from one study to the next.

“to the transfer tasks that are rarely consistent from one study to the next with over 120 different transfer tasks used across the 57 experiments reviewed in 51 studies”

May combine two claims.

In the reviewed sample, 30 of 57 experiments used single N-back training and 27 used dual N-back training, with varying transfer outcomes.

“In the current sample, 30 out of the 57 experiments adopted single N-back training (13 reporting transfer within WM, 11 reporting transfer beyond WM, and 6 reporting no transfer1) and 27 experiments employed dual N-back training (8 reporting transfer within WM, 9 reporting transfer beyond WM, and 10 reporting no transfer).”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

In the reviewed sample, training task modalities varied widely: 26 used auditory stimuli, 13 used visual non-spatial stimuli, and 18 used spatial stimuli, with varying transfer outcomes.

“In our sample, we find that training task modalities vary widely, with 26 using auditory stimuli (7 reporting transfer within WM, 11 reporting transfer beyond WM, and 8 reporting no transfer), 13 using visual stimuli (non-spatial) (5 reporting transfer within WM, 6 reporting transfer beyond WM, and 2 reporting no transfer), and 18 using spatial stimuli (9 reporting transfer within WM, 3 reporting”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

Only 8 experiments used the same training method, while 49 experiments had unique training conditions.

“Interestingly, only 8 experiments relied on the same training method, whereas 49 experiments had unique training conditions (Fig. 1).”

May combine two claims.

About half of the meta-analyses in the field report evidence of transfer, while the other half do not.

“The field has reached a point in which there is a battle of meta-analyses lingering with roughly half of them finding evidence of transfer while the others do not (see Table 1 for variety of individual studies upon which these meta-analyses are based).”

May combine two claims.

For most of the 122 tasks used, no reliability measures are reported, and it is unclear whether standard or custom forms were used, making reliability information difficult to find.

“However, for most of the 122 of tasks used, no reliability measures are reported, and it is unclear whether standard forms or custom forms of the tasks are employed, making it difficult to find information on the reliability in the extant literature.”

May combine two claims.

Out of 57 reviewed experiments, 25 employed some type of feedback and 32 either did not provide feedback or did not explicitly report it, with varying transfer outcomes.

“Out of the 57 experiments reviewed, 25 experiments employed some type of feedback (11 reporting transfer to untrained WM tasks, 6 reporting transfer beyond WM, and 8 reporting no transfer) while 32 experiments either did not provide feedback or did not explicitly report the use of feedback (16 reporting transfer within WM, 9 reporting transfer beyond WM, and 7 reporting no transfer).”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

In the reviewed sample, 52 experiments included at least one control group, with 22 having only an active control, 17 having only a passive control, and 13 having both.

“In our sample, 52 experiments included at least one control group: 22 experiments included only an active control group, 17 experiments included only a passive control group, and 13 experiments included both.”

May combine two claims.

N-back training studies, while seemingly similar, employ a wide variety of training features and assess transfer effects with a large and diverse selection of outcome measures.

“Here we show that N-back training studies, while seemingly similar, employ a wide variety of training features, and in addition, they assess transfer effects with a large and diverse selection of outcome measures.”

May combine two claims.

The review included 51 studies, and because 5 studies had more than one N-back training group considered separately, the total number of experiments was 57.

“In total, our research resulted in 51 studies (excluding the number of overlapping studies) (Fig. 2). Of these 51 studies, 5 studies included more than one N-back training group, which we considered separately, giving rise to a total of 57 experiments.”

May combine two claims.

In the reviewed experiments, 12 used a threshold below 90% to advance, while 34 used a 90% correct threshold, with varying transfer outcomes.

“Of 46 experiments, 12 adopted a threshold lower than 90% to achieve the next level (7 reporting transfer within WM, 1 reporting transfer beyond WM, and 4 reporting no transfer), while 34 adopted a threshold of 90% correct (16 reporting transfer within WM, 10 reporting transfer beyond WM, and 8 reporting no transfer).”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

Across 57 experiments, 122 different transfer tasks were used, and the number of outcome measures per study ranged from 1 to 20.

“Across 57 experiments, 122 different transfer tasks were employed (see Table 2), which speaks to the issue of variability in transfer tasks. The number of outcome measures per study ranged from 1 to as many as 20.”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.

May combine two claims.

In the reviewed sample, 47 of 57 experiments used training equal to or longer than 10 sessions, and 10 used fewer than 10 sessions, with varying transfer outcomes.

“In our sample, of the 57 experiments that measured both transfer to WM and beyond WM, 47 used training equal or longer than 10 sessions (29 reporting transfer within WM, 12 reporting transfer beyond WM, and 6 reporting no transfer), and 10 experiments used fewer than 10 sessions (5 reporting transfer within WM, 2 reporting transfer beyond WM, and 3 reporting no transfer).”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

Transfer beyond working memory, particularly to fluid intelligence, was assessed with 27 different tasks, and 48 of 57 experiments reported assessing fluid intelligence.

“Transfer beyond WM, in particular to fluid intelligence, was assessed with 27 different tasks. Forty-eight out of fifty-seven experiments reported assessing fluid intelligence.”

May combine two claims.

In the reviewed experiments, 46 reported long inter-stimulus intervals (1800-2500 ms), 9 used short intervals (500-1800 ms), and 2 did not report interval information.

“In the papers that we reviewed, we screened 57 experiments across single and dual N-back training and found 46 experiments that reported long ISIs (between 1800 and 2500 ms; 18 reporting transfer within WM, 14 reporting transfer beyond WM and 14reporting no transfer), 9 that used short ISIs (between 500 and 1800 ms; 8 reporting transfer within WM, and 1 reporting transfer beyond WM), while 2”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

The pattern may suggest that dual N-back training is more likely to yield transfer within and beyond working memory, while single N-back training is more likely to show transfer within working memory, but not all studies assessed both types of transfer.

“While this may suggest that dual N-back training is more likely to yield transfer within and beyond WM, as compared with single N-back, which seems more likely to show transfer within WM, it should be noted that not all studies assessed both types of transfer.”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

Probably not a substantial claim: may not pass S3 (load-bearing).

Training on a specific working memory task can improve performance on untrained working memory tasks.

“WM training on a given task can transfer to improvements in untrained working memory tasks (Blacker et al. 2017; Lilienthal et al. 2013; Chein and Morrison 2010; Borella et al. 2010)”

• cites Blacker et al. 2017 sourceNot checked yet

  Not checked in the free run.

• cites Lilienthal et al. 2013 sourceNot checked yet

  Not checked in the free run.

• cites Chein and Morrison 2010 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The absence of an explanation for this variability undermines confidence in working memory training research and makes it harder to judge the effectiveness of training.

“The lack of explanation regarding this variability not only casts a shadow on WM training research but also poses a significant hurdle when evaluating the effectiveness of WM training.”

May combine two claims.

Dual N-back tasks produce longer reaction times and more errors than single N-back tasks.

“For example, Jaeggi et al. (2003) showed that single and dual N-back tasks differ at the behavioral level with longer reaction times and more errors on dual N-back tasks compared with single N-back.”

• cites Jaeggi et al. (2003) sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Single and dual N-back tasks do not differ at the neural level; prefrontal activation increases with higher load regardless of task type.

“On the other hand, no differentiation between single and dual N-back tasks was observed at the neural level: prefrontal activation increased with higher load irrespective of task type.”

• cites Jaeggi et al. (2003) sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

The N-back task is a common working memory measure; it is an updating task requiring storage, maintenance, and manipulation, and it predicts individual differences in higher cognitive functions.

“One of the most common measures of WM is the N-back task, an updating task that requires multiple processes (storage, maintenance, and manipulation of information) and is predictive of inter-individual differences in higher cognitive functions (Jaeggi et al. 2010a, b).”

• cites Jaeggi et al. 2010a, bNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Working memory training can also transfer to tasks in other cognitive domains, such as fluid intelligence.

“as well as tasks pertaining to other cognitive domains such as fluid intelligence (Jaeggi et al. 2008; Heinzel et al. 2017; Chein and Morrison 2010; Borella et al. 2010)”

• cites Jaeggi et al. 2008 sourceNot checked yet

  Not checked in the free run.

• cites Heinzel et al. 2017 sourceNot checked yet

  Not checked in the free run.

• cites Chein and Morrison 2010 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a long-running debate about whether training can improve basic cognitive functions.

“A longstanding debate has regarded the extent to which training can improve our basic cognitive functions (Katz et al. 2018).”

• cites Katz et al. 2018 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Even though the single versus dual N-back comparison is the most powered available, differences between study methodologies largely prevent strong meta-analytic conclusions.

“Even though the single vs dual N-back dichotomy is the most powered of available comparisons, the differences between study methodologies, as described below, largely preclude strong meta-analytic conclusions.”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.
• AssumptionRests on an assumption: The transfer outcomes reported by the original studies are valid enough to be counted as evidence of transfer or no transfer.

May combine two claims.

Although reports on N-back training are increasing, the mechanisms of transfer and the factors that might affect them remain unclear.

“Although reports on N-back training are steadily increasing, the mechanisms of transfer and the factors that might impact them are still unclear.”

• cites Au et al. 2015Not checked yet

  Not checked in the free run.

• cites Soveri et al. 2017a, bNot checked yet

  Not checked in the free run.

• cites Melby-Lervåg and Hulme 2013 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Performance on N-back tasks correlates only weakly with performance on complex span tasks, so updating and span tasks should be considered separately.

“Performance on N-back tasks only correlates weakly with performance on complex span tasks (Redick and Lindsey 2013) therefore it makes sense to consider updating and span tasks separately.”

• cites Redick and Lindsey 2013 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Although matrix reasoning was the most common fluid intelligence test, using only one type of test is not enough to estimate fluid intelligence at the latent level.

“While matrix reasoning was the most common type of test used to assess fluid intelligence, which allows for a certain level of comparison across experiments, using just one type of test is not sufficient to estimate fluid intelligence at the latent level.”

May combine two claims.

Categorizing working memory tasks only by type and domain does not capture the full range of cognitive demands and may hide improvements in a subgroup of tasks.

“While some distinctions have been made in terms of task type (untrained N-back vs. WM tasks in Soveri et al. 2017a, b) and task domain (verbal vs. visuospatial WM in Melby-Lervåg and Hulme 2013; Melby-Lervåg et al. 2016; Schwaighofer et al. 2015), such categorization does not capture the full range of cognitive demands imposed by different WM tasks and may even mask improvements in a subgroup of”

• cites Soveri et al. 2017a, bNot checked yet

  Not checked in the free run.

• cites Melby-Lervåg and Hulme 2013 sourceNot checked yet

  Not checked in the free run.

• cites Melby-Lervåg et al. 2016 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Evidence on whether adaptivity improves working memory training gains is mixed: one study found greater gains for adaptive training, while others found no effect.

“Moreover, Holmes et al. (2009) showed that WM training gains were significantly greater for an adaptive training group compared with a non-adaptive training group, although others have failed to observe any effects of adaptivity on learning outcome (von Bastian and Eschen 2016).”

• cites Holmes et al. (2009) sourceNot checked yet

  Not checked in the free run.

• cites von Bastian and Eschen 2016 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The working memory training literature is mixed about what control conditions are used and how much control type alters transfer magnitude.

“However, literature on WM training is mixed both in regard to what control conditions are employed, some using active controls and others passive controls, and also the extent to which the control type seems to alter the magnitude of observed transfer (Au et al. 2015).”

• cites Au et al. 2015Not checked yet

  Not checked in the free run.

May combine two claims.

The diversity of transfer tasks across studies raises serious questions about whether the same underlying cognitive outcomes are measured and limits interpretation of the literature.

“Overall, this diversity of transfer tasks measured across studies raises serious issues of the extent to which the same underlying cognitive outcomes are assessed across studies and thus, limits the interpretation of the extant literature.”

• AssumptionRests on an assumption: The reviewed set of 57 experiments is representative of published N-back working memory training studies.

Classified as probably not a substantial claim. May combine two claims.

The type of control group plays a significant role in whether transfer is observed, and its impact relates to similarity between N-back training and control interventions, participant engagement, motivation, and expectations.

“It has also been argued that the type of control group plays a significant role in whether transfer is observed. The impact of control groups is related to the degree of similarity between the N-back training and the control interventions, and/or to the differential participant engagement and motivation, and/or participant expectations (Green et al. 2019).”

• cites Green et al. 2019Not checked yet

  Not checked in the free run.

May combine two claims.

Several factors lead to the diversity of methods in the field, and this diversity limits progress.

“We suggest that there are several factors that are leading to this diversity of methods, which we argue limit progress in the field.”

Probably not a substantial claim: may not pass S4 (based).

The conceptual understanding of working memory or fluid intelligence as domain-general processes leads researchers to overlook the importance of domain and task specificity.

“First, there is the conceptual understanding of WM or fluid intelligence as domain-general processes. This view presumably leads researchers to overlook the importance of domain and task specificity, assuming that it does not matter how a specific exercise or test on WM is given (type, modality, etc.), as all approaches would impact the same cognitive process.”

May combine two claims.

There is evidence that longer training produces more learning, reflected in more pronounced changes in brain regions involved in working memory, and that training duration affects results, though the appropriate amount is not well established.

“There is evidence that longer training leads to more learning in terms of more pronounced changes in brain regions involved in WM function (Dahlin et al. 2008; Lövdén et al. 2010).”

• cites Dahlin et al. 2008 sourceNot checked yet

  Not checked in the free run.

• cites Lövdén et al. 2010Not checked yet

  Not checked in the free run.

May combine two claims.

Large test batteries can cause participant fatigue and reduced engagement, and can create multiple comparison problems.

“Using large test batteries can give rise to participant fatigue and decreased participant engagement (Ackerman and Kanfer 2009), and it can also lead to issues with multiple comparison.”

• cites Ackerman and Kanfer 2009Not checked yet

  Not checked in the free run.

May combine two claims.

The diversity of transfer tests and batteries across studies is challenging because outcome measures vary in similarity to the trained task and their reliability and validity are often unclear.

“Overall, the diversity of transfer tests and batteries used across studies poses a challenge as these outcome measures vary in their degree of similarity with the trained task, and furthermore, their reliability and their validity in measuring the factor of interest are often unclear.”

Classified as probably not a substantial claim.

Because certain training task features had small sample sizes and the literature shows extensive variability, the authors can only speculate whether these factors are meaningful mediators and moderators.

“Given the small sample size of certain training task features and the extensive variability of methods in the literature, we can only speculate whether these factors are meaningful mediators and moderators.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Feedback is important for learning, especially in complex cognitive tasks and for monitoring goal progress, and it can facilitate learning.

“Feedback plays an important role in the process of learning, particularly in complex cognitive tasks and in monitoring goal progress (West et al. 2001).”

• cites West et al. 2001Not checked yet

  Not checked in the free run.

May combine two claims.

Most N-back training studies use adaptive training by adjusting difficulty based on individual performance, and adapting difficulty is engaging for participants.

“Most N-back training studies utilize adaptive training by adjusting the level of task difficulty based on individual performance, and it has been shown that adapting the difficulty level of the task is engaging for the participant (Jaeggi et al. 2014).”

• cites Jaeggi et al. 2014 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Active control conditions differ in their cognitive and perceptual demands, in their similarity to the experimental condition, and likely in the expectations they induce about training-related improvement, which makes it difficult to compare results across studies.

“These active control conditions differ in their cognitive and perceptual demands and similarity to the experimental condition, as well as most likely in the induced expectations about performance improvement due to training, again making it difficult to compare results across studies.”

May combine two claims.

Inter-stimulus intervals can affect the time available to process each stimulus and to use strategies such as rehearsal or grouping and comparison, and these strategies can change performance and likely affect learning outcomes.

“Inter-stimulus intervals (ISI) can have an important impact on the time available to process each stimulus and to engage in strategies such as rehearsal or grouping and comparison. The use of these strategies can modify performance levels, give rise to very different experiences during training, and thus likely impact learning outcomes (Laine et al. 2018).”

• cites Laine et al. 2018 sourceNot checked yet

  Not checked in the free run.