NIH Toolbox Cognition Battery (CB): Validation of Executive Function Measures in Adults

May combine two claims.

Adults and children over approximately 6 years of age generally slow down to maintain a high level of accuracy, so individual differences in executive function are often manifested as degrees of slowing rather than as errors.

“Adults and children over the age of approximately 6 years generally slow down to maintain a high level of accuracy, so individual differences in EF are often manifested as degrees of slowing rather than as errors (e.g., Davidson, Amso, Cruess-Anderson, & Diamond, 2006).”

• cites Davidson, Amso, Cruess-Anderson, & Diamond, 2006 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both the standard version of the Dimensional Change Card Sort and a more challenging version have shown excellent test-retest reliability in childhood (ICCs = .90-.94).

“Both the standard version of this task and a more challenging version have shown excellent test–retest reliability in childhood (ICCs = .90–.94; Beck, Schaefer, Pang, & Carlson, 2011).”

• cites Beck, Schaefer, Pang, & Carlson, 2011 sourcePartly supported

  The Discussion reports an ICC of .94 for the standard (cool) Integrated DCCS in preschool children. The Results discuss reliability for the more challenging Advanced DCCS, but the passages available from our copy do not establish an ICC of .90 for that version; they describe cool subtasks collectively as having moderate to excellent reliability. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

On tasks like the NIH Toolbox DCCS, older children and adults tend to slow down (increase reaction time) to maintain high accuracy, and this reaction-time slowing provides an index of executive function cost.

“On tasks like the NIH Toolbox DCCS, older children and adults have a tendency to slow down (> RT) to maintain a high level of accuracy, and this RT slowing provides an index of EF “cost” (Davidson et al., 2006).”

• cites Davidson et al., 2006 sourceSupported

May combine two claims.

The effect sizes for race/ethnicity were small to medium, whereas the effect sizes for educational level were medium to high.

“The effect sizes for race/ethnicity were small to medium, whereas the effect sizes for educational level were medium to high.”

May combine two claims.

White participants scored higher on both NIH Toolbox EF measures than Hispanic participants (medium effect size).

“White participants scored higher on both measures than Black participants (small effect size) and Hispanic participants (medium effect size).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the NIH Toolbox Flanker, the final sample was n = 237 after exclusions.

“For the NIH Toolbox Flanker, 27 adults did not complete the task, data from 2 adults were judged to be invalid based on the examiner’s notes, and 2 adults had too few correct trials upon which to base RT scores (i.e., fewer than two correct trials), leaving a final sample for this measure of n = 237.”

Classified as probably not a substantial claim.

A random subset of 89 participants (approximately 33% of the adult sample) was retested after 7 to 21 days (mean interval = 15.5 days, SD = 4.8) to assess test-retest reliability.

“To assess test–retest reliability, 89 participants (approximately 33% of the adult sample) were randomly selected to be retested after 7 to 21 days (mean interval = 15.5 days, SD = 4.8).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Research on cognitive flexibility has generally found decreases in switch costs across childhood and adolescence and subsequent increases in older adults.

“For example, research on cognitive flexibility has used a variety of task switching paradigms and generally found decreases in switch costs (i.e., the extent to which participants slow down on, or in the context of, switch trials) across childhood and adolescence, and subsequent increases in older adults (Cepeda, Kramer, & Gonzalez de Sather, 2001; Crone, Bunge, Van der Molen, & Ridderinkhof, 2006”

• cites Cepeda, Kramer, & Gonzalez de Sather, 2001 sourceNot checked — no full text of the source
• cites Crone, Bunge, Van der Molen, & Ridderinkhof, 2006 sourceNot checked — no full text of the source
• cites Reimers & Maylor, 2005 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Analyses with and without the flagged outliers gave similar results; results with outliers are reported.

“Analyses were conducted both with and without the participants flagged as outliers; results were similar, and results based on the larger sample size (with outliers) are reported here.”

Probably not a substantial claim: may not pass S3 (load-bearing).

After missing or excluded data, the final analytic samples for each NIH Toolbox executive function measure ranged from n = 237 to n = 264.

“As described below, data from some participants were missing or excluded from the final analyses (e.g., for failing to reach criterion during practice trials), leaving final samples that ranged from n = 237 to n = 264 for each measure.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Dimensional Change Card Sort was selected as the measure of cognitive flexibility, also known as task switching or set shifting.

“The Dimensional Change Card Sort (DCCS) was selected as the measure of cognitive flexibility, also known as task switching or set shifting.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the NIH Toolbox Flanker, 4 adults were flagged as outliers based on scores less than 4 points.

“An additional 4 adults were flagged as outliers based on scores that were less than 4 points (see below).”

Classified as probably not a substantial claim. May combine two claims.

Color-Word Interference Inhibition raw scores provide a measure of flexibility and susceptibility to interference, with higher scores indicating better performance.

“Color-Word Interference Inhibition raw scores provide a measure of flexibility and susceptibility to interference; higher scores indicate better performance.”

• cites Delis, Kaplan, & Kramer, 2001Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Most childhood measures of executive function are too easy for elderly adults, and many classic neuropsychological measures of executive function are either too difficult for young children or inappropriate for other reasons.

“Whereas most childhood measures of EF are too easy for elderly adults, many classic neuropsychological measures of EF, such as the Wisconsin Card Sorting Test (Grant & Berg, 1948) or the Color-Word Stroop task (Stroop, 1935), are either too difficult for young children or inappropriate for other reasons (e.g., the Stroop task assumes not only that participants be literate, but also that reading be”

• cites Grant & Berg, 1948 sourceNot checked — no full text of the source
• cites Stroop, 1935 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The NIH Toolbox DCCS two-vector scoring method includes children and adults on the same metric scale, allowing for instrument comparisons across the lifespan.

“This new scoring method includes children and adults on the same metric scale, allowing for instrument comparisons across the lifespan.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Across the lifespan, executive function appears to follow an inverted-U-shaped curve, rising and then falling.

“Developmental research on EF has revealed that when considered across the lifespan, EF appears to follow an inverted-U-shaped curve, rising and then falling (see Jacques & Marcovitch, 2010, for a review).”

• cites Jacques & Marcovitch, 2010 sourceNot checked — no full text of the source

May combine two claims.

The D-KEFS Color-Word Test measures both cognitive flexibility and the ability to inhibit attention and responding.

“The D-KEFS Color-Word Test, which is based on the Stroop Color-Word test (Stroop, 1935), measures both cognitive flexibility and the ability to inhibit attention and responding.”

• cites Delis, Kaplan, & Kramer, 2001Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Stroop, 1935 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Dimensional Change Card Sort is based on Luria's seminal work on rule use and has been used extensively to study the development of executive function in childhood.

“This task, designed by Zelazo and colleagues (e.g., Frye, Zelazo, & Palfai, 1995; Zelazo, 2006), is based on Luria’s seminal work on rule use and has been used extensively to study the development of EF in childhood.”

• cites Frye, Zelazo, & Palfai, 1995 sourceNot checked — no full text of the source
• cites Zelazo, 2006 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

The NIH Toolbox Cognition Battery contains two computer-based instruments assessing executive function: the Dimensional Change Card Sort (a measure of cognitive flexibility) and a flanker task (a measure of inhibitory control and selective attention).

“The NIHTB-CB contains two computer-based instruments assessing executive function: the Dimensional Change Card Sort (a measure of cognitive flexibility) and a flanker task (a measure of inhibitory control and selective attention).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the NIH Toolbox DCCS, 11 adults were flagged as outliers (but not excluded) based on scores less than 4 points.

“An additional 11 adults were flagged as outliers (but not excluded from the final sample) based on scores that were less than 4 points—well below the range of scores for the remainder of the adult sample (see below).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The PPVT-4 provides an index of relatively crystallized knowledge and is often used as a proxy for full scale IQ or general developmental level.

“As a test of receptive vocabulary, the PPVT-4 provides an index of relatively crystallized knowledge, and is often used as a proxy for full scale IQ or general developmental level (Kline, 2001).”

• cites Kline, 2001 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The flyers advertised for healthy volunteers, but no further health screening or exclusions were applied.

“The flyers advertised for healthy volunteers but no further health screening or exclusions were applied.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the NIH Toolbox DCCS Test, the final sample was n = 244 after exclusions.

“For the NIH Toolbox DCCS Test, 20 adults did not complete the task, data from 2 adults were judged to be invalid based on the examiner’s notes (e.g., stopping mid task, etc.), and 2 adults had too few correct trials upon which to base RT scores (i.e., fewer than 2 correct trials), leaving a final sample for this measure of n = 244.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Executive function encompasses top-down neurocognitive processes involved in conscious, goal-directed control of thought, action, and emotion, including cognitive flexibility, inhibitory control, and working memory.

“Executive function (EF), also called cognitive control, is a construct that encompasses the top-down neurocognitive processes involved in the conscious, goal-directed control of thought, action, and emotion—processes such as cognitive flexibility, inhibitory control, and working memory (Miyake et al., 2000).”

• cites Miyake et al., 2000 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In a cross-sectional study of participants ages 7 to 82 years, switch costs decreased from childhood into young adulthood, stayed fairly constant until approximately 60 years of age, and then increased.

“In a cross-sectional study of participants ages 7 to 82 years, for example, Cepeda et al. (2001) found that switch costs (above and beyond perceptual speed, working memory, and non-switch reaction time) decreased from childhood into young adulthood and then stayed fairly constant until approximately 60 years of age, after which they increased.”

• cites Cepeda et al. (2001) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The NIH Toolbox Cognition Battery was designed for use in epidemiologic studies and clinical trials for ages 3 to 85.

“The NIHTB-CB was designed for use in epidemiologic studies and clinical trials for ages 3 to 85.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Lifespan studies from middle childhood into late adulthood have typically revealed both the rise and the fall of executive function.

“Nonetheless, lifespan studies from middle childhood into late adulthood have typically revealed both the rise and the fall of EF (Bialystok & Craik, 2010; Jacques & Marcovitch, 2010; Salthouse & Davis, 2006).”

• cites Bialystok & Craik, 2010 sourceNot checked — no full text of the source
• cites Jacques & Marcovitch, 2010 sourceNot checked — no full text of the source
• cites Salthouse & Davis, 2006Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The absence of measures suitable across a wide range of childhood ages and into adulthood has made it difficult to characterize the lifespan development of executive function in detail.

“It should be noted, however, that the absence of measures suitable across a wide range of childhood ages and into adulthood has made it difficult to characterize the lifespan development of EF in detail.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Proper consents were obtained and approved by the relevant Institutional Review Boards.

“Proper consents were obtained and approved by the relevant Institutional Review Boards.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The NIH Toolbox provides a set of standardized tests that can be used longitudinally across the lifespan, from ages 3 to 85 years.

“The NIH Toolbox provides a set of standardized tests that can be used longitudinally across the lifespan, from ages 3 to 85 years.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Stratified sampling guidelines were used to enhance demographic variability, and the final sample was distributed across gender, education, and race/ethnicity.

“Stratified sampling guidelines were used to enhance demographic variability, and the final sample was indeed distributed across gender (119 males), highest education level (Mean = 13.4 years; range = 4–20; SD = 2.9), and race/ethnicity (148 non-Hispanic White, 75 Black or African American, 38 Hispanic or Latino, and 7 multi-racial).”

Probably not a substantial claim: may not pass S3 (load-bearing).

No participants failed to meet the practice criterion for the NIH Toolbox DCCS Test.

“No participants failed to meet the practice criterion.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No participants failed to meet the practice criterion for the NIH Toolbox Flanker Inhibitory Control and Attention Test.

“No participants failed to meet the practice criterion.”

May combine two claims.

The adult validation sample consisted of 268 self-described healthy adults recruited at four university-based sites using stratified sampling guidelines to target demographic variability for age (20-85 years), gender, education, and ethnicity.

“A total of 268 self-described healthy adults were recruited at four university-based sites, using stratified sampling guidelines to target demographic variability for age (20–85 years), gender, education and ethnicity.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the standard Dimensional Change Card Sort for children, most 3-year-olds perseverate during the post-switch phase, whereas most 5-year-olds switch flexibly.

“Most 3-year-olds perseverate during the post-switch phase, continuing to sort test cards by the first dimension, whereas most 5-year-olds switch flexibly (e.g., Dick, Overton, & Kovacs, 2005; Kirkham, Cruess, & Diamond, 2003; Zelazo, Müller, Frye, & Marcovitch, 2003).”

• cites Dick, Overton, & Kovacs, 2005 sourceNot checked — no full text of the source
• cites Kirkham, Cruess, & Diamond, 2003 sourceNot checked — no full text of the source
• cites Zelazo, Müller, Frye, & Marcovitch, 2003 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Preliminary analyses indicated increased variability of performance during the last few trials of the Flanker test.

“Again, preliminary analyses indicated increased variability of performance during the last few trials of the test, so scoring was based on the first 20 (out of 25) trials in each block (fish/arrows).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Toolbox measures of EF focus on its relatively "cool" cognitive aspects, often associated with lateral prefrontal cortex and elicited by relatively abstract, decontextualized problems.

“Second, the Toolbox measures of EF focus on its relatively “cool” cognitive aspects, often associated with lateral prefrontal cortex and elicited by relatively abstract, decontextualized problems.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This pattern of results is generally consistent with the existing literature, which has revealed similar correlations between socioeconomic status and EF in childhood.

“This pattern of results is generally consistent with the existing literature, which has revealed similar correlations between socioeconomic status and EF in childhood (e.g., Noble, Norman, and Farah, 2004).”

• cites Noble, Norman, and Farah, 2004Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Given the many aspects of neurological and behavioral function assessed by the NIH Toolbox, a complete and comprehensive assessment of validity was not possible.

“First, given the many aspects of neurological and behavioral function assessed by the NIH Toolbox, a complete and comprehensive assessment of validity was not possible.”

Probably not a substantial claim: may not pass S4 (based).

The new NIH Toolbox EF measures are the first standardized measures of EF appropriate for use from early childhood (age 3 years) to old age (age 85).

“The new NIH Toolbox EF measures are the first standardized measures of EF appropriate for use from early childhood (age 3 years) to old age (age 85).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The use of a single measure (D-KEFS Color Word Interference: Inhibition score) as an index of convergent validity for both Toolbox EF measures was less than ideal.

“The use of a single measure (D-KEFS Color Word Interference: Inhibition score) as an index of convergent validity for both Toolbox EF measures was less than ideal.”

May combine two claims.

Overall, results reveal excellent sensitivity to age-related changes during adulthood, excellent test-retest reliability, and adequate to good convergent and discriminant validity.

“Overall, results reveal excellent sensitivity to age-related changes during adulthood, excellent test–retest reliability, and adequate to good convergent and discriminant validity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The current validation study was not designed specifically to evaluate cognitive health in individuals with neurocognitive disabilities.

“Although the current validation study included a diverse range of participants, it was not designed specifically to evaluate cognitive health in individuals with neurocognitive disabilities.”