Cognitive weaknesses or impairments on the NIH toolbox cognition battery in children and adolescents: base rates in a normative sample and proposed methods for classification

Performance on the crystalized composite is relatively resistant to the effects of brain injury.

“Performance on the crystalized composite is relatively resistant to the effects of brain injury (Tulsky et al., 2017)”

• cites Tulsky et al., 2017 sourceSupported

Stratifying base rates using a measure of crystalized intelligence (as defined by the NIHTB-CB with measures of receptive vocabulary and single word reading) may underestimate the abilities of English language learners and youth with certain neurodevelopmental disorders, such as reading disorder (dyslexia) and language disorder.

“stratifying base rates using a measure of crystalized intelligence (as defined by the NIHTB-CB with measures of receptive vocabulary and single word reading) may underestimate the abilities of English language learners (Ortiz, 2019) and youth with certain neurodevelopmental disorders, such as reading disorder (i.e., dyslexia) and language disorder (de Bree et al., 2022; Gray et al., 2019).”

• cites Ortiz, 2019 sourceNot checked — no full text of the source
• cites de Bree et al., 2022 sourcePartly supported

  In “DLD and word reading” (page unavailable), the paper reports that children with developmental language disorder often have smaller vocabularies and lower word-reading outcomes than typically developing peers. Its results also find below-average word-reading outcomes in the DLD groups. It does not test whether NIHTB-CB crystallized-intelligence scores used to stratify base rates underestimate broader abilities, or establish that claim for children with dyslexia. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Gray et al., 2019 sourcePartly supported

  The Discussion (page unavailable) reports that children with dyslexia or developmental language disorder were distributed across working-memory classes and that working-memory profiles were not synonymous with diagnostic categories. This supports the concern that reading or language difficulties do not fully describe these children's abilities. The paper does not evaluate NIH Toolbox crystallized-intelligence scores or show that stratifying base rates with those scores underestimates ability. The English-language-learner portion is attributed to a different citation. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

In a sample of healthy children, the NIHTB-CB crystallized composite score correlated highly with composite scores of language abilities (r = 0.90) and fluid cognitive abilities (r = 0.71).

“In a sample of healthy children, the NIHTB-CB crystallized composite score correlated highly with composite scores of language abilities (r = 0.90) and fluid cognitive abilities (e.g., tests of attention, processing speed, working memory, and executive functions) (r = 0.71), showing a strong correspondence between this score and performances on several cognitive domains (Akshoomoff et al., 2013).”

• cites Akshoomoff et al., 2013 sourcePartly supported

  The Results > Construct Validity section reports a median correlation of r = .71 between the Toolbox Crystallized and gold-standard Fluid composites, supporting that part of the claim. It reports a median of r = .88—not r = .90—for the analogous Crystallized composites, and says the correlations were calculated separately for two child age groups rather than for the full sample. The Method > “Gold Standard” Cognitive Measures section identifies the crystallized comparison as reading and vocabulary measures. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Among healthy children and adolescents in the NIHTB-CB normative sample, the base rate of one or more low fluid scores ≤5th percentile was 15.9% for age-adjusted norms and 16.3% for demographically adjusted norms.

“and nearly a fifth obtained one or more scores ≤ 5th percentile (age-adjusted norms: 15.9%; demographically adjusted norms: 16.3%).”

When using age-corrected norms based on the full sample, there are two options that classify around 16% of the sample as having possible cognitive weakness or impairment.

“when using age-corrected norms based on the full sample, there are two options that classify around 16% of the sample as having possible cognitive weakness or impairment.”

Estimated overall intellectual ability (i.e., the crystalized composite scores) was strongly associated with the frequency of low NIHTB-CB fluid test scores.

“Estimated overall intellectual ability (i.e., the crystalized composite scores) was strongly associated with the frequency of low NIHTB-CB fluid test scores.”

Among healthy children and adolescents in the NIHTB-CB normative sample, the base rate of one or more low fluid scores ≤25th percentile was 62.4% for age-adjusted norms and 65.2% for demographically adjusted norms.

“Roughly two-thirds of children and adolescents obtained one or more scores ≤ 25th percentile (age-adjusted norms: 62.4%; demographically adjusted norms: 65.2%)”

Only 7.1% of youth obtained one or more low scores ≤ 2nd percentile.

“only 7.1% obtained one or more low scores ≤ 2nd percentile.”

It was common for children and adolescents in the standardization sample to obtain at least one low score across the five NIHTB-CB fluid subtests.

“It was common for children and adolescents in the standardization sample to obtain at least one low score across the five NIHTB-CB fluid subtests.”

Among youth with below average crystallized composite (SS ≤89), 76.1% obtained one or more age-adjusted fluid scores ≤25th percentile.

“For example, considering age-adjusted norms, one or more scores ≤ 25th percentile was obtained by 76.1% of youth with a below average crystalized composite (i.e., SS ≤ 89), compared to 51.9% of youth with an above average crystalized composite (i.e., SS ≥ 110).”

Across the NIHTB-CB fluid subtests, nearly two-thirds of children and adolescents obtained one or more scores ≤ 25th percentile.

“Across the NIHTB-CB fluid subtests, nearly two-thirds of children and adolescents obtained one or more scores ≤ 25th percentile”

Among youth with above average crystallized composite, 9.0% (age-adjusted) and 10.2% (demographically adjusted) obtained one or more fluid scores ≤5th percentile.

“For example, only about 1 in 10 youth with above average crystalized composites obtained any scores ≤ 5th percentile (age-adjusted norms: 9.0%; demographically adjusted norms: 10.2%), whereas nearly 1 in 3 youth with below average crystalized composites obtained at least one score this low (age norms: 29.2%; demographically adjusted norms: 24.8%).”

The number of low scores obtained is related to the cutoff used to define a low score.

“the number of low scores obtained is related to the cutoff used to define a low score.”

Nearly a fifth of children and adolescents obtained one or more low scores ≤ 5th percentile across the NIHTB-CB fluid subtests.

“nearly a fifth obtained one or more low scores ≤ 5th percentile”

When low scores were defined as falling ≤ 25th percentile, 62.4% of youth obtained at least one low age-adjusted NIHTB-CB fluid score.

“when low scores were defined as falling ≤ 25th percentile, 62.4% of youth obtained at least one low age-adjusted NIHTB-CB fluid score”

For youth with average crystallized ability (SS=100-109/T=50-56), the base rate of having either three or more scores ≤25th percentile or two or more scores ≤9th percentile is 11.5% for age-adjusted norms and 10.6% for demographically adjusted norms.

“Another potential algorithm to identify a cognitive weakness or impairment for youth with the same crystalized ability is whether an examinee obtains three or more scores ≤ 25th percentile or two or more scores ≤ 9th percentile. The base rate for having either of these patterns of low scores is 11.5% for age-adjusted norms and 10.6% for demographically adjusted norms.”

Among healthy children and adolescents in the NIHTB-CB normative sample, the base rate of one or more low fluid scores ≤16th percentile was 45.1% for age-adjusted norms and 47.1% for demographically adjusted norms.

“about half obtained one or more scores ≤ 16th percentile (age-adjusted norms: 45.1%; demographically adjusted norms: 47.1%)”

Among youth with above average crystallized composite (T ≥57), 56.6% obtained one or more demographically adjusted fluid scores ≤25th percentile.

“Similarly, considering demographically adjusted norms, one or more scores ≤ 25th percentile were obtained by 75.6% of youth with a below average crystalized composite (i.e., T ≤ 43) compared to 56.6% of youth with an above average crystalized composite (i.e., T ≥ 57).”

Among youth with above average crystallized composite (SS ≥110), 51.9% obtained one or more age-adjusted fluid scores ≤25th percentile.

“For example, considering age-adjusted norms, one or more scores ≤ 25th percentile was obtained by 76.1% of youth with a below average crystalized composite (i.e., SS ≤ 89), compared to 51.9% of youth with an above average crystalized composite (i.e., SS ≥ 110).”

The likelihood of a child or adolescent obtaining at least one score at or below the 9th percentile when administered all five fluid subtests is 29.1% (nearly one third of the normative sample using age-adjusted norms).

“the likelihood of a child or adolescent obtaining at least one score at or below the 9th percentile when administered all five fluid subtests is 29.1% (i.e., nearly one third of the normative sample using age-adjusted norms).”

Among youth with below average crystallized composite (T ≤43), 75.6% obtained one or more demographically adjusted fluid scores ≤25th percentile.

“Similarly, considering demographically adjusted norms, one or more scores ≤ 25th percentile were obtained by 75.6% of youth with a below average crystalized composite (i.e., T ≤ 43) compared to 56.6% of youth with an above average crystalized composite (i.e., T ≥ 57).”

Half of children and adolescents obtained one or more scores ≤ 16th percentile across the NIHTB-CB fluid subtests.

“half obtained one or more scores ≤ 16th percentile”

Per Algorithm #7 in Table 3, the likelihood of an examinee obtaining any of these three performance patterns is 11.1%.

“per Algorithm #7 in Table 3, the likelihood of an examinee obtaining any of these three performance patterns is 11.1%.”

Among youth with below average crystallized composite, 29.2% (age-adjusted) and 24.8% (demographically adjusted) obtained one or more fluid scores ≤5th percentile.

“For example, only about 1 in 10 youth with above average crystalized composites obtained any scores ≤ 5th percentile (age-adjusted norms: 9.0%; demographically adjusted norms: 10.2%), whereas nearly 1 in 3 youth with below average crystalized composites obtained at least one score this low (age norms: 29.2%; demographically adjusted norms: 24.8%).”

Among healthy children and adolescents in the NIHTB-CB normative sample, the base rate of one or more low fluid scores ≤9th percentile was 29.1% for age-adjusted norms and 24.9% for demographically adjusted norms.

“between a third and a fourth obtained one or more scores ≤ 9th percentile (age-adjusted norms: 29.1%; demographically adjusted norms: 24.9%)”

Between a third and a fourth of children and adolescents obtained one or more scores ≤ 9th percentile across the NIHTB-CB fluid subtests.

“between a third and a fourth obtained one or more scores ≤ 9th percentile”

For Algorithm #10, 10.6% of children and adolescents from the NIHTB-CB normative sample had one of the two performance patterns.

“For Algorithm #10, 10.6% of children and adolescents from the NIHTB-CB normative sample had one of the two performance patterns.”

Applying those criteria to the NIHTB-CB five fluid subtests, 45.1% (using age-adjusted norms) to 47.1% (using demographically adjusted norms) of children and adolescents from the normative sample would meet the psychometric criteria for mild neurocognitive disorder.

“Applying those criteria to the NIHTB-CB five fluid subtests, 45.1% (using age-adjusted norms) to 47.1% (using demographically adjusted norms) of children and adolescents from the normative sample would meet the psychometric criteria for mild neurocognitive disorder”

Among the children and adolescents from the NIHTB-CB standardization sample included in this study, only 9.7% met at least one of the criteria for Algorithm #1 and thus could be classified showing cognitive weakness or impairment on the fluid tests.

“Among the children and adolescents from the NIHTB-CB standardization sample included in this study, only 9.7% met at least one of these criteria and thus could be classified showing cognitive weakness or impairment on the fluid tests using Algorithm #1 (see Table 3).”

Youth with low crystallized composite scores are more likely to obtain low fluid test scores.

“Youth with low crystallized composite scores are more likely to obtain low fluid test scores (see Table 1 and Figure 1).”

For Algorithm #2, the base rate is 13.3% of children and adolescents in this sample.

“For the second option (Algorithm #2), there is only one criterion: three or more scores ≤ 25th percentile, with a base rate of 13.3% of children and adolescents in this sample.”

Obtaining three or more scores within a conventional low average classification range (i.e., 10th-25th percentile) is Uncommon for children and adolescents of high average crystalized ability, occurring in fewer than 10% of the normative sample at this ability level.

“obtaining three or more scores within a conventional low average classification range (i.e., 10th–25th percentile) is Uncommon for children and adolescents of high average crystalized ability, occurring in fewer than 10% of the normative sample at this ability level.”

In the full sample, the base rate of obtaining one or more low scores ≤ 5th percentile on the five fluid NIHTB-CB tests is about 16% (age-adjusted norms: 15.9%; demographically adjusted norms: 16.3%).

“In the full sample, the base rate of obtaining one or more low scores ≤ 5th percentile on the five fluid NIHTB-CB tests is about 16% (age-adjusted norms: 15.9%; demographically adjusted norms: 16.3%).”

For Algorithm #11, 12.6% of the NIHTB-CB normative sample had this performance pattern.

“For Algorithm #11, 12.6% of the NIHTB-CB normative sample had this performance pattern.”

For youth with average crystallized ability (SS=100-109/T=50-56), the base rate of having two or more fluid scores ≤16th percentile is 14.2% for age-adjusted norms and 12.6% for demographically adjusted norms.

“For example, a potential algorithm to identify a developmental weakness or acquired cognitive impairment for youth with average crystalized ability (SS = 100–109/T = 50–56) is having two or more scores ≤ 16th percentile. The base rate for this pattern of low scores is 14.2% for age-adjusted norms and 12.6% for demographically adjusted norms.”

When administered a cognitive test battery, children and adolescents in the general population frequently obtain one or more low scores.

“When administered a cognitive test battery, children and adolescents in the general population frequently obtain one or more low scores (Brooks and Iverson, 2010, 2012; Brooks et al., 2013a; Cook et al., 2019).”

• cites Brooks and Iverson, 2010, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Brooks et al., 2013aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Cook et al., 2019 sourceNot checked — no full text of the source

Obtaining one or more low scores, even at or below the 5th percentile, is surprisingly common among healthy individuals who have no known clinical conditions.

“When considered in the context of a test battery, obtaining one or more low scores, even at or below the 5th percentile (which might reasonably be interpreted as occurring relatively infrequently in isolation), is surprisingly common among healthy individuals who have no known clinical conditions.”

The reported base rates apply to the interpretation of all five fluid tests among participants administered all seven NIHTB-CB subtests (i.e., five fluid and two crystalized tests).

“The reported base rates apply to the interpretation of all five fluid tests among participants administered all seven NIHTB-CB subtests (i.e., five fluid and two crystalized tests).”

The crystallized composite score can be used as a proxy for overall intellectual ability.

“The crystallized composite score, which consists of a receptive vocabulary test and a single word reading test, was used as a proxy for overall intellectual ability consistent with prior research on the NIHTB-CB (Holdnack et al., 2017b).”

• cites Holdnack et al., 2017bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Base rates of low NIHTB-CB scores among children and adolescents have not been reported.

“but base rates have not been reported among children and adolescents.”

The frequency of obtaining one or more low scores across a battery (multivariate base rates) differs markedly from the expected frequency for any given test considered in isolation.

“the frequency of obtaining one or more low scores across a battery (so called multivariate base rates) differ markedly from the expected frequency for any given test considered in isolation.”

• AssumptionRests on an assumption: Low scores are more frequent when more tests are administered and interpreted.

If fewer subtests are administered in practice, these base rates and algorithms will not be applicable.

“If fewer subtests are administered in practice, these base rates and algorithms will not be applicable.”

When evaluating children with acquired injuries to the brain, their obtained crystalized composite scores might not closely correspond with their premorbid functioning.

“when evaluating children with acquired injuries to the brain it is important to appreciate that their obtained crystalized composite scores might not closely correspond with their premorbid functioning.”

DSM-5 psychometric diagnostic criteria for mild neurocognitive disorder require a standardized cutoff of approximately 1 SD below the mean on neuropsychological testing for defining cognitive impairment.

“A pattern that occurred in roughly 16% of the normative sample was selected based on DSM-5 psychometric diagnostic criteria for mild neurocognitive disorder, which requires a standardized cutoff of approximately 1 SD below the mean on neuropsychological testing for defining cognitive impairment (American Psychiatric Association, 2013).”

• cites American Psychiatric Association, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

These results align with many prior studies demonstrating that the likelihood of a child or adolescent obtaining a low score is not equivalent to the percentile rank for that score.

“These results align with many prior studies demonstrating that the likelihood of a child or adolescent obtaining a low score is not equivalent to the percentile rank for that score.”

• AssumptionRests on an assumption: Low scores are more frequent when more tests are administered and interpreted.

Given that the data were from the normative sample, the percentage of youth obtaining a low score on each test would be relatively close to the percentile rank (i.e., approximately 16% would obtain a score at or below the 16th percentile).

“Given that we analyzed data from the normative sample, the percentage of youth obtaining a low score on each test would be relatively close to the percentile rank (i.e., approximately 16% would obtain a score at or below the 16th percentile).”

Because youth from the normative sample who had preexisting conditions were excluded, the percentage of included participants scoring below the percentile cutoff for each test was somewhat lower than the percentile cutoff would suggest.

“given that we excluded youth from the normative sample who had preexisting conditions, this would mean that the percentage of included participants scoring below the percentile cutoff for each test was somewhat lower than the percentile cutoff would suggest”

If a clinician or researcher were to apply each of these performance patterns in succession, rather than selecting and applying one criterion a priori, the base rate and, therefore the probability of false-positive findings, would be considerably higher than any of the individual performance patterns applied in isolation.

“If a clinician or researcher were to apply each of these performance patterns in succession, rather than selecting and applying one criterion a priori, the base rate and, therefore the probability of false-positive findings, would be considerably higher than any of the individual performance patterns applied in isolation.”