CoChat
Sign Up
Cited by 67Traumatic Brain Injury ResearchPsychological Testing and AssessmentCognitive Functions and Memory

Using the NIH Toolbox Cognition Battery (NIHTB-CB) in individuals with traumatic brain injury.

Open access
Tulsky, David S., Carlozzi, Noelle E., Holdnack, James, Heaton, Robert K., Wong, Alex, Goldsmith, Arielle, Heinemann, Allen W.·2017·Rehabilitation Psychology
DOI 10.1037/rep0000174PMC PMC6462276
Open this paper in CoChat — free account

Abstract

PURPOSE/OBJECTIVE: The NIH Toolbox for the Assessment of Neurological Behavior and Function Cognition Battery (NIHTB-CB) is a common data element for use in individuals with traumatic brain injury (TBI). This study evaluates its sensitivity and specificity in distinguishing individuals with complicated mild, moderate, or severe TBI, and provides support for the construct validity of the NIHTB-CB in individuals with TBI. RESEARCH METHOD: One hundred eighty-two individuals with TBI (n = 83 complicated mild/moderate; n = 99 severe) completed the NIHTB-CB and neuropsychological criterion measures. Complete data were obtained on 158 participants. A control sample of 158 individuals without known neurological impairment was extracted from the NIHTB-CB normative sample. Multivariate analyses of variance determined the sensitivity of the NIHTB-CB measures to TBI and injury severity (complicated mild/moderate TBI, severe TBI, and controls) on the demographically corrected NIHTB-CB composite scores and seven subtests. A descriptive analysis of the sensitivity of each subtest was conducted. Finally, correlations between NIHTB-CB measures and criterion tests assessed convergent and discriminant validity. RESULTS: Multivariate analyses indicated that there was a main effect for group (complicated mild/moderate vs. severe vs. controls) for fluid scores in the NIHTB-CB as opposed to only marginally significant results for the verbal scores. Moderate to strong relationships were found between the NIHTB-CB measures and their corresponding neuropsychological measures (convergent validity), whereas much smaller correlations were found between measures of different cognitive domains (discriminant validity). CONCLUSIONS: Findings provide evidence of construct validity and the clinical utility of the NIHTB-CB in individuals with TBI. (PsycINFO Database Record

AI snapshot

Written from the paper’s full text.

TL;DR

The NIH Toolbox Cognition Battery (NIHTB-CB) showed sensitivity to traumatic brain injury (TBI) and injury severity, especially on fluid cognition measures, while crystallized/language measures were largely spared. It also showed evidence of convergent and discriminant validity against established neuropsychological tests in adults with complicated mild/moderate or severe TBI.

Key findings

  • For the two demographically corrected composites, MANOVA showed a significant group effect, F(4, 626) = 17.04, p < .0001, Pillai’s Trace = .197, partial eta² = .098. The complicated mild/moderate and severe TBI groups performed worse than controls on the Fluid Cognition Composite, and the severe TBI group performed worse than the complicated mild/moderate group; the Crystallized Cognition Composite was marginally significant overall, F(2, 313) = 4.89, p < .01, but post hoc comparisons were not statistically significant.
  • For the seven subtests, MANOVA showed a significant group effect, F(14, 614) = 5.57, p < .0001, Pillai’s Trace = .225, partial eta² = .113. Significant group differences occurred on all fluid subtests; the severe TBI group performed worse than controls on all comparisons, and the complicated mild/moderate group performed worse than controls on Picture Sequence Memory, List Sorting, Flanker, and DCCS. The TBI groups differed on Picture Sequence Memory, Pattern Comparison, and Flanker.
  • Sensitivity was highest for Picture Sequence Memory at a T-score cutoff ≤ 40: .36 for complicated mild/moderate vs. controls and .52 for severe TBI vs. controls, with a 13% false positive rate; the Fluid Composite showed similar values. False positive rates were 11–16%, and language-based measures were minimally sensitive to TBI.
  • Convergent validity correlations ranged from 0.42 to 0.83 (median = 0.68). The largest correlations were .83 for Oral Reading Recognition with WRAT-4 Reading and .80 for Picture Vocabulary with PPVT; the lowest correlations were .13 and .10 between Pattern Comparison and DCCS with language measures.

Methods

  • Multisite validation study: 182 individuals with medically documented TBI (n = 83 complicated mild/moderate; n = 99 severe) recruited from three rehabilitation centers, with complete data on 158 TBI participants; eligibility required age ≥ 18, hospital admission within 24 hours of TBI, English comprehension at a 5th-grade level, and at least one year since injury. A demographically matched control sample of 158 adults was extracted from the NIHTB-CB standardization sample (N = 1,038 English-speaking adults), matched on age, education, ethnicity, and gender.
  • TBI severity was classified by lowest emergency department GCS within 24 hours: ≤ 8 severe, 9–12 moderate, and 13–15 complicated mild if neuroimaging showed acute brain abnormality. Participants completed the NIHTB-CB (7 subtests and 3 composites) and criterion neuropsychological measures.
  • Analyses included MANOVAs for group differences on demographically corrected composite and subtest scores, descriptive sensitivity/specificity at T ≤ 40, and Pearson correlations using Fisher’s z′ averaging to assess convergent and discriminant validity; effect sizes were reported as Cohen’s d.

Limitations

  • Stated: the sample may not be representative of the TBI population because participants had to complete an 8-hour battery over two days; prior participation in other clinical or research evaluations was not excluded; the original dual-screen monitor format was used, so generalization to the newer iPad version is unclear; and the control sample did not complete the established neuropsychological measures, preventing estimation of inflated or attenuated correlations.
  • Stated: missing data were not random and were associated with education level; tests requiring rapid responses were most likely to be missing in the TBI group. The NIHTB-CB also lacks methods for estimating performance invalidity, and its brevity may limit detailed cognitive assessment.
  • Evident from the text: the study is cross-sectional and does not establish longitudinal sensitivity or recovery trajectories; the control group came from a normative sample rather than the same study procedures, which limits direct comparison of NIHTB-CB and gold-standard measures.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1037/rep0000174

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

9 of 50 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    The cognitive sequelae of TBI are well documented.

    “The cognitive sequelae of TBI are well documented.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Donders & Levitt, 2012 source
    • cites Tulsky et al., 2014 source
    • cites Carlozzi, Kirsch, et al., 2015 source
  • Premisecitation

    The degree of cognitive change and level of impairment after TBI vary as a function of the severity of injury.

    “The degree of cognitive change and level of impairment after TBI vary as a function of the severity of injury (Carlozzi, Tulsky, Kail, & Beaumont, 2013; Rohling, Meyers, & Millis, 2003).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Carlozzi, Tulsky, Kail, & Beaumont, 2013 source

      The “Cognitive Processing, and Biological Substrates” section says children with closed head injury have slower processing speed than typically developing peers. It does not say that cognitive change or impairment varies with injury severity, and its finding concerns processing speed in children rather than cognitive impairment after TBI generally. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Rohling, Meyers, & Millis, 2003 source
  • Premisecitation

    Initial validation work supported a multidimensional structure of cognition with episodic memory, vocabulary, reading, working memory, and a processing speed/executive functioning abilities.

    “The initial validation work also supported a multidimensional structure of cognition with episodic memory, vocabulary, reading, working memory, and a processing speed/executive functioning”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Mungas et al., 2014 source
    • cites Mungas et al., 2013 source

      The Abstract and Discussion identify the five listed dimensions in children aged 8–15, including a combined executive function/processing speed factor. The claim leaves out the age qualification: for children aged 3–6, the paper found only vocabulary, reading, and a combined fluid abilities dimension. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingcitation

    Individuals in current sample had higher performance scores on Pattern Comparison compared to expected with WAIS-III and WAIS-IV Processing Speed Indices.

    “However, while the Pattern Comparison test is sensitive to the effects of TBI, individuals in the current sample had higher performance scores compared to what would have been expected with the WAIS-III (Donders et al., 2001) and the WAIS-IV Processing Speed Indices (Carlozzi, Beaumont, et al., 2015; Carlozzi, Kirsch, et al., 2015; Donders & Strong, 2015).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Donders et al., 2001
    • cites Carlozzi, Beaumont, et al., 2015 source

      The Results (Table 3; page unavailable) provide age-group norms for Pattern Comparison, but do not compare scores in the citing paper’s sample with WAIS-III or WAIS-IV Processing Speed Indices. Searches for those indices and TBI-related expected performance found no passage in our copy making that comparison. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Carlozzi, Kirsch, et al., 2015 source
  • Premisecitation

    Demographically-corrected norms demonstrate better accuracy with less demographic bias on cognitive measures than age-only adjusted scores in neurological populations.

    “Demographically-corrected norms demonstrate better accuracy with less demographic bias (Casaletto et al., 2015; R. K. Heaton, Ryan, L., & Grant, I., 2009; Taylor & Heaton, 2001) on cognitive measures than age-only adjusted scores in neurological populations.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Casaletto et al., 2015 source

      The Discussion (page unavailable) describes fully demographically adjusted scores as the best indicator of possible impairment, and the Abstract reports that these scores removed significant demographic associations. But the study used neurologically healthy participants; it did not demonstrate better accuracy than age-only scores in neurological populations. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites R. K. Heaton, Ryan, L., & Grant, I., 2009

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Taylor & Heaton, 2001 source
  • Classified as probably not a substantial claim. 30
  • May combine two claims. 44
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Initial validation work provided initial evidence of convergent and discriminant validity among NIHTB-CB subtests.

    “provided initial evidence of convergent and discriminant validity among NIHTB-CB subtests (Mungas et al., 2014; Mungas et al., 2013).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Mungas et al., 2014 source
    • cites Mungas et al., 2013 source

  • May combine two claims.

    Findingown data

    Picture Sequence Memory (7.2%) and Vocabulary (6.3%) were the most frequently missing data and Oral Reading the least frequent (1.9%).

    “Picture Sequence Memory (7.2%) and Vocabulary (6.3%) were the most frequently missing data and Oral Reading the least frequent (1.9%.).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Post hoc analysis did not reveal statistically significant differences between specific groups on crystallized subtests.

    “Regardless, post hoc analysis did not reveal statistically significant differences between specific groups.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There was no difference in missing values by age, race, or gender.

    “there was no difference in missing values by age, race, or gender”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In its current configuration, NIHTB-CB does not have methods for estimating performance invalidity.

    “In its current configuration, the NIHTB-CB does not have methods for estimating performance invalidity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Pattern Comparison test significantly differentiates TBI from control groups and was sensitive to severity of injury.

    “The Pattern Comparison test is a choice reaction time test which significantly differentiates TBI from control groups and was sensitive to severity of injury.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Individual test generally has relatively high correlation with aggregate score and lower correlations with tests of different constructs.

    “As described above, the individual test generally has a relatively high correlation with the aggregate score and lower correlations with tests of different constructs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The highest sensitivity is observed for Picture Sequence Memory at a cut-off of T-score ≤ 40, .36 for complicated mild/moderate vs. controls and .52 for severe TBI vs. controls at a 13% false positive rate.

    “The highest sensitivity is observed for Picture Sequence Memory at a cut-off of T-score ≤ 40, .36 for complicated mild/moderate vs. controls and .52 for severe TBI vs. controls at a 13% false positive rate.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Over 50% of severe and 30% of complicated mild/moderate TBI groups had scores in impaired range on Picture Sequence Memory.

    “Over 50% of the severe and 30% of the complicated mild/moderate TBI groups had scores in the impaired range.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both Flanker and Dimensional Change Card Sort tests differentiated TBI from control groups; only Flanker was sensitive to severity effects.

    “Both tests differentiated the TBI from control groups; however, only the Flanker test was sensitive to severity effects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Missing data occurred at a rate of 20% for individuals with 12 or fewer years of education and 9% for those with at least some college education.

    “Missing data occurred at a rate of 20% for individuals with 12 or fewer years of education and 9% for those with at least some college education.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We obtained moderate rates of missing data that were not random but associated with education level.

    “We obtained moderate rates of missing data that were not random but associated with education level.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    Battery requires only 30 minutes to complete.

    “The battery requires only 30 minutes to complete.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In control group, 11% had low scores, compared to 27% and 32% of complicated mild/moderate and severe TBI, respectively.

    “In the control group, 11% had low scores, compared to 27% and 32% of the complicated mild/moderate and severe TBI, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Pattern Comparison and Picture Sequence Memory are correlated moderately with measures across most domains; the highest correlations were with the gold standard measures that measure the same constructs of processing speed and episodic memory, respectively.

    “Finally, the Pattern Comparison and Picture Sequence Memory are correlated moderately with measures across most domains; however, the highest correlations were with the gold standard measures that measure the same constructs of processing speed and episodic memory, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Base rate tables show 16% of controls will have low scores on Oral Reading Test; 25% of individuals with severe injuries have low scores at 1 SD cutoff.

    “Base rate tables show that 16% of controls will have low scores on the Oral Reading Test; by comparison, 25% of individuals with severe injuries have low scores at the 1 SD cutoff.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There were group differences between the two TBI groups for Picture Sequence Memory Test, Pattern Comparison, and Flanker Inhibitory Control and Attention Test, with the severe group performing worst.

    “There were also group differences between the two TBI groups for Picture Sequence Memory Test, Pattern Comparison, and Flanker Inhibitory Control and Attention Test, with the severe group performing worst”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Types of tests that were missing differed by TBI group: tests requiring rapid responses were most likely to be missing.

    “The types of tests that were missing differed by TBI group:tests requiring rapid responses were most likely to be missing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The complicated mild/moderate group performed worse than controls on Picture Sequence Memory Test, List Sorting, and Flanker Inhibitory Control and Attention Test and DCCS.

    “the complicated mild/moderate group performed worse than controls on Picture Sequence Memory Test, List Sorting, and Flanker Inhibitory Control and Attention Test and DCCS”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Significant correlations in moderate range obtained between working memory tasks (0.56) and between executive functioning tests (0.46 and 0.42 in magnitude).

    “Significant correlations in the moderate range were obtained between the working memory tasks (0.56) and between the executive functioning tests (0.46 and 0.42 in magnitude).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    High correlations between language tests (e.g., 0.80 and 0.83) consistent with previous findings.

    “As expected, we observe high correlations between language tests (e.g., 0.80 and 0.83) which is consistent with previous findings showing correlations between language measures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The complicated mild/moderate group had moderate effect sizes vs. controls on the Fluid Composite, Picture Sequence Memory, and Flanker Inhibitory Control and Attention Test and did not have any large effects.

    “The complicated mild/moderate group had moderate effect sizes vs. controls on these variables and did not have any large effects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There were significant group differences on age and time since injury.

    “There were significant group differences on age, F (2, 313) = 6.39, p< .01 and time since injury, F (1, 156) = 6.46, p <.05.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Groups did not differ on gender, race, ethnicity, or education.

    “Groups did not differ on gender, χ² (2, N = 318) = 0.124, p > .05, race, χ² (12, N =318) = 1.99, p > .05, ethnicity, χ² (2, N = 317) = 2.28, p > .05, or education, F (2, 313) = 1.64, p > .05.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Rates of missing test data were significantly higher in the TBI sample for Dimensional Change Card Sort, Flanker Inhibitory Control and Attention Test, and Pattern Comparison.

    “Rates of missing test data were significantly higher in the TBI sample for Dimensional Change Card Sort (6.6 % vs. 0.0%; χ² (1) = 10.36, p < .01); Flanker Inhibitory Control and Attention Test (6.0 % vs. 0.0%; χ² (1) = 9.32, p < .01), and Pattern Comparison (7.1 % vs. 0.6%; χ² (1) = 8.93, p < .01).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The presence of missing data was not specific to TBI (relative to the control sample) nor more impaired individuals within the TBI groups.

    “The presence of missing data was not specific to TBI (relative to the control sample) nor more impaired individuals within the TBI groups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both Flanker and Dimensional Change Card Sort tests were sensitive to TBI vs. control, second only to Picture Sequence Memory.

    “Both tests were sensitive to TBI vs. control, second only to Picture Sequence Memory.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    On language and crystallized abilities, mild/moderate TBI group was not significantly different from control group while severe TBI group only showed a trend for lower reading scores.

    “Finally, on tests of language and crystalized abilities, the mild/moderate TBI group was not significantly different from the control group while the severe TBI group only showed a trend for lower reading scores (Carlozzi, Kirsch, et al., 2015; Iverson et al., 2013).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Carlozzi, Kirsch, et al., 2015 source
    • cites Iverson et al., 2013 source

  • May combine two claims.

    Findingown data

    Picture Vocabulary did not show statistically significant differences between severe TBI and control groups, but level of performance relative to Oral Reading is nearly identical (48.3 vs. 48.4).

    “While Picture Vocabulary did not show statististically significant differences between the severe TBI and control groups, the level of performance relative to Oral Readingis nearly identical (48.3 vs. 48.4).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    If no GCS was available, TBI cases were classified based on the detailed description of their injury and confirmed by a neuropsychologist independent of performance on the NIHTB-CB or neurocognitive tests.

    “If no GCS was available, TBI cases were classified based on the detailed description of their injury and confirmed by a neuropsychologist independent of performance on the NIHTB-CB or neurocognitive tests (see Figure 1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Each examiner practiced the administration procedures by administering them to a minimum of 5 cases.

    “Each examiner practiced the administration procedures by administering them to a minimum of 5 cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The NIHTB standardization sample was weighted to be demographically representative of the 2010 US Census data.

    “The NIHTB standardization sample was weighted to be demographically representative of the 2010 US Census data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Improvements in cognition are typically seen within the first two years post-injury.

    “In general, improvements in cognition are typically seen within the first two years post-injury”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites S. Dikmen, Machamer, Temkin, & McLean, 1990 source
    • cites S. Dikmen, Reitan, & Temkin, 1983 source
    • cites Schretlen & Shapiro, 2003 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In many people with TBI, there is not a full return to baseline functioning.

    “although, in many people, there is not a full return to baseline functioning”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites S. Dikmen, Machamer, Temkin, & McLean, 1990 source
    • cites S. Dikmen, Reitan, & Temkin, 1983 source
    • cites Schretlen & Shapiro, 2003 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The NIH Interagency Common Data Elements Project included the NIHTB as a recommended CDE for TBI studies.

    “the NIH Interagency Common Data Elements Project included it as a recommended CDE for TBI studies (E.A. Wilde et al., 2010).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites E.A. Wilde et al., 2010 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All protocols were double-scored to minimize scoring errors.

    “All protocols were double-scored to minimize scoring errors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    The construct validity and clinical utility of the NIHTB for assessing neurocognitive impairments in clinical disorders or individuals with disabilities has not been established.

    “Since the NIHTB was developed for use in the general population, its construct validity and clinical utility for assessing neurocognitive impairments in clinical disorders or individuals with disabilities has not been established.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The NIHTB is a set of integrated and co-normed tools to assess cognitive, emotional, motor, and sensory health across the lifespan.

    “The NIHTB is a set of integrated and co-normed tools to assess cognitive, emotional, motor, and sensory health across the lifespan.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Gershon et al., 2010 source
    • cites Gershon, Wagster, et al., 2013 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    After the initial training, examiners were observed and certified by one of two authors of this paper in a live testing session.

    “After the initial training, they were observed and certified by one of two authors of this paper (NC and DT) in a live testing session.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Language abilities are generally spared or less affected in individuals with TBI.

    “Language abilities, in contrast, are generally spared or less affected (Carlozzi, Kirsch, et al., 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Carlozzi, Kirsch, et al., 2015 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Examiners were recertified annually to ensure that tests continued to be administered in a standardized manner.

    “We recertified examiners annually to ensure that tests continued to be administered in a standardized manner.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    An examiner who was trained and certified according to a well-established protocol administered all cognitive tests.

    “An examiner who was trained and certified according to a well-established protocol (described below) administered all cognitive tests.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductcitation

    Complicated mild TBI was combined with moderate TBI because long-term outcomes for individuals with complicated mild more closely resemble individuals with moderate TBI than they do mild TBI.

    “We combined complicated mild TBI with moderate TBI (see below for definitions of severity of injury) on the basis of previous work suggesting that long-term outcomes for individuals with complicated mild more closely resemble individuals with moderate TBI than they do mild TBI (Kashluba, Hanks, Casey, & Millis, 2008a).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kashluba, Hanks, Casey, & Millis, 2008a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    In cases where scoring agreement did not exceed 95%, the examiner was retrained and the scoring for 10 new cases was reviewed.

    “In cases where scoring agreement did not exceed 95%, the examiner was retrained and the scoring for 10 new cases was reviewed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    One of the authors reviewed 10 test protocols from each examiner, rescored the protocol, and provided feedback about deviations from standard procedures.

    “One of the authors (NC) reviewed 10 test protocols from each examiner, rescored the protocol, and provided feedback about deviations from standard procedures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The control sample was demographically matched to the total TBI group on age, education level, ethnicity, and gender.

    “Our subset of these participants was matched to the total TBI group on key demographic variables (i.e., age, education level, ethnicity, and gender).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Some individuals with relatively severe injuries may have average or better test scores, whereas individuals with less severe injuries may have below expected cognitive performance.

    “Some individuals with relatively severe injuries may have average or better test scores, whereas individuals with less severe injuries may have below expected cognitive performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    One of primary advantages of using NIHTB-CB is its relatively short administration time.

    “One of the primary advantages of using the NIHTB-CB is its relatively short administration time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    With only single measures of key cognitive functions, it may be insufficient to identify deficits related to specific cognitive domains or by skills within a domain.

    “Furthemore, with only single measures of key cognitive functions, it may be insufficient to identify deficits related to specific cognitive domains (e.g. executive dysfunction compared to the more general “fluid” category) or by skills within a domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This strength may be a limitation when researchers or clinicians require information about detailed aspects of cognition.

    “This strength may be a limitation when researchers or clinicians require information about detailed aspects of cognition, such as identifying retrieval vs. encoding deficits or differentiating slow vs. errors in performance.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Brevity allows evaluation of range of abilities without fatiguing examinees.

    “Brevity allows evaluation of a range of abilities without fatiguing examinees.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Findings provide initial evidence of construct validity and the clinical utility of the NIHTB-CB in individuals with TBI.

    “Findings provide initial evidence of construct validity and the clinical utility of the NIHTB-CB in individuals with TBI.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Performance on the NIHTB-CB is an important indicator of the measures’ construct validity and clinical utility in individuals with TBI.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    It is unclear if results will completely generalize to newer iPad version.

    “It is unclear if the results will completely generalize to the newer iPad version.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study included rigorous training of psychometricians for proper administration and data integrity.

    “The study also included rigorous training of psychometricians for proper administration and data integrity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    We did not exclude volunteers if they had participated in other clinical or research evaluations which conceivably may have included neuropsychological tests; exposure to tests may affect their performance.

    “We did not exclude volunteers if they had participated in other clinical or research evaluations which conceivably may have included neuropsychological tests; exposure to tests may affect their performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Presence of TBI was confirmed with medical records, making this a well worked up sample.

    “While the presence of a TBI in this study was confirmed with medical records, making this a well worked up sample”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Processing speed impairment is a well-documented sequela of TBI and often yields largest effect sizes along with memory.

    “Processing speed impariment represents another domain of cognitive functioning that is a well-documented sequela of TBI and, along with memory functioning, often yields the largest effect sizes (Iverson et al., 2013).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Iverson et al., 2013 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Of critical importance in clinical and research settings evaluating individuals with TBI is need to identify invalid responses.

    “Of critical importance in clinical and research settings evaluting individuals with TBI is the need to identify invalid responses (Heilbronner et al., 2009; Larrabee, 2012).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Heilbronner et al., 2009 source
    • cites Larrabee, 2012 source

  • May combine two claims.

    Interpretationown data

    This study provides evidence that NIHTB-CB tests are sensitive to effects of TBI and severity of injury.

    “This study provides evidence that the NIHTB-CB tests are sensitive to the effects of TBI and severity of injury.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The higher correlations of Flanker and DCCS with processing speed may reflect the importance of timed performance in the NIHTB-CB measures of executive functioning.

    “which may reflect the importance of timed performance in the NIHTB-CB measures of executive functioning”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Unlike Picture Sequence Memory, both Flanker and Dimensional Change Card Sort tests require fast performance and processing speed.

    “Unlike Picture Sequence Memory, both Flanker Inhibitory Control and Attention Test and Dimensional Change Card Sort tests require a fast performance and processing speed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Cognitive tests are comprised of related constructs and many subtests will have moderate to high correlations with each other.

    “Cognitive tests are comprised of related constructs (Holdnack & Weiss, 2006; Tulsky & Price, 2003) and many of the subtests will have moderate to high correlations with each other.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Holdnack & Weiss, 2006 source
    • cites Tulsky & Price, 2003 source

  • May combine two claims.

    Interpretationown data

    We observed group differences and patterns of correlations consistent with previous studies and these results provide evidence of convergent-discriminant validity for NIHTB-CB language measures and good convergent validity for List Sorting, Pattern Comparison and Picture Memory tests.

    “We observed group differences and patterns of correlations that are consistent with previous studies and these results provide evidence of convergent-discriminant validity for the NIHTB-CB language measures and good convergent validity for the List Sorting, Pattern Comparison and Picture Memory tests.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Considering processing speed is a frequent deficit in TBI, it is not surprising that executive function tests based on reaction time also have high correlation with measures of processing speed.

    “Considering that processing speed is a frequent deficit in individuals with TBI, it is not surprising that executive function tests based on reaction time also would have a high correlation with measures of processing speed.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Processing speed is a frequent deficit in individuals with TBI.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Correlations between executive functioning tasks in other studies also have tended to be in moderate range, around magnitude of coefficients obtained in this study.

    “Correlations between executive functioning tasks in other studies also have tended to be in the moderate range, around the magnitude of the coefficients obtained in this study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Legacy measures were not administered in normative study, so no direct comparison is possible.

    “Unfortunately, the legacy measures were not administered in the normative study, so no direct comparison is possible.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    NIHTB-CB Executive Functioning tests appear to be tapping the construct for which they were developed, but they also reflect processing speed.

    “While the NIHTB-CB Executive Functioning tests appear to be tapping the construct for which they were developed, they also reflect processing speed.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    Control sample was derived from NIHTB-CB standardization study sample and these individuals did not complete most concurrent validity measures.

    “Rather, the control sample was derived from the NIHTB-CB standardization study sample and these individuals did not complete most of the concurrent validity measures.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Level of performance is consistent with that reported in studies of established measures, though NIHTB-CB Pattern Comparison Processing Speed test may slightly underestimate impairment.

    “The level of performance is consistent with that reported in studies of established measures, though the NIHTB-CB Pattern Comparison Processing Speed test may slightly underestimate impairment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Multi-center data collection insured greater representation of individuals from different rehabilitation centers.

    “Multi-center data collection insured greater representation of individuals from different rehabilitation centers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study had a large clinical sample of well-defined TBI severities.

    “The strengths of the study include a large clinical sample of well-defined TBI severities.”
    Quote in the paperNot a substantial claim
71 of 71

  • The paper relies on this without support.

    The clinical utility of the NIHTB for use with individuals with TBI has yet to be established.

    “However, the clinical utility of the NIHTB for use with individuals with TBI has yet to be established.”

  • The paper relies on this without support.

    Validation research for the NIHTB in individuals with TBI is limited.

    “Although validation research for the NIHTB in individuals with TBI is limited”

  • The paper relies on this without support.

    Performance on the NIHTB-CB is an important indicator of the measures’ construct validity and clinical utility in individuals with TBI.

    “Therefore, the performance of individuals with TBI on the NIHTB is an important indicator of the measures’ construct validity and clinical utility.”

  • The paper relies on this without support.

    TBI causes cognitive impairment (the impact of TBI on cognition is well documented).

    “as the impact of TBI on cognition is well documented”

  • The paper relies on this without support.

    Memory complaints and impairment are hallmark symptoms of TBI.

    “Memory complaints and impairment are hallmark symptoms of TBI”

  • The paper relies on this without support.

    A significant subset of the TBI population recovers quite well.

    “it is known that a significant subset of the TBI population recovers quite well”

  • The paper relies on this without support.

    Processing speed is a frequent deficit in individuals with TBI.

    “processing speed is a frequent deficit in individuals with TBI”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentχ²(2) = 0.73reported p > .05 · recomputed p = .694

    “χ2 (2, N =364) = 0.73, p > .05”

  • Consistentχ²(3) = 9.74reported p < .05 · recomputed p = .021

    “χ2 (3, N =364) = 9.74, p < .05”

  • Consistentχ²(1) = 10.36reported p < .01 · recomputed p = .001

    “χ2 (1) = 10.36, p < .01”

  • Consistentχ²(1) = 9.32reported p < .01 · recomputed p = .002

    “χ2 (1) = 9.32, p < .01”

  • Consistentχ²(1) = 8.93reported p < .01 · recomputed p = .003

    “χ2 (1) = 8.93, p < .01”

  • Consistentχ²(2) = 0.124reported p > .05 · recomputed p = .940

    “χ2 (2, N = 318) = 0.124, p > .05”

  • Consistentχ²(12) = 1.99reported p > .05 · recomputed p = .999

    “χ2 (12, N =318) = 1.99, p > .05”

  • Consistentχ²(2) = 2.28reported p > .05 · recomputed p = .320

    “χ2 (2, N = 317) = 2.28, p > .05”

  • ConsistentF(2, 313) = 1.64reported p > .05 · recomputed p = .196

    “F (2, 313) = 1.64, p > .05”

  • ConsistentF(2, 313) = 6.39reported p < .01 · recomputed p = .002

    “F (2, 313) = 6.39, p < .01”

  • ConsistentF(1, 156) = 6.46reported p < .05 · recomputed p = .012

    “F (1, 156) = 6.46, p < .05”

  • ConsistentF(4, 626) = 17.04reported p < .0001 · recomputed p < .001

    “F (4, 626) = 17.04, p < .0001”

  • ConsistentF(2, 313) = 4.89reported p < .01 · recomputed p = .008

    “F (2,313) =4.89, p < .01”

  • ConsistentF(14, 614) = 5.57reported p < .0001 · recomputed p < .001

    “F (14, 614) = 5.57, p < .0001”

  • ConsistentF(2, 313) = 4.66reported p < .05 · recomputed p = .010

    “F (2,313) =4.66, p < .05”

  • ConsistentF(2, 313) = 3.45reported p < .05 · recomputed p = .033

    “F (2,313) =3.45, p < .05”

16 of 16

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Testing described as both one visit and two days

    Minor

    The methods say participants were required to come to the center on one occasion, but the limitations say testing spanned two days.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Cohen J, Cohen P, West SG, & Aiken LS. (2003). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.). Mahweh, NJ: Lawrence Earlbaum Associates, Publishers.
  2. Dikmen SS, Bauer PJ, Weintraub S, Mungas D, Slotkin J, Beaumont JL, … Heaton RK (2014b). Measuring episodic memory across the lifespan: NIH Toolbox Picture Sequence Memory Test. J Int Neuropsychol Soc, 20(6), 611– 619. doi: 10.1017/S1355617714000460 24960230 PMC4254833 · DOI
  3. Heaton RK, Ryan L, & Grant I (2009). Demographic influences and use of demographically corrected norms in neuropsychological assessment. In I. G. K. A. (Eds.) (Ed.), Neuropsychological assessment of neuropsychiatric and neuromedical disorders (pp. 127– 155). New York, NY: Oxford University Press.
  4. Kashluba S, Hanks RA, Casey JE, & Millis SR (2008b). Neuropsychologic and functional outcome after complicated mild traumatic brain injury. Arch Phys Med Rehabil, 89(5), 904– 911. 18452740
  5. Tulsky DS, Carlozzi NE, Chevalier N, Espy KA, Beaumont JL, & Mungas D (2013). V. NIH Toolbox Cognition Battery (CB): measuring working memory. Monogr Soc Res Child Dev, 78(4), 70– 87. doi: 10.1111/mono.12035 23952203 PMC4427028 · DOI · Paper page
  6. Wilde EA, Whiteneck GG, Bogner J, Bushnik T, Cifu DX, Dikmen S, … von Steinbuechel N (2010). Recommendations for the use of common outcome measures in traumatic brain injury research. Arch Phys Med Rehabil, 91(11), 1650– 1660 e1617. doi: S0003–9993(10)00656–8 [pii] 10.1016/j.apmr.2010.06.033 21044708 · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Tulsky, D. S., Carlozzi, N. E., Holdnack, J., Heaton, R. K., Wong, A., Goldsmith, A., & Heinemann, A. W.
Affiliations
  • University of Delaware (US)
  • University of Michigan (US)
  • University of California San Diego (US)
  • Washington University in St. Louis (US)
  • Shirley Ryan AbilityLab (US)
ORCID
Venue
Rehabilitation Psychology
Publisher
American Psychological Association (APA)
Published
2017-11
Volume
62
Issue
4
Pages
413-424
ISSN
  • 0090-5550 (ISSN-L)
  • 1939-1544
DOI
10.1037/rep0000174
PMC
PMC6462276
OpenAlex
W2778503564
Semantic Scholar
442a84e390ebe57332a8bf8f5e77db5c61681188
Cited by
67
References
59
Citations by year
2026: 3 · 2025: 13 · 2024: 6 · 2023: 6 · 2022: 6 · 2021: 7 · 2020: 10 · 2019: 4 · 2018: 4 · 2017: 8
FWCI
5.60
Type
journal article
Language
en
Open access
Licence
Locations
  • Rehabilitation Psychology (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institute on Disability, Independent Living, and Research Rehabilitation (H133B090024)
  • Blueprint for Neuroscience Research (HHS-N-2602006-00007-C)
Topics
  • Health Sciences › Medicine › Epidemiology › Traumatic Brain Injury Research
  • Social Sciences › Psychology › Applied Psychology › Psychological Testing and Assessment
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
Keywords
NIH Toolbox Cognition Battery, traumatic brain injury, construct validity, sensitivity and specificity, injury severity, mild traumatic brain injury, severe traumatic brain injury, fluid intelligence, verbal ability, neuropsychological assessment, cognitive assessment
MeSH terms
Brain Injuries, Traumatic, Adult, Cognition, Cognition Disorders, Female, Humans, Male, National Institutes of Health (U.S.), Neuropsychological Tests, Psychometrics, Sensitivity and Specificity, Severity of Illness Index, United States, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review