Cited by 683Mental Health Research TopicsAttention Deficit Hyperactivity DisorderFunctional Brain Connectivity Studies

The DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.

Understanding Heterogeneity in Clinical Cohorts Using Normative Models: Beyond Case-Control Studies

Open access
Marquand, Andre F., Rezek, Iead, Buitelaar, Jan, Beckmann, Christian F.·2016·Biological Psychiatry

Abstract

BACKGROUND: Despite many successes, the case-control approach is problematic in biomedical science. It introduces an artificial symmetry whereby all clinical groups (e.g., patients and control subjects) are assumed to be well defined, when biologically they are often highly heterogeneous. By definition, it also precludes inference over the validity of the diagnostic labels. In response, the National Institute of Mental Health Research Domain Criteria proposes to map relationships between symptom dimensions and broad behavioral and biological domains, cutting across diagnostic categories. However, to date, Research Domain Criteria have prompted few methods to meaningfully stratify clinical cohorts. METHODS: We introduce normative modeling for parsing heterogeneity in clinical cohorts, while allowing predictions at an individual subject level. This approach aims to map variation within the cohort and is distinct from, and complementary to, existing approaches that address heterogeneity by employing clustering techniques to fractionate cohorts. To demonstrate this approach, we mapped the relationship between trait impulsivity and reward-related brain activity in a large healthy cohort (N = 491). RESULTS: We identify participants who are outliers within this distribution and show that the degree of deviation (outlier magnitude) relates to specific attention-deficit/hyperactivity disorder symptoms (hyperactivity, but not inattention) on the basis of individualized patterns of abnormality. CONCLUSIONS: Normative modeling provides a natural framework to study disorders at the individual participant level without dichotomizing the cohort. Instead, disease can be considered as an extreme of the normal range or as-possibly idiosyncratic-deviation from normal functioning. It also enables inferences over the degree to which behavioral variables, including diagnostic labels, map onto biology.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces normative modeling with Gaussian process regression to parse heterogeneity in clinical cohorts without dichotomizing into case-control groups. In 491 healthy Human Connectome Project participants, it maps delay discounting to reward-related brain activity and shows that deviation magnitude relates to ADHD hyperactivity but not inattention.

Key findings

  • The normative model predicted that participants who discount rewards more strongly also more strongly engage a network of cortical and subcortical regions; this network overlaps with, but is not identical to, the gambling-task network.
  • Most participants fit the normative model well, including many with high hyperactivity (>8); none of these showed brainordinates deviating from the norm (p < .05, false discovery rate corrected) or were outliers under extreme value distributions (all p > .05).
  • Negative deviance correlated with hyperactivity for the top 1% of maximal deviation (r = .91, p = .03, n = 5), top 5% (r = .53, p = .007, n = 25), 10% (r = .31, p = .03, n = 50), 15% (r = .36, p = .002, n = 74), and 20% (r = .22, p = .03, n = 99); positive deviance did not correlate with hyperactivity (all p > .4).
  • Deviance was not associated with inattention (all p > .2); 22 participants had normative probability maps with brain regions deviating from the normative model (p < .05, false discovery rate corrected), and no brain region deviated in more than three participants.

Methods

  • Gaussian process regression was used to build normative models independently for each brain region, predicting reward-related functional MRI activity from delay discounting; normative probability map Z scores and extreme value statistics summarized subject-level abnormality.
  • The sample was 491 healthy Human Connectome Project participants (288 females; mean age 27 years, range 22–36 years) who completed an incentive processing gambling task, the Achenbach Adult Self-Report, and a delay discounting task; ADHD symptom means were 3.03 for inattention, 2.37 for hyperactivity, and 5.40 for total scores.
  • Model estimation used 10-fold cross-validation accommodating family structure; analyses related deviations to inattention and hyperactivity, with additional checks excluding high head motion and regressing out motion-related components.

Limitations

  • The paper states three areas for future work: evaluating stability of abnormality indices over time, handling extremely large data sets because Gaussian process models scale poorly computationally, and improving model accuracy by modeling spatial correlations.
  • The paper notes that head motion is an important source of artifactual variation in functional MRI; it reports excluding high-motion subjects and repeating analyses after motion correction, with conclusions remaining valid.
  • Evident from the text: the demonstration uses a single healthy cohort and relates deviations to self-reported ADHD symptoms rather than a clinical ADHD cohort, so clinical validation and longitudinal stability are not established here.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.biopsych.2015.12.023

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 The classical case-control approach assumes that cases and controls each form a well-defined group (A). This may often be a reasonable assumption, but in practice many other scenarios are possible. The clinical population may be composed of multiple groups, each having distinct pathology (B); disease-related variation may be nested within healthy variation (C); or the clinical group may be diffuse and heterogeneous as a result of misdiagnosis, comorbidities, or an aggregation of different pathologies (D).
Figure 1 The classical case-control approach assumes that cases and controls each form a well-defined group (A). This may often be a reasonable assumption, but in practice many other scenarios are possible. The clinical population may be composed of multiple groups, each having distinct pathology (B); disease-related variation may be nested within healthy variation (C); or the clinical group may be diffuse and heterogeneous as a result of misdiagnosis, comorbidities, or an aggregation of different pathologies (D).CC BY · PubMed Central
Figure 2 Overview of the proposed normative modeling approach showing the steps in the pipeline. (A) Estimate the normative model with Gaussian processes. This provides the ability to predict brain activity for any (observed or unobserved) value of the clinical covariates along with measures of predictive confidence (blue contour lines). The contours of predictive confidence can be interpreted as centiles of predictive confidence for the cohort (blue numerals, right). (B) For each subject, compute a normative probability map that quantifies the deviation from the normative model at each brain region. (C) Generate a summary measure of abnormality for each subject using extreme value statistics, which can be related to clinically relevant variables. (D) The imaging phenotype can be examined more closely, for example, by thresholding the normative probability maps using established techniques. This can provide insight into the brain mechanisms for subjects that do not fit the normative model. See text for full details. EVD, extreme value distribution; FDR, false discovery rate; NPM, normative probability map.
Figure 2 Overview of the proposed normative modeling approach showing the steps in the pipeline. (A) Estimate the normative model with Gaussian processes. This provides the ability to predict brain activity for any (observed or unobserved) value of the clinical covariates along with measures of predictive confidence (blue contour lines). The contours of predictive confidence can be interpreted as centiles of predictive confidence for the cohort (blue numerals, right). (B) For each subject, compute a normative probability map that quantifies the deviation from the normative model at each brain region. (C) Generate a summary measure of abnormality for each subject using extreme value statistics, which can be related to clinically relevant variables. (D) The imaging phenotype can be examined more closely, for example, by thresholding the normative probability maps using established techniques. This can provide insight into the brain mechanisms for subjects that do not fit the normative model. See text for full details. EVD, extreme value distribution; FDR, false discovery rate; NPM, normative probability map.CC BY · PubMed Central
Figure 3 Area under the delay discounting curve (AUC) for the participants included in the normative model for low ($200) and high ($40,000) reward. For both reward levels, lower values are associated with steeper discounting of future reward. Participants discounted small rewards more than large rewards (t491 = −24.97, p < .001), and many participants strongly discounted both large and small rewards, depicted by a skew of the point cloud toward the y axis and an increasing density of points toward the bottom left corner, respectively. The arrow indicates an increase in overall delay discounting along the axis of maximum variance (i.e., principal eigenvector). The numbered circles indicate the positions selected for the spatial representation of the normative model relative to the baseline model labeled “B” (see text for details).
Figure 3 Area under the delay discounting curve (AUC) for the participants included in the normative model for low ($200) and high ($40,000) reward. For both reward levels, lower values are associated with steeper discounting of future reward. Participants discounted small rewards more than large rewards (t491 = −24.97, p < .001), and many participants strongly discounted both large and small rewards, depicted by a skew of the point cloud toward the y axis and an increasing density of points toward the bottom left corner, respectively. The arrow indicates an increase in overall delay discounting along the axis of maximum variance (i.e., principal eigenvector). The numbered circles indicate the positions selected for the spatial representation of the normative model relative to the baseline model labeled “B” (see text for details).CC BY · PubMed Central
Figure 4 Spatial representation of the normative model. These maps show the predictions made by the normative model for the (fictitous) data points described in Figure 3, obtained after retraining the model using all available data. (A) The expected response. This shows increasing engagement of a network of brain regions with increasing overall delay discounting (rows). The numeral indexing in each row corresponds to the points in covariate space described in Figure 3. To assist visualization, these images have been rescaled such that the maximum across all images is equal to one. (B) An example of the expected variation, which was relatively constant for these points. This image has been rescaled such that the maximum variance in the image is equal to 1. (C) The standardized mean squared error for the normative model under cross-validation (averaged across all cross-validation folds). Comparison with (A) shows that the regions that could be accurately predicted (cool colors) correspond to regions that exhibit variation under different degrees of delay discounting.
Figure 4 Spatial representation of the normative model. These maps show the predictions made by the normative model for the (fictitous) data points described in Figure 3, obtained after retraining the model using all available data. (A) The expected response. This shows increasing engagement of a network of brain regions with increasing overall delay discounting (rows). The numeral indexing in each row corresponds to the points in covariate space described in Figure 3. To assist visualization, these images have been rescaled such that the maximum across all images is equal to one. (B) An example of the expected variation, which was relatively constant for these points. This image has been rescaled such that the maximum variance in the image is equal to 1. (C) The standardized mean squared error for the normative model under cross-validation (averaged across all cross-validation folds). Comparison with (A) shows that the regions that could be accurately predicted (cool colors) correspond to regions that exhibit variation under different degrees of delay discounting.CC BY · PubMed Central
Figure 5 The relationship between the overall deviance from the normative model and hyperactivity scores (center), along with the histograms of the component measures (left and bottom). This figure allows us to determine whether subjects that have high clinical symptoms show a good or a poor fit to the normative model. For illustrative purposes, contour lines show the density of points in the figure. Most points fit the normative model well, but some of these subjects also score highly on hyperactivity (blue arrow). Other subjects who score highly on hyperactivity do not fit the normative model (red arrow). Circled subjects are discussed further in the text. NPM, normative probability map.
Figure 5 The relationship between the overall deviance from the normative model and hyperactivity scores (center), along with the histograms of the component measures (left and bottom). This figure allows us to determine whether subjects that have high clinical symptoms show a good or a poor fit to the normative model. For illustrative purposes, contour lines show the density of points in the figure. Most points fit the normative model well, but some of these subjects also score highly on hyperactivity (blue arrow). Other subjects who score highly on hyperactivity do not fit the normative model (red arrow). Circled subjects are discussed further in the text. NPM, normative probability map.CC BY · PubMed Central
Figure 6 (A) An unthresholded t-statistic image of the main task effect, estimated using a classical general linear model (reward-baseline). Warm colors indicate greater activation during reward, and blue colors indicate reduced activation during reward. We have shown an unthresholded map because this sample has very high power (being estimated from nearly 500 subjects). Thus, nearly all brain regions survive conventional statistical thresholding. (B) Normative probability maps that describe the brain regions that deviate from the normative model in the 10 subjects having the most extreme deviations (p < .05, false discovery rate corrected) (also see Figure S3 in Supplement 1). Warm colors indicate greater activity than would be predicted by the normative model, and cool colors indicate reduced activity relative to the normative model. Subjects are ranked by hyperactivity symptom scores with the rank indicated by the numerals (1 = highest hyperactivity). The two most extreme deviations circled in Figure 5 are indicated by red boxes.
Figure 6 (A) An unthresholded t-statistic image of the main task effect, estimated using a classical general linear model (reward-baseline). Warm colors indicate greater activation during reward, and blue colors indicate reduced activation during reward. We have shown an unthresholded map because this sample has very high power (being estimated from nearly 500 subjects). Thus, nearly all brain regions survive conventional statistical thresholding. (B) Normative probability maps that describe the brain regions that deviate from the normative model in the 10 subjects having the most extreme deviations (p < .05, false discovery rate corrected) (also see Figure S3 in Supplement 1). Warm colors indicate greater activity than would be predicted by the normative model, and cool colors indicate reduced activity relative to the normative model. Subjects are ranked by hyperactivity symptom scores with the rank indicated by the numerals (1 = highest hyperactivity). The two most extreme deviations circled in Figure 5 are indicated by red boxes.CC BY · PubMed Central

Verification

Claims the paper makes

36 Own interpretation · 21 Own result · 20 Not checked yet · 11 No basis given · 2 Own method · 1 Partly supported · accuracy not yet measured

Verify all 27 references — free account

Show all 47 claims

  • It may state two separate claims as one. 22
  • Probably not a substantial claim. 28
44 of 44
  • The authors state this as an assumption.

    Understanding healthy variation is a prerequisite to understanding disease variation.

    “Our approach is predicated on the assertions that 1) understanding healthy variation is a prerequisite to understanding disease variation”
  • The authors state this as an assumption.

    Determining where each subject lies within the population range is required because variation associated with most disorders overlaps with normal variation.

    “that this requires 2) the ability to determine where each subject lies within the population range because variation associated with most disorders overlaps with normal variation.”
  • The authors state this as an assumption.

    Disease may, but may not necessarily, occur as an extreme deviation from a normal pattern.

    “In this study, we consider that disease may—but may not necessarily—occur as an extreme deviation from a normal pattern.”
  • The paper relies on this without support.

    The healthy cohort used to construct the normative model is representative of the normal population.

    “We used normative models constructed from spectra of clinically relevant variables (delay discounting) to predict reward-related brain activity in a large, healthy cohort.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

1 retracted

Show all 54 references

Cited by

Sorted by citations

Metadata

Authors
Marquand, A. F., Rezek, I., Buitelaar, J., & Beckmann, C. F.
Affiliations
Radboud University Nijmegen (NL)
Venue
Biological Psychiatry
Publisher
Elsevier BV
Published
2016-10
Volume
80
Issue
7
Pages
552-561
ISSN
0006-3223 (ISSN-L)
OpenAlex
W2229782077
Cited by
683
References
54
Citations by year
2026: 125 · 2025: 131 · 2024: 91 · 2023: 63 · 2022: 62 · 2021: 89 · 2020: 36 · 2019: 32 · 2018: 29 · 2017: 17 · 2016: 8
FWCI
40.80
Type
journal article
Language
en
Open access
Locations
  • Biological Psychiatry (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, cc-by)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Funders
  • Eli Lilly and Company
  • Wellcome Trust (024.001.006, 098369/Z/12/Z, 098369)
  • European Commission (278948, 602805)
  • Servier
  • H. Lundbeck A/S
  • Horizon 2020 Framework Programme (H2020/2014–2020, 643051, 603016, 642996, 024.001.006)
  • Cilag
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Mental Health Research Topics
  • Health Sciences › Medicine › Psychiatry and Mental health › Attention Deficit Hyperactivity Disorder
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
Keywords
normative modeling, clinical heterogeneity, Research Domain Criteria, personalized prediction, impulsivity, ADHD symptoms, hyperactivity, diagnostic labels, outlier detection
MeSH terms
Adult, Attention Deficit Disorder with Hyperactivity, Brain, Data Interpretation, Statistical, Female, Humans, Impulsive Behavior, Magnetic Resonance Imaging, Male, Reward, Models, Statistical, Cluster Analysis, Case-Control Studies, Young Adult, Functional Neuroimaging, Delay Discounting

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review