The DOI matches its Crossref record; 52 of the 55 references it cites were found in a registry or an index, and 3 are not checked yet.
The meaning of significant mean group differences for biomarker discovery
Open accessAbstract
Over the past decade, biomarker discovery has become a key goal in psychiatry to aid in the more reliable diagnosis and prognosis of heterogeneous psychiatric conditions and the development of tailored therapies. Nevertheless, the prevailing statistical approach is still the mean group comparison between “cases” and “controls,” which tends to ignore within-group variability. In this educational article, we used empirical data simulations to investigate how effect size, sample size, and the shape of distributions impact the interpretation of mean group differences for biomarker discovery. We then applied these statistical criteria to evaluate biomarker discovery in one area of psychiatric research—autism research. Across the most influential areas of autism research, effect size estimates ranged from small ( d = 0.21, anatomical structure) to medium ( d = 0.36 electrophysiology, d = 0.5, eye-tracking) to large ( d = 1.1 theory of mind). We show that in normal distributions, this translates to approximately 45% to 63% of cases performing within 1 standard deviation (SD) of the typical range, i.e., they do not have a deficit/atypicality in a statistical sense. For a measure to have diagnostic utility as defined by 80% sensitivity and 80% specificity, Cohen’s d of 1.66 is required, with still 40% of cases falling within 1 SD. However, in both normal and nonnormal distributions, 1 (skewness) or 2 (platykurtic, bimodal) biologically plausible subgroups may exist despite small or even nonsignificant mean group differences. This conclusion drastically contrasts the way mean group differences are frequently reported. Over 95% of studies omitted the “on average” when summarising their findings in their abstracts (“autistic people have deficits in X”), which can be misleading as it implies that the group-level difference applies to all individuals in that group. We outline practical approaches and steps for researchers to explore mean group comparisons for the discovery of stratification biomarkers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Simulations of the degree to which 2 groups overlap at different effect sizes. (a) Percentage of autistic individuals (red) falling within 1 SD and 2 SDs of the control (blue) distribution at effect size of d = 0.2, 0.5, 1, and 2.7. 0 = mean, σ = SD. Simulations based on 10,000 random draws assuming the same SD and absolute mean difference in the population. The red shaded area indicates the % of cases above 2 SDs. (b) Although sample size does not bias the effect size estimates themselves, it does substantially affect their precision, which is reflected in the width of the CI. The precision of effect size estimates with sample sizes of N = 20 and N = 100. Purple shading denotes CIs around 1 SD of the mean and red shading CIs around 2 SDs of the mean. For example, for a small effect size at Cohen’s d of 0.2, with N = 100 participants per group, between 60% and 75% of autistic people would fall within 1 SD of the control mean. With smaller samples of N = 20 per group, ranges grow to 40%–85% within 1 SD and to 75%–100% within 2 SDs. At Cohen’s d of 0.5, with N = 100 versus N = 20, 55%–71% versus 45%–80% of people with ASD would fall within 1 SD and between 89%–97% versus 85%–100% within 2 SDs of the TD mean, etc. Hence, with small sample sizes, the range of possible results is so wide that it is difficult to make accurate inferences of the frequency or severity of cases who have abnormalities on that measure from single studies. As recently noted, studies with small sample sizes (low power), paired with publication bias and file drawer effects as well as high sample variability (true heterogeneity within a condition), can lead a whole field to overestimate the magnitude of the true population effect [49,50]. ASD, autism spectrum disorder; CI, confidence interval; SD, standard deviation.](/paper-figures/c5503c04d4d74c59b2fcb557199a8eaa/the-meaning-of-significant-mean-group-differences-for-biomarker-discovery-figure-1.jpg)

![Fig 3 Average effect sizes of meta-analyses per modality. The distributions of the original data included in the meta-analyses were often not reported. This is exemplified in S1 Table where we checked information on the data distributions of the 49 original papers included in a review of emotion recognition [20]. However, in the majority of papers, parametric statistics were employed, which may be taken as indicating normal distribution. EF, executive function; ET, eye-tracking; fMRI, functional MRI; MMN, mismatch negativity; PRS, polygenic risk score; ROI, region of interest; sMRI, structural MRI; SNP, single-nucleotide polymorphism; ToM, theory of mind.](/paper-figures/a5de641bcc1b495e8b198e1df006e17c/the-meaning-of-significant-mean-group-differences-for-biomarker-discovery-figure-3.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, this metric may not be interpretable if derived out of non-normal distributions, as is often encountered in neuroimaging 8 .”
From Candidate diagnostic biomarkers for neurodevelopmental disorders in children and adolescents: a systematic review (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Although there are no established benchmarks for these metrics, quantitative measures that allow diagnostic accuracy with at least 80% sensitivity and 80% specificity are often considered as clinically useful 8”
From Candidate diagnostic biomarkers for neurodevelopmental disorders in children and adolescents: a systematic review (2023) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Whenever effect sizes were reported, these were generally in the low or moderate range, and certainly not in the range of an effect size of d=1.66 that would be needed to lead to a sensitivity and specificity of 80% 8 .”
From Candidate diagnostic biomarkers for neurodevelopmental disorders in children and adolescents: a systematic review (2023) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · cited 1×
“As recently noted, studies with small sample sizes (low power), paired with publication bias and file drawer effects as well as high sample variability (true heterogeneity within a condition), can lead a whole field to overestimate the magnitude of the true population effect [49,50].”
BackgroundInsel, T., Cuthbert, B., Garvey, M., Heinssen, R., Pine, D. S., Quinn, K., Sanislow, C., & Wang, P. (2010) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2010.09091379 · Cited by 7,067
Biomarkers and surrogate endpoints: Preferred definitions and conceptual framework
(2001) · Clinical Pharmacology & Therapeutics · doi:10.1067/mcp.2001.113989 · Cited by 6,762
Toward the future of psychiatric diagnosis: the seven pillars of RDoC
Cuthbert, B. N., & Insel, T. R. (2013) · BMC Medicine · doi:10.1186/1741-7015-11-126 · Cited by 3,053 · cited 1×
“This biomarker-clinical phenotype relationship could be linear or nonlinear such that the atypicality only becomes clinically relevant from a certain degree or “tipping point” [39].”
BackgroundIdentification of common genetic risk variants for autism spectrum disorder
Autism Spectrum Disorder Working Group of the Psychiatric Genomics Consortium, Grove, J., BUPGEN, Major Depressive Disorder Working Group of the Psychiatric Genomics Consortium, 23andMe Research Team, Ripke, S., Als, T. D., Mattheisen, M., Walters, R. K., Won, H., et al. (2019) · Nature Genetics · doi:10.1038/s41588-019-0344-8 · Cited by 2,421
Resting-state connectivity biomarkers define neurophysiological subtypes of depression
Drysdale, A. T., Grosenick, L., Downar, J., Dunlop, K., Mansouri, F., Meng, Y., Fetcho, R. N., Zebley, B., Oathes, D. J., Etkin, A., et al. (2017) · Nature Medicine · doi:10.1038/nm.4246 · Cited by 2,170
Comparing effect sizes in follow-up studies: ROC Area, Cohen's d, and r.
Rice, M. E., & Harris, G. T. (2005) · Law and Human Behavior · doi:10.1007/s10979-005-6832-7 · Cited by 1,608
The unicorn, the normal curve, and other improbable creatures.
Micceri, T. (1989) · Psychological Bulletin · doi:10.1037//0033-2909.105.1.156 · Cited by 1,485
Attention-deficit/hyperactivity disorder
Faraone, S. V., Asherson, P., Banaschewski, T., Biederman, J., Buitelaar, J. K., Ramos-Quiroga, J. A., Rohde, L. A., Sonuga-Barke, E. J. S., Tannock, R., & Franke, B. (2015) · Nature Reviews Disease Primers · doi:10.1038/nrdp.2015.20 · Cited by 1,479
Self-citation
Cited by
Evidence for embracing normative modeling
Rutherford, S., Barkema, P., Tso, I. F., Sripada, C., Beckmann, C. F., Ruhe, H. G., & Marquand, A. F. (2023) · eLife · doi:10.7554/elife.85082 · Cited by 175
Self-citationCortese, S., Solmi, M., Michelini, G., Bellato, A., Blanner, C., Canozzi, A., Eudave, L., Farhat, L. C., Højlund, M., Köhler‐Forsberg, O., et al. (2023) · World Psychiatry · doi:10.1002/wps.21037 · Cited by 147
Haghayegh, F., Norouziazad, A., Haghani, E., Feygin, A. A., Rahimi, R. H., Ghavamabadi, H. A., Sadighbayan, D., Madhoun, F., Papagelis, M., Felfeli, T., et al. (2024) · Advanced Science · doi:10.1002/advs.202400595 · Cited by 129
A Systematic Review of Aperiodic Neural Activity in Clinical Investigations
Donoghue, T. (2025) · European Journal of Neuroscience · doi:10.1111/ejn.70255 · Cited by 99
Restoring the missing person to personalized medicine and precision psychiatry
Gómez-Carrillo, A., Paquin, V., Dumas, G., & Kirmayer, L. J. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1041433 · Cited by 64
Self-citation
Metadata
- Authors
- Loth, E., Ahmad, J., Chatham, C., López, B., Carter, B., Crawley, D., Oakley, B., Hayward, H., Cooke, J., San José Cáceres, A., Bzdok, D., Jones, E., Charman, T., Beckmann, C., Bourgeron, T., Toro, R., Buitelaar, J., Murphy, D., & Dumas, G.
- Affiliations
- King's College London (GB)
- University of Greenwich (GB)
- University of Portsmouth (GB)
- Canadian Institute for Advanced Research (CA)
- Birkbeck, University of London (GB)
- Radboud University Medical Center (NL)
- Centre National de la Recherche Scientifique (FR)
- Corresponding author
- Dumas, Guillaume
- ORCID
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-11-18
- Volume
- 17
- Issue
- 11
- Pages
- e1009477
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC8601419
- OpenAlex
- W3213490299
- Cited by
- 52
- References
- 52
- Citations by year
- 2026: 7 · 2025: 11 · 2024: 9 · 2023: 13 · 2022: 12
- FWCI
- 4.91
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Portal (King's College London) (publishedVersion, other-oa)
- BIROn (Birkbeck, University of London) (acceptedVersion)
- Greenwich Academic Literature Archive (University of Greenwich) (acceptedVersion)
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
- HAL (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Funders
- innovative medicines initiative (777394)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Life Sciences › Neuroscience › Cognitive Neuroscience › Autism Spectrum Disorder Research
- Keywords
- biomarker discovery, effect size, within-group variation, autism spectrum disorder, diagnostic value, sensitivity and specificity, non-normal distributions, sample size, case-control comparison, theory of mind, eye tracking, electrophysiology, anatomical structures
- MeSH terms
- Autistic Disorder, Mental Disorders, Computer Simulation, Humans, Individuality, Neuropsychology, Biomarkers, Normal Distribution, Case-Control Studies, Sample Size, Computational Biology, Neuropsychiatry, Neurodevelopmental Disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The meaning of significant mean group differences for biomarker discovery. https://app.cochat.ai/papers/10.1371/journal.pcbi.1009477
- The paper
- Loth, E., Ahmad, J., Chatham, C., López, B., Carter, B., Crawley, D., Oakley, B., Hayward, H., Cooke, J., San José Cáceres, A., Bzdok, D., Jones, E., Charman, T., Beckmann, C., Bourgeron, T., Toro, R., Buitelaar, J., Murphy, D., & Dumas, G. (2021). The meaning of significant mean group differences for biomarker discovery. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1009477
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.