Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Effect sizes are commonly interpreted using heuristics established by Cohen (e.g., small: r = .1, medium r = .3, large r = .5), despite mounting evidence that these guidelines are mis-calibrated to the effects typically found in psychological research. This study’s aims were to 1) describe the distribution of effect sizes across multiple instruments, 2) consider factors qualifying the effect size distribution, and 3) identify examples as benchmarks for various effect sizes. For aim one, effect size distributions were illustrated from a large, diverse sample of 9/10-year-old children. This was done by conducting Pearson’s correlations among 161 variables representing constructs from all questionnaires and tasks from the Adolescent Brain and Cognitive Development Study® baseline data. To achieve aim two, factors qualifying this distribution were tested by comparing the distributions of effect size among various modifications of the aim one analyses. These modified analytic strategies included comparisons of effect size distributions for different types of variables, for analyses using statistical thresholds, and for analyses using several covariate strategies. In aim one analyses, the median in-sample effect size was .03, and values at the first and third quartiles were .01 and .07. In aim two analyses, effects were smaller for associations across instruments, content domains, and reporters, as well as when covarying for sociodemographic factors. Effect sizes were larger when thresholding for statistical significance. In analyses intended to mimic conditions used in “real-world” analysis of ABCD data, the median in-sample effect size was .05, and values at the first and third quartiles were .03 and .09. To achieve aim three, examples for varying effect sizes are reported from the ABCD dataset as benchmarks for future work in the dataset. In summary, this report finds that empirically determined effect sizes from a notably large dataset are smaller than would be expected based on existing heuristics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
The Need for a New Medical Model: A Challenge for Biomedicine
Engel, G. L. (1977) · Science · doi:10.1126/science.847460 · Cited by 12,887
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Whiteside, S. P., & Lynam, D. R. (2001) · Personality and Individual Differences · doi:10.1016/s0191-8869(00)00064-7 · Cited by 4,123 · cited 1×
“These previously validated instruments include the Child Behavior Checklist [15], the Urgency, Premeditation, Perseverance, Sensation Seeking, and Positive Urgency (UPPS-P) impulsive behavior scale [16], the Vancouver Index of Acculturation [17], among others.”
MethodEvaluating Effect Size in Psychological Research: Sense and Nonsense
Funder, D. C., & Ozer, D. J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847202 · Cited by 3,295 · cited 4×
“The current results suggest that effect sizes for self-report and cognitive task data collected from children aged 9–10 may be more consistent with recently proposed alternative heuristic schemes [5, 7, 8] than those Cohen previously suggested [3].”
BackgroundResultInfluentialEffect size guidelines for individual differences researchers
Gignac, G. E., & Szodorai, E. T. (2016) · Personality and Individual Differences · doi:10.1016/j.paid.2016.06.069 · Cited by 2,860
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
The biopsychosocial model 25 years later: principles, practice, and scientific inquiry
Borrell-Carrio, F. (2004) · The Annals of Family Medicine · doi:10.1370/afm.245 · Cited by 1,546
Psychological testing and psychological assessment: A review of evidence and issues.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., & Reed, G. M. (2001) · American Psychologist · doi:10.1037//0003-066x.56.2.128 · Cited by 1,469 · cited 1×
“For example, Meyer and colleagues [6] lay out benchmarks for effects of various sizes helping researchers develop better concrete understanding about effect sizes using examples from daily life (e.”
BackgroundRecruiting the ABCD sample: Design considerations and procedures
Garavan, H., Bartsch, H., Conway, K., Decastro, A., Goldstein, R. Z., Heeringa, S., Jernigan, T., Potter, A., Thompson, W., & Zahs, D. (2018) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2018.04.004 · Cited by 1,393
Self-citation
Cited by
Sex differences in functional brain networks involved in interoception: An fMRI study
Alfano, V., Cavaliere, C., Di Cecca, A., Ciccarelli, G., Salvatore, M., Aiello, M., & Federico, G. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2023.1130025 · Cited by 32
Owens, M. M., Albaugh, M. D., Allgaier, N., Yuan, D., Robert, G., Cupertino, R. B., Spechler, P. A., Juliano, A., Hahn, S., Banaschewski, T., et al. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-01956-4 · Cited by 22
Self-citationSuzuki, T., Gu, P., Grove, T. B., Hammond, T., Collins, K. M., Pamidighantam, P., Arnold, P. D., Taylor, S. F., Liu, Y., Gehring, W. J., et al. (2023) · Biological Psychiatry · doi:10.1016/j.biopsych.2022.10.020 · Cited by 20
Patterns of psychopathology and cognition in sex chromosome aneuploidy
Rau, S., Whitman, E. T., Schauder, K., Gogate, N., Lee, N. R., Kenworthy, L., & Raznahan, A. (2021) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-021-09407-9 · Cited by 14
“However, there is growing recognition that correlations between different dimensions of cognition and psychopathology can be weak at the population level and require large sample sizes to be reliably estimated [64], which poses a particular challenge for estimating such correlations within individually rare SCA subgro…”
BackgroundMarshall, A. T., Adise, S., Cardenas-Iniguez, C., Hippolyte, O. K., Parchment, C. A., Villalobos, T. I., Wong, L. T., Cisneros, C. P., Kan, E. C., Palmer, C. E., et al. (2023) · Health Psychology · doi:10.1037/hea0001254 · Cited by 10
Metadata
- Authors
- Owens, M. M., Potter, A., Hyatt, C. S., Albaugh, M., Thompson, W. K., Jernigan, T., Yuan, D., Hahn, S., Allgaier, N., & Garavan, H.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-09-23
- Volume
- 16
- Issue
- 9
- Pages
- e0257535
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W3163533628
- Cited by
- 156
- References
- 14
- FWCI
- 13.28
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study. https://app.cochat.ai/papers/10.1371/journal.pone.0257535
- The paper
- Owens, M. M., Potter, A., Hyatt, C. S., Albaugh, M., Thompson, W. K., Jernigan, T., Yuan, D., Hahn, S., Allgaier, N., & Garavan, H. (2021). Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study. PLOS ONE. https://doi.org/10.1371/journal.pone.0257535
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.