The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Excess Significance Bias in the Literature on Brain Volume Abnormalities
Abstract
CONTEXT: Many studies report volume abnormalities in diverse brain structures in patients with various mental health conditions. OBJECTIVE: To evaluate whether there is evidence for an excess number of statistically significant results in studies of brain volume abnormalities that suggest the presence of bias in the literature. DATA SOURCES: PubMed (articles published from January 2006 to December 2009). STUDY SELECTION: Recent meta-analyses of brain volume abnormalities in participants with various mental health conditions vs control participants with 6 or more data sets included, excluding voxel-based morphometry. DATA EXTRACTION: Standardized effect sizes were extracted in each data set, and it was noted whether the results were "positive" (P < .05) or not. For each data set in each meta-analysis, I estimated the power to detect at α = .05 an effect equal to the summary effect of the respective meta-analysis. The sum of the power estimates gives the number of expected positive data sets. The expected number of positive data sets can then be compared against the observed number. DATA SYNTHESIS: From 8 articles, 41 meta-analyses with 461 data sets were evaluated (median, 10 data sets per meta-analysis) pertaining to 7 conditions. Twenty-one of the 41 meta-analyses had found statistically significant associations, and 142 of 461 (31%) data sets had positive results. Even if the summary effect sizes of the meta-analyses were unbiased, the expected number of positive results would have been only 78.5 compared with the observed number of 142 (P < .001). CONCLUSION: There are too many studies with statistically significant results in the literature on brain volume abnormalities. This pattern suggests strong biases in the literature, with selective outcome reporting and selective analyses reporting being possible explanations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Our results indicated that the median statistical power of these studies was 8% across 461 individual studies contributing to 41 separate meta-analyses, which were drawn from eight articles that were published between 2006 and 2009.”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [73]
Citations
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationThe Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517 · cited 1×
“05, but it should be interpreted with caution owing to the possibility of both false positives and false negatives.(28,29) For I(2), as a rough guide, values exceeding 50% are considered as large heterogeneity beyond chance, and values exceeding 75% are considered very large heterogeneity.”
BackgroundSelf-citationSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · cited 1×
“Many meta-analyses in this field have applied asymmetry (funnelplot) tests thatdeterminewhethersmall studiesgive differentresults fromlargerstudies.(19) If so, suchsmall-study effectsmaybeduetopublicationbiasorotherreportingbias.”
MethodThe case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Self-citationMeta-analysis of regional brain volumes in schizophrenia
Wright, I. C., Rabe-Hesketh, S., Woodruff, P. W. R., David, A. S., Murray, R. M., & Bullmore, E. T. (2000) · American Journal of Psychiatry · doi:10.1176/ajp.157.1.16 · Cited by 1,662
Hippocampal volume and depression: a meta-analysis of MRI studies
Videbech, P., & Ravnkilde, B. (2004) · American Journal of Psychiatry · doi:10.1176/appi.ajp.161.11.1957 · Cited by 1,590
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546
Self-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Self-citationUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254
Self-citationHonea, R., Crow, T. J., Passingham, D., & Mackay, C. E. (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.12.2233 · Cited by 1,157
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052
Self-citationKoolschijn, P. C. M. P., van Haren, N. E. M., Lensvelt‐Mulders, G. J. L. M., Hulshoff Pol, H. E., & Kahn, R. S. (2009) · Human Brain Mapping · doi:10.1002/hbm.20801 · Cited by 902
A meta-analysis of structural brain abnormalities in PTSD
KARL, A., SCHAEFER, M., MALTA, L., DORFEL, D., ROHLEDER, N., & WERNER, A. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2006.03.004 · Cited by 881
Bogerts, B. (1985) · Archives of General Psychiatry · doi:10.1001/archpsyc.1985.01790310046006 · Cited by 880
Voxel-wise meta-analysis of grey matter changes in obsessive-compulsive disorder
Radua, J., & Mataix-Cols, D. (2009) · British Journal of Psychiatry · doi:10.1192/bjp.bp.108.055046 · Cited by 863
Brain abnormality in schizophrenia
Lawrie, S. M., & Abukmeil, S. S. (1998) · The British Journal of Psychiatry · doi:10.1192/bjp.172.2.110 · Cited by 845
Steen, R. G., Mull, C., Mcclure, R., Hamer, R. M., & Lieberman, J. A. (2006) · British Journal of Psychiatry · doi:10.1192/bjp.188.6.510 · Cited by 832
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751
Self-citationBremner, J. D. (1999) · Biological Psychiatry · doi:10.1016/s0006-3223(99)00009-8 · Cited by 724
Meta-analysis, Database, and Meta-regression of 98 Structural Imaging Studies in Bipolar Disorder
Kempton, M. J., Geddes, J. R., Ettinger, U., Williams, S., & Grasby, P. M. (2008) · Archives of General Psychiatry · doi:10.1001/archpsyc.65.9.1017 · Cited by 547
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al‐Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 533
Self-citationStanfield, A. C., McIntosh, A. M., Spencer, M. D., Philip, R., Gaur, S., & Lawrie, S. M. (2008) · European Psychiatry · doi:10.1016/j.eurpsy.2007.05.006 · Cited by 524
Statistics Notes: Blinding in clinical trials and other studies
Day, S. J. (2000) · BMJ · doi:10.1136/bmj.321.7259.504 · Cited by 487
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 474
Self-citationVoxelwise Meta-Analysis of Gray Matter Abnormalities in Bipolar Disorder
Bora, E., Fornito, A., Yücel, M., & Pantelis, C. (2010) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.01.020 · Cited by 378
Fornito, A., Yücel, M., Patti, J., Wood, S. J., & Pantelis, C. (2009) · Schizophrenia Research · doi:10.1016/j.schres.2008.12.011 · Cited by 359
Brain Volumes in Relatives of Patients With Schizophrenia
Boos, H. B. M., Aleman, A., Cahn, W., Pol, H. E. H., & Kahn, R. S. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.3.297 · Cited by 356
Magnetic Resonance Imaging of Hippocampal Subfields in Posttraumatic Stress Disorder
Wang, Z., Neylan, T. C., Mueller, S. G., Lenoci, M., Truran, D., Marmar, C. R., Weiner, M. W., & Schuff, N. W. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2009.205 · Cited by 307
Meta-analysis of brain volume changes in obsessive-compulsive disorder
Rotge, J.-Y., Guehl, D., Dilharreguy, B., Tignol, J., Bioulac, B., Allard, M., Burbaud, P., & Aouizerate, B. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.06.019 · Cited by 284
A magnetic resonance imaging study of putamen nuclei in major depression
Husain, M. M., McDonald, W. M., Doraiswamy, P. M., Figiel, G. S., Na, C., Escalona, P. R., Boyko, O. B., Nemeroff, C. B., & Krishnan, K. R. R. (1991) · Psychiatry Research: Neuroimaging · doi:10.1016/0925-4927(91)90001-7 · Cited by 256
Haller, J. W., Banerjee, A., Christensen, G. E., Gado, M., Joshi, S. C., Miller, M. I., Sheline, Y. I., Vannier, M. W., & Csernansky, J. G. (1997) · Radiology · doi:10.1148/radiology.202.2.9015081 · Cited by 219
Pereira, T., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.13 · Cited by 161
Self-citationSteen, R. G., Ogg, R. J., Reddick, W. E., & Kingsley, P. B. (1997) · PubMed Central · Cited by 127
Amygdala volumes in mood disorders — Meta-analysis of magnetic resonance volumetry studies
Hájek, T., Kopeček, M., Kožený, J., Gunde, E., Alda, M., & Höschl, C. (2008) · Journal of Affective Disorders · doi:10.1016/j.jad.2008.10.007 · Cited by 121
Disclose all data in publications
Baggerly, K. (2010) · Nature · doi:10.1038/467401b · Cited by 63
Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results
Ioannidis, J. P. A. (2006) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-05-0921 · Cited by 62
Self-citationWang, D., Chalk, J. B., Rose, S., de Zubicaray, G. I., Cowin, G., Galloway, G. J., Barnes, D., Spooner, D., Doddrell, D. M., & Semple, J. M. (2002) · Magnetic Resonance Imaging · doi:10.1016/s0730-725x(02)00472-1 · Cited by 61
Segmentation and measurement of brain structures in MRI including confidence bounds
Ballester, M. Á. G., Zisserman, A., & Brady, M. (2000) · Medical Image Analysis · doi:10.1016/s1361-8415(00)00013-x · Cited by 52
Kavvoura, F. K., McQueen, M. B., Khoury, M. J., Tanzi, R. E., Bertram, L., & Ioannidis, J. P. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn206 · Cited by 48
Self-citationAn open letter to the scientific journals
Ball, C. A., Sherlock, G. J., Parkinson, H., Rocca‐Serra, P., Brooksbank, C., Causton, H. C., Cavalieri, D., Gaasterland, T., Hingamp, P., Holstege, F. C. P., et al. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.11.1409 · Cited by 45
Volume estimation of prefrontal cortical subfields using MRI and stereology
Howard, M. A., Roberts, N., García‐Fiñana, M., & Cowell, P. E. (2003) · Brain Research Protocols · doi:10.1016/s1385-299x(02)00202-7 · Cited by 45
Tofts, P. S., Barker, G. J., Filippi, M., Gawne-Cain, M. L., & Lai, M. (1997) · Magnetic Resonance Imaging · doi:10.1016/s0730-725x(96)00339-6 · Cited by 23
Chen, J. J., & Chen, C. (2018) · Routledge eBooks · doi:10.1201/9781351110273-130001126 · Cited by 9
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641
Self-citationBrain Volumes in Schizophrenia: A Meta-Analysis in Over 18 000 Subjects
Haijma, S. V., Van Haren, N., Cahn, W., Koolschijn, P. C. M. P., Hulshoff Pol, H. E., & Kahn, R. S. (2013) · Schizophrenia Bulletin · doi:10.1093/schbul/sbs118 · Cited by 1,012
Theodoratou, E., Tzoulaki, I., Zgaga, L., & Ioannidis, J. P. A. (2014) · BMJ · doi:10.1136/bmj.g2035 · Cited by 956
Self-citationThe neuroscience of socioeconomic status: Correlates, causes, and consequences
Farah, M. J. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.034 · Cited by 653
Fusar-Poli, P., Smieskova, R., Kempton, M. J., Ho, B. C., Andreasen, N. C., & Borgwardt, S. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.06.001 · Cited by 512
Metadata
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Archives of General Psychiatry
- Publisher
- American Medical Association (AMA)
- Published
- 2011-08-01
- Volume
- 68
- Issue
- 8
- Pages
- 773
- ISSN
- 0003-990X (ISSN-L)
- OpenAlex
- W2169662981
- Cited by
- 299
- References
- 48
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 8 · 2023: 5 · 2022: 5 · 2021: 25 · 2020: 55 · 2019: 17 · 2018: 23 · 2017: 19 · 2016: 17 · 2015: 27 · 2014: 26 · 2013: 19 · 2012: 36
- FWCI
- 29.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Archives of General Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- publication bias, selective outcome reporting, meta-analysis, mental illness, statistical power, standardized effect size, neuroimaging
- MeSH terms
- Mental Disorders, Brain, Humans, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
