Excess Significance Bias in the Literature on Brain Volume Abnormalities
Abstract
CONTEXT: Many studies report volume abnormalities in diverse brain structures in patients with various mental health conditions. OBJECTIVE: To evaluate whether there is evidence for an excess number of statistically significant results in studies of brain volume abnormalities that suggest the presence of bias in the literature. DATA SOURCES: PubMed (articles published from January 2006 to December 2009). STUDY SELECTION: Recent meta-analyses of brain volume abnormalities in participants with various mental health conditions vs control participants with 6 or more data sets included, excluding voxel-based morphometry. DATA EXTRACTION: Standardized effect sizes were extracted in each data set, and it was noted whether the results were "positive" (P < .05) or not. For each data set in each meta-analysis, I estimated the power to detect at α = .05 an effect equal to the summary effect of the respective meta-analysis. The sum of the power estimates gives the number of expected positive data sets. The expected number of positive data sets can then be compared against the observed number. DATA SYNTHESIS: From 8 articles, 41 meta-analyses with 461 data sets were evaluated (median, 10 data sets per meta-analysis) pertaining to 7 conditions. Twenty-one of the 41 meta-analyses had found statistically significant associations, and 142 of 461 (31%) data sets had positive results. Even if the summary effect sizes of the meta-analyses were unbiased, the expected number of positive results would have been only 78.5 compared with the observed number of 142 (P < .001). CONCLUSION: There are too many studies with statistically significant results in the literature on brain volume abnormalities. This pattern suggests strong biases in the literature, with selective outcome reporting and selective analyses reporting being possible explanations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The median statistical power of studies of brain volume abnormalities was 8% across 461 individual studies contributing to 41 separate meta-analyses, drawn from eight articles published between 2006 and 2009.
“Our results indicated that the median statistical power of these studies was 8% across 461 individual studies contributing to 41 separate meta-analyses, which were drawn from eight articles that were published between 2006 and 2009.”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [73]
Cites
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · see citing works
The Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,657 · see citing works
Quantitative Synthesis in Systematic Reviews
Lau, J., Ioannidis, J. P. A., & Schmid, C. H. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-9-199711010-00008 · Cited by 2,517 · cited 1× · see citing works
“05, but it should be interpreted with caution owing to the possibility of both false positives and false negatives.(28,29) For I(2), as a rough guide, values exceeding 50% are considered as large heterogeneity beyond chance, and values exceeding 75% are considered very large heterogeneity.”
BackgroundSelf-citationSelective publication of antidepressant trials and its influence on apparent efficacy
Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008) · New England Journal of Medicine · doi:10.1056/nejmsa065779 · Cited by 2,435 · see citing works
Publication Bias in Meta‐Analysis: Prevention, Assessment and Adjustments
(2005) · doi:10.1002/0470870168 · Cited by 2,368 · see citing works
A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints
Harbord, R. M., Egger, M., & Sterne, J. A. C. (2006) · Statistics in Medicine · doi:10.1002/sim.2380 · Cited by 2,267 · cited 1× · see citing works
“Many meta-analyses in this field have applied asymmetry (funnelplot) tests thatdeterminewhethersmall studiesgive differentresults fromlargerstudies.(19) If so, suchsmall-study effectsmaybeduetopublicationbiasorotherreportingbias.”
MethodThe case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · see citing works
Self-citationMeta-analysis of regional brain volumes in schizophrenia
Wright, I. C., Rabe-Hesketh, S., Woodruff, P. W. R., David, A. S., Murray, R. M., & Bullmore, E. T. (2000) · American Journal of Psychiatry · doi:10.1176/ajp.157.1.16 · Cited by 1,662 · see citing works
Hippocampal volume and depression: a meta-analysis of MRI studies
Videbech, P., & Ravnkilde, B. (2004) · American Journal of Psychiatry · doi:10.1176/appi.ajp.161.11.1957 · Cited by 1,590 · see citing works
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · see citing works
Self-citationWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · see citing works
Self-citationUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254 · see citing works
Self-citationHonea, R., Crow, T. J., Passingham, D., & Mackay, C. E. (2005) · American Journal of Psychiatry · doi:10.1176/appi.ajp.162.12.2233 · Cited by 1,157 · see citing works
The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey
Ioannidis, J. P. A., & Trikalinos, T. A. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.060410 · Cited by 1,052 · see citing works
Self-citationKoolschijn, P. C. M. P., van Haren, N. E. M., Lensvelt‐Mulders, G. J. L. M., Hulshoff Pol, H. E., & Kahn, R. S. (2009) · Human Brain Mapping · doi:10.1002/hbm.20801 · Cited by 902 · see citing works
A meta-analysis of structural brain abnormalities in PTSD
KARL, A., SCHAEFER, M., MALTA, L., DORFEL, D., ROHLEDER, N., & WERNER, A. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2006.03.004 · Cited by 881 · see citing works
Bogerts, B. (1985) · Archives of General Psychiatry · doi:10.1001/archpsyc.1985.01790310046006 · Cited by 880 · see citing works
Voxel-wise meta-analysis of grey matter changes in obsessive-compulsive disorder
Radua, J., & Mataix-Cols, D. (2009) · British Journal of Psychiatry · doi:10.1192/bjp.bp.108.055046 · Cited by 863 · see citing works
Brain abnormality in schizophrenia
Lawrie, S. M., & Abukmeil, S. S. (1998) · The British Journal of Psychiatry · doi:10.1192/bjp.172.2.110 · Cited by 845 · see citing works
Steen, R. G., Mull, C., Mcclure, R., Hamer, R. M., & Lieberman, J. A. (2006) · British Journal of Psychiatry · doi:10.1192/bjp.188.6.510 · Cited by 832 · see citing works
An exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
Self-citationBremner, J. D. (1999) · Biological Psychiatry · doi:10.1016/s0006-3223(99)00009-8 · Cited by 703 · see citing works
Meta-analysis, Database, and Meta-regression of 98 Structural Imaging Studies in Bipolar Disorder
Kempton, M. J., Geddes, J. R., Ettinger, U., Williams, S., & Grasby, P. M. (2008) · Archives of General Psychiatry · doi:10.1001/archpsyc.65.9.1017 · Cited by 533 · see citing works
Stanfield, A. C., McIntosh, A. M., Spencer, M. D., Philip, R., Gaur, S., & Lawrie, S. M. (2008) · European Psychiatry · doi:10.1016/j.eurpsy.2007.05.006 · Cited by 524 · see citing works
Why Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al‐Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 504 · see citing works
Self-citationStatistics Notes: Blinding in clinical trials and other studies
Day, S. J. (2000) · BMJ · doi:10.1136/bmj.321.7259.504 · Cited by 487 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 473 · see citing works
Self-citationVoxelwise Meta-Analysis of Gray Matter Abnormalities in Bipolar Disorder
Bora, E., Fornito, A., Yücel, M., & Pantelis, C. (2010) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.01.020 · Cited by 378 · see citing works
Fornito, A., Yücel, M., Patti, J., Wood, S. J., & Pantelis, C. (2009) · Schizophrenia Research · doi:10.1016/j.schres.2008.12.011 · Cited by 359 · see citing works
Brain Volumes in Relatives of Patients With Schizophrenia
Boos, H. B. M., Aleman, A., Cahn, W., Pol, H. E. H., & Kahn, R. S. (2007) · Archives of General Psychiatry · doi:10.1001/archpsyc.64.3.297 · Cited by 344 · see citing works
Magnetic Resonance Imaging of Hippocampal Subfields in Posttraumatic Stress Disorder
Wang, Z., Neylan, T. C., Mueller, S. G., Lenoci, M., Truran, D., Marmar, C. R., Weiner, M. W., & Schuff, N. W. (2010) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2009.205 · Cited by 300 · see citing works
Meta-analysis of brain volume changes in obsessive-compulsive disorder
Rotge, J.-Y., Guehl, D., Dilharreguy, B., Tignol, J., Bioulac, B., Allard, M., Burbaud, P., & Aouizerate, B. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.06.019 · Cited by 284 · see citing works
A magnetic resonance imaging study of putamen nuclei in major depression
Husain, M. M., McDonald, W. M., Doraiswamy, P. M., Figiel, G. S., Na, C., Escalona, P. R., Boyko, O. B., Nemeroff, C. B., & Krishnan, K. R. R. (1991) · Psychiatry Research: Neuroimaging · doi:10.1016/0925-4927(91)90001-7 · Cited by 256 · see citing works
Haller, J. W., Banerjee, A., Christensen, G. E., Gado, M., Joshi, S. C., Miller, M. I., Sheline, Y. I., Vannier, M. W., & Csernansky, J. G. (1997) · Radiology · doi:10.1148/radiology.202.2.9015081 · Cited by 219 · see citing works
Pereira, T., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010) · Research Synthesis Methods · doi:10.1002/jrsm.13 · Cited by 161 · see citing works
Self-citationSteen, R. G., Ogg, R. J., Reddick, W. E., & Kingsley, P. B. (1997) · PubMed Central · Cited by 127 · see citing works
Amygdala volumes in mood disorders — Meta-analysis of magnetic resonance volumetry studies
Hájek, T., Kopeček, M., Kožený, J., Gunde, E., Alda, M., & Höschl, C. (2008) · Journal of Affective Disorders · doi:10.1016/j.jad.2008.10.007 · Cited by 120 · see citing works
Disclose all data in publications
Baggerly, K. (2010) · Nature · doi:10.1038/467401b · Cited by 63 · see citing works
Journals Should Publish All “Null” Results and Should Sparingly Publish “Positive” Results
Ioannidis, J. P. A. (2006) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-05-0921 · Cited by 62 · see citing works
Self-citationWang, D., Chalk, J. B., Rose, S., de Zubicaray, G. I., Cowin, G., Galloway, G. J., Barnes, D., Spooner, D., Doddrell, D. M., & Semple, J. M. (2002) · Magnetic Resonance Imaging · doi:10.1016/s0730-725x(02)00472-1 · Cited by 61 · see citing works
Segmentation and measurement of brain structures in MRI including confidence bounds
Ballester, M. Á. G., Zisserman, A., & Brady, M. (2000) · Medical Image Analysis · doi:10.1016/s1361-8415(00)00013-x · Cited by 50 · see citing works
Kavvoura, F. K., McQueen, M. B., Khoury, M. J., Tanzi, R. E., Bertram, L., & Ioannidis, J. P. A. (2008) · American Journal of Epidemiology · doi:10.1093/aje/kwn206 · Cited by 47 · see citing works
Self-citationAn open letter to the scientific journals
Ball, C. A., Sherlock, G. J., Parkinson, H., Rocca‐Serra, P., Brooksbank, C., Causton, H. C., Cavalieri, D., Gaasterland, T., Hingamp, P., Holstege, F. C. P., et al. (2002) · Bioinformatics · doi:10.1093/bioinformatics/18.11.1409 · Cited by 45 · see citing works
Volume estimation of prefrontal cortical subfields using MRI and stereology
Howard, M. A., Roberts, N., García‐Fiñana, M., & Cowell, P. E. (2003) · Brain Research Protocols · doi:10.1016/s1385-299x(02)00202-7 · Cited by 45 · see citing works
Tofts, P. S., Barker, G. J., Filippi, M., Gawne-Cain, M. L., & Lai, M. (1997) · Magnetic Resonance Imaging · doi:10.1016/s0730-725x(96)00339-6 · Cited by 23 · see citing works
Chen, J. J., & Chen, C. (2018) · Routledge eBooks · doi:10.1201/9781351110273-130001126 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
Self-citationBrain Volumes in Schizophrenia: A Meta-Analysis in Over 18 000 Subjects
Haijma, S. V., Van Haren, N., Cahn, W., Koolschijn, P. C. M. P., Hulshoff Pol, H. E., & Kahn, R. S. (2013) · Schizophrenia Bulletin · doi:10.1093/schbul/sbs118 · Cited by 1,012 · see citing works
Theodoratou, E., Tzoulaki, I., Zgaga, L., & Ioannidis, J. P. A. (2014) · BMJ · doi:10.1136/bmj.g2035 · Cited by 956 · see citing works
Self-citationThe neuroscience of socioeconomic status: Correlates, causes, and consequences
Farah, M. J. (2017) · Neuron · doi:10.1016/j.neuron.2017.08.034 · Cited by 653 · see citing works
Fusar-Poli, P., Smieskova, R., Kempton, M. J., Ho, B. C., Andreasen, N. C., & Borgwardt, S. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.06.001 · Cited by 512 · see citing works
Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Self-citationConnor, J. P., Gullo, M. J., White, A., & Kelly, A. B. (2014) · Current Opinion in Psychiatry · doi:10.1097/yco.0000000000000069 · Cited by 462 · see citing works
The secret lives of experiments: Methods reporting in the fMRI literature
Carp, J. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.07.004 · Cited by 473 · see citing works
Systematic meta-review and quality assessment of the structural brain alterations in schizophrenia
Shepherd, A. M., Laurens, K. R., Matheson, S. L., Carr, V. J., & Green, M. J. (2012) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2011.12.015 · Cited by 412 · see citing works
Li, X., Meng, X., Timofeeva, M. N., Tzoulaki, I., Tsilidis, K. K., Ioannidis, J. P. A., Campbell, H. J., & Τheodoratou, E. (2017) · BMJ · doi:10.1136/bmj.j2376 · Cited by 402 · see citing works
Self-citationNeuroimaging of the periaqueductal gray: State of the field
Linnman, C., Moulton, E. A., Barmettler, G., Becerra, L. R., & Borsook, D. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.11.095 · Cited by 402 · see citing works
On the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
Carp, J. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00149 · Cited by 381 · see citing works
Scientific Utopia: I. Opening Scientific Communication
Nosek, B. A., & Bar‐Anan, Y. (2012) · Psychological Inquiry · doi:10.1080/1047840x.2012.692215 · Cited by 366 · see citing works
Eliot, L., Ahmed, A., Khan, H., & Patel, J. Y. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.02.026 · Cited by 351 · see citing works
Pietschnig, J., Penke, L., Wicherts, J. M., Zeiler, M., & Voracek, M. (2015) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2015.09.017 · Cited by 316 · see citing works
Sucksmith, E., Roth, I., & Hoekstra, R. A. (2011) · Neuropsychology Review · doi:10.1007/s11065-011-9183-9 · Cited by 336 · see citing works
Evaluation of excess significance bias in animal studies of neurological diseases
Tsilidis, K. K., Panagiotou, O. A., Sena, E. S., Aretouli, E., Evangelou, E., Howells, D. W., Salman, R. A.-S., Macleod, M. R., & Ioannidis, J. P. A. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001609 · Cited by 295 · see citing works
Self-citationMackey, S., Allgaier, N. A., Chaarani, B., Spechler, P. A., Orr, C., Bunn, J., Allen, N. B., Nelly Alia-klein, A., Batalla, A., Blaine, S. K., et al. (2018) · American Journal of Psychiatry · doi:10.1176/appi.ajp.2018.17040415 · Cited by 266 · see citing works
Meta-analytic methods for neuroimaging data explained
Radua, J., & Mataix-Cols, D. (2012) · Biology of Mood & Anxiety Disorders · doi:10.1186/2045-5380-2-6 · Cited by 246 · see citing works
Lack of evidence to favor specific preventive interventions in psychosis: a network meta‐analysis
Davies, C., Cipriani, A., Ioannidis, J. P. A., Radua, J., Stahl, D., Provenzani, U., McGuire, P., & Fusar‐Poli, P. (2018) · World Psychiatry · doi:10.1002/wps.20526 · Cited by 229 · see citing works
Self-citationHall, W., Carter, A., & Forlini, C. (2015) · The Lancet Psychiatry · doi:10.1016/s2215-0366(14)00126-6 · Cited by 233 · see citing works
Gregory, S., Ffytche, D., Simmons, A., Kumari, V., Howard, M., Hodgins, S., & Blackwood, N. (2012) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2012.222 · Cited by 185 · see citing works
Hippocampal volume across age: Nomograms derived from over 19,700 people in UK Biobank
Nobis, L., Manohar, S., Smith, S. M., Alfaro‐Almagro, F., Jenkinson, M., Mackay, C. E., & Masud Husain, B. (2019) · NeuroImage Clinical · doi:10.1016/j.nicl.2019.101904 · Cited by 226 · see citing works
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210 · see citing works
Self-citationMullane, K., & Williams, M. J. (2012) · Biochemical Pharmacology · doi:10.1016/j.bcp.2012.11.014 · Cited by 212 · see citing works
Clinical Outcome Prediction by MicroRNAs in Human Cancer: A Systematic Review
Nair, V. S., Maeda, L. S., & Ioannidis, J. P. A. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs027 · Cited by 213 · see citing works
Self-citationPublication bias in meta‐analyses from the Cochrane Database of Systematic Reviews
Kiciński, M., Springate, D. A., & Kontopantelis, E. (2015) · Statistics in Medicine · doi:10.1002/sim.6525 · Cited by 213 · see citing works
Systematic Meta-Analysis of Insula Volume in Schizophrenia
Shepherd, A. M., Matheson, S. L., Laurens, K. R., Carr, V. J., & Green, M. J. (2012) · Biological Psychiatry · doi:10.1016/j.biopsych.2012.04.020 · Cited by 207 · see citing works
Murphy, S. E., Norbury, R., Godlewska, B. R., Cowen, P. J., Mannie, Z. M., Harmer, C. J., & Munafo, M. R. (2012) · Molecular Psychiatry · doi:10.1038/mp.2012.19 · Cited by 205 · see citing works
The Challenges and Promise of Neuroimaging in Psychiatry
Linden, D. E. J. (2012) · Neuron · doi:10.1016/j.neuron.2011.12.014 · Cited by 204 · see citing works
Umbrella reviews: what they are and why we need them
Papatheodorou, S. (2019) · European Journal of Epidemiology · doi:10.1007/s10654-019-00505-6 · Cited by 201 · see citing works
Daniels, J. K., Lamke, J.-P., Gaebler, M., Walter, H., & Scheel, M. (2013) · Depression and Anxiety · doi:10.1002/da.22044 · Cited by 190 · see citing works
Neuroanatomy of developmental dyslexia: Pitfalls and promise
Ramus, F., Altarelli, I., Jednoróg, K., Zhao, J., & di Covella, L. S. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.08.001 · Cited by 192 · see citing works
The human hippocampus is not sexually-dimorphic: Meta-analysis of structural MRI volumes
Tan, A., Ma, W., Vira, A., Marwha, D., & Eliot, L. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2015.08.050 · Cited by 180 · see citing works
Li, D., Lu, Y., Yuan, S., Cai, X., He, Y., Chen, J., Wu, Q., He, D., Fang, A., Bo, Y., et al. (2022) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqac074 · Cited by 185 · see citing works
The Neurobiology of “Food Addiction” and Its Implications for Obesity Treatment and Policy
Carter, A., Hendrikse, J. J., Lee, N., Yücel, M., Verdejo‐García, A., Andrews, Z. B., & Hall, W. (2016) · Annual Review of Nutrition · doi:10.1146/annurev-nutr-071715-050909 · Cited by 176 · see citing works
Objective and subjective cognitive enhancing effects of mixed amphetamine salts in healthy people
Ilieva, I. P., Boland, J. R., & Farah, M. J. (2012) · Neuropharmacology · doi:10.1016/j.neuropharm.2012.07.021 · Cited by 164 · see citing works
Power Posing: P-Curving the Evidence
Simmons, J., & Simonsohn, U. (2017) · Psychology Science · doi:10.1177/0956797616658563 · Cited by 147 · see citing works
Acar, F., Seurinck, R., Eickhoff, S. B., & Moerkerke, B. (2018) · PLOS ONE · doi:10.1371/journal.pone.0208177 · Cited by 154 · see citing works
“In neuroimaging literature, it has been shown that there is an excess of positive findings in literature [24, 25].”
BackgroundDai, S., Wellens, J., Yang, N., Li, D., Wang, J., Wang, L., Yuan, S., He, Y., Song, P., Munger, R., et al. (2024) · Clinical Nutrition · doi:10.1016/j.clnu.2024.04.016 · Cited by 151 · see citing works
Ioannidis, J. P. A. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.002 · Cited by 150 · see citing works
Self-citationReproducible and replicable pain research: a critical review
Lee, H., Lamb, S. E., Bagg, M. K., Toomey, E., Cashin, A. G., & Moseley, G. L. (2018) · Pain · doi:10.1097/j.pain.0000000000001254 · Cited by 150 · see citing works
Ching, C. R. K., Hibar, D. P., Gurholt, T. P., Nunes, A., Thomopoulos, S. I., Abé, C., Agartz, I., Brouwer, R. M., Cannon, D. M., de Zwarte, S. M. C., et al. (2020) · Human Brain Mapping · doi:10.1002/hbm.25098 · Cited by 143 · see citing works
Neuroanatomical abnormalities in chronic tinnitus in the human brain
Adjamian, P., Hall, D. A., Palmer, A. R., Allan, T. W., & Langers, D. R. M. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.05.013 · Cited by 141 · see citing works
Potential Reporting Bias in Neuroimaging Studies of Sex Differences
David, S., Naudet, F., Laude, J. R., Raduà, J., Fusar-Poli, P., Chu, I., Stefanick, M., & Ioannidis, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-23976-1 · Cited by 123 · see citing works
Self-citationPotential Reporting Bias in fMRI Studies of the Brain
David, S. P., Ware, J. J., Chu, I. M., Loftus, P. D., Fusar-Poli, P., Radua, J., Munafò, M. R., & Ioannidis, J. P. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0070104 · Cited by 130 · see citing works
Self-citationThe Cambridge Handbook of Personality Psychology
(2020) · Cambridge University Press eBooks · doi:10.1017/9781108264822 · Cited by 140 · see citing works
Lessons of ALS imaging: Pitfalls and future directions — A critical review
Bede, P., & Hardiman, O. M. (2014) · NeuroImage Clinical · doi:10.1016/j.nicl.2014.02.011 · Cited by 122 · see citing works
Liu, D., Meng, X., Tian, Q., Cao, W., Fan, X., Wu, L., Song, M., Meng, Q., Wang, W., & Wang, Y. (2021) · Advances in Nutrition · doi:10.1093/advances/nmab142 · Cited by 127 · see citing works
RDoC: Translating promise into progress
Patrick, C. J., & Hajcak, G. (2016) · Psychophysiology · doi:10.1111/psyp.12612 · Cited by 114 · see citing works
Quantifying Inhibitory Control as Externalizing Proneness: A Cross-Domain Model
Venables, N. C., Foell, J., Yancey, J. R., Kane, M. J., Engle, R. W., & Patrick, C. J. (2018) · Clinical Psychological Science · doi:10.1177/2167702618757690 · Cited by 116 · see citing works
Focal abnormalities in idiopathic generalized epilepsy: A critical review of the literature
Seneviratne, U. K., Cook, M. J., & D'Souza, W. J. (2014) · Epilepsia · doi:10.1111/epi.12688 · Cited by 114 · see citing works
Everything Matters: The ReproNim Perspective on Reproducible Neuroimaging
Kennedy, D. N., Abraham, S. A., Bates, J. F., Crowley, A., Ghosh, S., Gillespie, T., Goncalves, M., Grethe, J. S., Halchenko, Y. O., Hanke, M., et al. (2019) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2019.00001 · Cited by 113 · see citing works
Does psychotherapy work? An umbrella review of meta‐analyses of randomized controlled trials
Dragioti, E., Karathanos, V. T., Gerdle, B. U. C., & Εvangelou, Ε. (2017) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12713 · Cited by 112 · see citing works
Coronado-Montoya, S., Levis, A. W., Kwakkenbos, L., Steele, R. J., Turner, E. H., & Thombs, B. D. (2016) · PLOS ONE · doi:10.1371/journal.pone.0153220 · Cited by 107 · see citing works
Bias in Associations of Emerging Biomarkers With Cardiovascular Disease
Tzoulaki, I., Siontis, K., Εvangelou, Ε., & Ioannidis, J. P. A. (2013) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2013.3018 · Cited by 108 · see citing works
Self-citationvan Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084896 · Cited by 110 · see citing works
Gray Matter Abnormalities in Social Anxiety Disorder: Primary, Replication, and Specificity Studies
Talati, A., Pantazatos, S. P., Schneier, F. R., Weissman, M. M., & Joy Hirsch, C. (2012) · Biological Psychiatry · doi:10.1016/j.biopsych.2012.05.022 · Cited by 108 · see citing works
Molendijk, M. L., Bus, B. A. A., Spinhoven, P., Kaimatzoglou, A., Voshaar, R. C. O., Penninx, B. W. J. H., van IJzendoorn, M. H., & Elzinga, B. M. (2012) · American Journal of Medical Genetics Part B Neuropsychiatric Genetics · doi:10.1002/ajmg.b.32078 · Cited by 107 · see citing works
Kok, M. G. M., de Ronde, M. W. J., Moerland, P. D., Ruijter, J. M., Creemers, E. E., & Pinto-Sietsma, S. J. (2018) · Biomolecular Detection and Quantification · doi:10.1016/j.bdq.2017.11.002 · Cited by 105 · see citing works
Clinically meaningful biomarkers for psychosis: A systematic and quantitative review
Prata, D. P., Mechelli, A., & Kapur, S. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.05.010 · Cited by 100 · see citing works
Statins and Multiple Noncardiovascular Outcomes
He, Y., Li, X., Gašević, D., Brunt, E., McLachlan, F., Millenson, M. E., Timofeeva, M. N., Ioannidis, J. P. A., Campbell, H. J., & Τheodoratou, E. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0808 · Cited by 96 · see citing works
Self-citationTsilidis, K. K., Papatheodorou, S., Εvangelou, Ε., & Ioannidis, J. P. A. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs437 · Cited by 93 · see citing works
Self-citationTemperament and brain networks of attention
Posner, M. I., & Rothbart, M. K. (2018) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2017.0254 · Cited by 87 · see citing works
Candidate and non-candidate genes in behavior genetics
Jonathan Flint, C., & Munafo, M. R. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.07.005 · Cited by 85 · see citing works
Mathur, M. B., & VanderWeele, T. J. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1464 · Cited by 88 · see citing works
Detecting publication selection bias through excess statistical significance
Stanley, T. D., Doucouliagos, H., Ioannidis, J. P. A., & Carter, E. C. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1512 · Cited by 79 · see citing works
Self-citationJednoróg, K., Marchewka, A., Altarelli, I., Lopez, A. K. M., van Ermingen‐Marbach, M., Grande, M., Grabowska, A., Heim, S., & Ramus, F. (2015) · Human Brain Mapping · doi:10.1002/hbm.22734 · Cited by 82 · see citing works
The impact of sample size on the reproducibility of voxel-based lesion-deficit mappings
Lorca‐Puls, D. L., Gajardo‐Vidal, A., White, J., Seghier, M. L., Leff, A. P., Green, D. W., Crinion, J., Ludersdorfer, P., Hope, T. M. H., Bowman, H., et al. (2018) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2018.03.014 · Cited by 82 · see citing works
Patrick, C. J., Iacono, W. G., & Venables, N. C. (2019) · Psychological Assessment · doi:10.1037/pas0000713 · Cited by 80 · see citing works
Umbrella Reviews: What They Are and Why We Need Them
Papatheodorou, S., & Εvangelou, Ε. (2021) · Methods in molecular biology · doi:10.1007/978-1-0716-1566-9_8 · Cited by 79 · see citing works
McGrath, L. M., & Stoodley, C. J. (2019) · Journal of Neurodevelopmental Disorders · doi:10.1186/s11689-019-9287-8 · Cited by 78 · see citing works
Voxel-based morphometry multi-center mega-analysis of brain structure in social anxiety disorder
Bas‐Hoogendam, J. M., van Steenbergen, H., Pannekoek, J. N., Fouché, J., Löchner, C., Hattingh, C. J., Cremers, H. R., Furmark, T., Månsson, K. N. T., Frick, A., et al. (2017) · NeuroImage Clinical · doi:10.1016/j.nicl.2017.08.001 · Cited by 75 · see citing works
Bernanke, J., Luna, A., Chang, L., Bruno, E., Dworkin, J., & Posner, J. (2022) · The Lancet Psychiatry · doi:10.1016/s2215-0366(21)00505-8 · Cited by 75 · see citing works
Association of Hormonal Contraceptive Use With Adverse Health Outcomes
Brabaharan, S., Veettil, S. K., Kaiser, J. E., Rao, V. R. R., Wattanayingcharoenchai, R., Maharajan, M. K., Insin, P., Talungchit, P., Anothaisintawee, T., Thakkinstian, A., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.43730 · Cited by 77 · see citing works
The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates
Nuijten, M., van Assen, M., Veldkamp, C. L. S., & Wicherts, J. (2015) · Review of General Psychology · doi:10.1037/gpr0000034 · Cited by 60 · see citing works
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78 · see citing works
Positive results receive more citations, but only in some disciplines
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-012-0757-y · Cited by 67 · see citing works
Basing psychiatric classification on scientific foundation: Problems and prospects
Uher, R., & Rutter, M. L. (2012) · International Review of Psychiatry · doi:10.3109/09540261.2012.721346 · Cited by 67 · see citing works
The neuroanatomy of psychotic diathesis: A meta-analytic review
Palaniyappan, L. K., Balain, V., & Liddle, P. F. (2012) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2012.06.007 · Cited by 64 · see citing works
Neuroimaging distinction between neurological and psychiatric disorders
Crossley, N. A., Scott, J., Ellison-Wright, I., & Mechelli, A. (2015) · The British Journal of Psychiatry · doi:10.1192/bjp.bp.114.154393 · Cited by 64 · see citing works
Bartoš, F., Maier, M., Wagenmakers, E., Nippold, F., Doucouliagos, H., Ioannidis, J., Otte, W., Sladekova, M., Deresssa, T. K., Bruns, S. B., et al. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1703 · Cited by 59 · see citing works
Self-citationthe EU-AIMS LEAP group, Moessnang, C., Baumeister, S., Tillmann, J., Goyard, D., Charman, T., Ambrosino, S., Baron‐Cohen, S., Beckmann, C. F., Bölte, S., et al. (2020) · Molecular Autism · doi:10.1186/s13229-020-0317-x · Cited by 65 · see citing works
Bite-size science and its undesired side effects
Bertamini, M., & Munafò, M. R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691611429353 · Cited by 62 · see citing works
Sex is a defining feature of neuroimaging phenotypes in major brain disorders
Salminen, L. E., Tubi, M., Bright, J. K., Thomopoulos, S., Wieand, A., & Thompson, P. (2020) · Human Brain Mapping · doi:10.1002/hbm.25438 · Cited by 58 · see citing works
Rapid eye movement sleep behavior disorder in Parkinson's disease: Magnetic resonance imaging study
Ford, A. H., Duncan, G. W., Firbank, M., Yarnall, A. J., Khoo, T. K., Burn, D. J., & O'Brien, J. T. (2013) · Movement Disorders · doi:10.1002/mds.25367 · Cited by 59 · see citing works
Markozannes, G., Aretouli, E., Rintou, E., Dragioti, E., Damigos, D., Ntzani, E., Εvangelou, Ε., & Tsilidis, K. K. (2017) · BMC Psychology · doi:10.1186/s40359-017-0200-5 · Cited by 58 · see citing works
Klauser, P., Xin, L., Fournier, M., Griffa, A., Cleusix, M., Jenni, R., Cuénod, M., Rolf Gruetter, A., Hagmann, P., Conus, P., et al. (2018) · Translational Psychiatry · doi:10.1038/s41398-018-0266-8 · Cited by 57 · see citing works
Fusar-Poli, P., Radua, J., Frascarelli, M., Mechelli, A., Borgwardt, S., Di Fabio, F., Biondi, M., Ioannidis, J. P. A., & David, S. P. (2014) · Human Brain Mapping · doi:10.1002/hbm.22384 · Cited by 56 · see citing works
Self-citationIs there an excess of significant findings in published studies of psychotherapy for depression?
Flint, J., Cuijpers, P. J., Horder, J., Koole, S. L., & Munafo, M. R. (2014) · Psychological Medicine · doi:10.1017/s0033291714001421 · Cited by 56 · see citing works
Neural underpinnings of individual differences in emotion regulation: A systematic review
Morawetz, C., & Basten, U. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105727 · Cited by 60 · see citing works
Wang, L., Zhang, R., Yu, L., Xiao, J., Zhou, X., Li, X., Song, P., & Li, X. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.690219 · Cited by 54 · see citing works
Impact of Reporting Bias in Network Meta-Analysis of Antidepressant Placebo-Controlled Trials
Trinquart, L., Abbé, A., & Ravaud, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0035219 · Cited by 54 · see citing works
Klauser, P., Zhou, J. H., Lim, J. K. W., Poh, J. S., Zheng, H., Tng, H. Y., Krishnan, R. K. R., Lee, J., Keefe, R. S. E., Adcock, R. A., et al. (2015) · Schizophrenia Bulletin · doi:10.1093/schbul/sbv012 · Cited by 54 · see citing works
Bias in Peripheral Depression Biomarkers
Carvalho, A. F., Köhler, C. A., Brunoni, A. R., Miskowiak, K. W., Herrmann, N., Lanctôt, K. L., Hyphantis, T. N., Quevedo, J., Fernandes, B. S., & Berk, M. (2016) · Psychotherapy and Psychosomatics · doi:10.1159/000441457 · Cited by 52 · see citing works
Voxel-based morphometry: current perspectives
Scarpazza, C., & Simone, M. S. D. (2016) · Neuroscience and Neuroeconomics · doi:10.2147/nan.s66439 · Cited by 51 · see citing works
Forty years of structural imaging in psychosis: promises and truth
Fusar‐Poli, P., & Meyer‐Lindenberg, A. (2016) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.12619 · Cited by 50 · see citing works
de Goede, J., Reeuwijk, K. G., Braun, K. P. J., le Cessie, S., Durston, S., Engels, R. C. M. E., Goudriaan, A. E., Moons, K. G. M., Vollebergh, W., Vries, T. J. D., et al. (2020) · Advances in Nutrition · doi:10.1093/advances/nmaa170 · Cited by 50 · see citing works
Clinical Psychological Science
Wright, A. G. C., & Hallquist, M. N. (2020) · Cambridge University Press eBooks · doi:10.1017/9781316995808.001 · Cited by 47 · see citing works
Genetic psychophysiology: Advances, problems, and future directions
Anokhin, A. P. (2014) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2014.04.003 · Cited by 47 · see citing works
- Authors
- Ioannidis, J. P. A.
- Affiliations
- University of Ioannina (GR)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- Archives of General Psychiatry
- Publisher
- American Medical Association (AMA)
- Published
- 2011-08-01
- Volume
- 68
- Issue
- 8
- Pages
- 773
- ISSN
- 0003-990X (ISSN-L)
- OpenAlex
- W2169662981
- Cited by
- 299
- References
- 48
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 8 · 2023: 5 · 2022: 5 · 2021: 25 · 2020: 55 · 2019: 17 · 2018: 23 · 2017: 19 · 2016: 17 · 2015: 27 · 2014: 26 · 2013: 19 · 2012: 36
- FWCI
- 29.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Archives of General Psychiatry (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- publication bias, selective outcome reporting, meta-analysis, mental illness, statistical power, standardized effect size, neuroimaging
- MeSH terms
- Mental Disorders, Brain, Humans, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
