Replication and contradiction of highly cited research papers in psychiatry: 10-year follow-up
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Background: Contradictions and initial overestimates are not unusual among highly cited studies. However, this issue has not been researched in psychiatry. Aims: To assess how highly cited studies in psychiatry are replicated by subsequent studies. Method: We selected highly cited studies claiming effective psychiatric treatments in the years 2000 through 2002. For each of these studies we searched for subsequent studies with a better-controlled design, or with a similar design but a larger sample. Results: Among 83 articles recommending effective interventions, 40 had not been subject to any attempt at replication, 16 were contradicted, 11 were found to have substantially smaller effects and only 16 were replicated. The standardised mean differences of the initial studies were overestimated by 132%. Studies with a total sample size of 100 or more tended to produce replicable results. Conclusions: Caution is needed when a study with a small sample size reports a large effect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wood, L., Egger, M., Gluud, L. L., Schulz, K. F., Jüni, P., Altman, D. G., Gluud, C., Martin, R. M., Wood, A. J. G., & Sterne, J. A. C. (2008) · BMJ · doi:10.1136/bmj.39465.451748.ad · Cited by 2,538
Why Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495
Contradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465
Nonsteroidal Antiinflammatory Drugs and the Risk of Alzheimer's Disease
in 't Veld, B. A., Ruitenberg, A., Hofman, A., Launer, L. J., van Duijn, C. M., Stijnen, T., Breteler, M. M. B., & Stricker, B. H. C. (2001) · New England Journal of Medicine · doi:10.1056/nejmoa010178 · Cited by 1,196
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Marshall, M., Lockwood, A., Bradley, C., Adams, C., Joy, C., & Fenton, M. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.3.249 · Cited by 870
Leucht, S., Hierl, S., Kissling, W., Dold, M., & Davis, J. M. (2012) · British Journal of Psychiatry · doi:10.1192/bjp.bp.111.096594 · Cited by 451
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451
Empirical Evaluation of Very Large Treatment Effects of Medical Interventions
Pereira, T. V., Horwitz, R. I., & Ioannidis, J. P. A. (2012) · JAMA · doi:10.1001/jama.2012.13444 · Cited by 257
Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time
Trikalinos, T. A., Churchill, R., Ferri, M., Leucht, S., Tuunainen, A., Wahlbeck, K., & Ioannidis, J. P. A. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.02.018 · Cited by 177
Cited by
O. B. O. T. C.-S. G., Grant, S., Mayo-Wilson, E., Montgomery, P., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2735-z · Cited by 348
on behalf of the CONSORT-SPI Group, Montgomery, P., Grant, S., Mayo-Wilson, E., Macdonald, G., Michie, S., Hopewell, S., & Moher, D. (2018) · Trials · doi:10.1186/s13063-018-2733-1 · Cited by 327
A Scoping Review of Machine Learning in Psychotherapy Research
Aafjes-van Doorn, K., Kamsteeg, C., Bate, J., & Aafjes, M. (2021) · Psychotherapy Research · doi:10.1080/10503307.2020.1808729 · Cited by 179
The influence of ketamine on drug discovery in depression
Kraus, C., Wasserman, D., Henter, I. D., Acevedo-Diaz, E., Kadriu, B., & Zarate, C. A. (2019) · Drug Discovery Today · doi:10.1016/j.drudis.2019.07.007 · Cited by 75
“For example, a study examining clinical trial results published between 2000 and 2002 in the five-highest-ranked psychiatry journals at the time found that, although highly cited, 48% of the initial findings were not replicated until 2013 [26].”
Poor replication validity of biomedical association studies reported by newspapers
Dumas-Mallet, E., Smith, A., Boraud, T., & Gonon, F. (2017) · PLOS ONE · doi:10.1371/journal.pone.0172650 · Cited by 76
“Likewise, highly cited early studies investigating the effectiveness of psychiatric treatments are often contradicted by subsequent studies [16].”
ResultMedeiros, G. C., Gould, T. D., Prueitt, W. L., Nanavati, J., Grunebaum, M. F., Farber, N. B., Singh, B., Selvaraj, S., Machado-Vieira, R., Achtyes, E. D., et al. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01652-1 · Cited by 75
“First, most sample sizes have been relatively small (the median sample size included in this study was 33 individuals), which are likely to lead to imprecise effect sizes that replicate poorly [28, 124].”
BackgroundWeiser, M., Levi, L., Zamora, D., Biegon, A., SanGiovanni, J. P., Davidson, M., Burshtein, S., Gonen, I., Radu, P., Slobozean Pavalache, K., et al. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1842 · Cited by 42
Control interventions in randomised trials among people with mental health disorders
Faltinsen, E., Todorovac, A., Staxen Bruun, L., Hróbjartsson, A., Gluud, C., Kongerslev, M. T., Simonsen, E., & Storebø, O. J. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000050.pub2 · Cited by 40
Metadata
- Authors
- Tajika, A., Ogawa, Y., Takeshima, N., Hayasaka, Y., & Furukawa, T. A.
- Venue
- British Journal of Psychiatry
- Publisher
- Royal College of Psychiatrists
- Published
- 2015-10
- Volume
- 207
- Issue
- 4
- Pages
- 357-362
- ISSN
- 0007-1250 (ISSN-L)
- 1472-1465
- OpenAlex
- W2132169592
- Cited by
- 61
- References
- 10
- FWCI
- 4.78
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.