The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis.
Open accessAbstract
ὠφελέειν, ἢ μὴ βλάπτειν (Primum non nocere) – Hιppocrates’ principle should still guide daily medical prescribing. Therefore, assessing evidence of psychopharmacologic agents’ safety and harms is essential. Randomised controlled trials (RCTs) and observational studies may provide complementary information about harms of psychopharmacologic medications from both experimental and real-world settings. It is considered that RCTs provide a better control of confounding variables, while observational studies provide evidence from larger samples, longer follow-ups, in more representative samples, which may be more reflective of real-life clinical scenarios. However, this may not always hold true. Moreover, in observational studies, safety data are poorly or inconsistently reported, precluding reliable quantitative synthesis in meta-analyses. Beyond individual studies, meta-analyses, which represent the highest level of ‘evidence’, can be misleading, redundant and of low methodological quality. Overlapping meta-analyses sometimes even reach different conclusions on the same topic. Meta-analyses should be assessed systematically. Descriptive reviews of reviews can be poorly informative. Conversely, ‘umbrella reviews’ can use a quantitative approach to grade evidence. In this editorial, we present the main factors involved in the assessment of psychopharmacologic agents’ harms from individual studies, meta-analyses and umbrella reviews. Study design features, sample size, number of the events of interest, summary effect sizes, p-values, heterogeneity, 95% prediction intervals, confounding factor adjustment and tests of bias (e.g., small-study effects and excess significance) can be combined with other assessment tools, such as AMSTAR and GRADE to create a framework for assessing the credibility of evidence.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The grading system can only warn about potential systematic biases and cannot establish their nature or extent.
“Fifth, we used a grading system that can provide only warnings of the potential presence of systematic biases but cannot provide evidence of the nature and extent of these biases, 16,32,33 just as umbrella reviews cannot supply any comparative ranking as in network meta-analyses.”
From Association of Antidepressant Use With Adverse Health Outcomes: A Systematic Umbrella Review (2019) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Statistical power analysis for the behavioral sciences
Cohen, J. (1990) · Computers Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,406 · see citing works
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Quantifying heterogeneity in a meta‐analysis
Higgins, J. P. T., & Thompson, S. G. (2002) · Statistics in Medicine · doi:10.1002/sim.1186 · Cited by 36,763 · see citing works
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636 · see citing works
Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rucker, G., Harbord, R. M., Schmid, C. H., et al. (2011) · BMJ · doi:10.1136/bmj.d4002 · Cited by 7,709 · see citing works
Self-citationThe ASA Statement on <i>p</i> -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · see citing works
Cipriani, A., Furukawa, T. A., Salanti, G., Chaimani, A., Atkinson, L. Z., Ogawa, Y., Leucht, S., Ruhe, H. G., Turner, E. H., Higgins, J. P. T., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(17)32802-7 · Cited by 3,541 · cited 2× · see citing works
“Introduction Psychopharmacologic agents may differ in both efficacy and safety (Leucht et al., 2013; Solmi et al., 2017; Cipriani et al., 2018).”
Self-citationA manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
Self-citationNew Effect Size Rules of Thumb
Sawilowsky, S. S. (2009) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1257035100 · Cited by 3,254 · see citing works
Leucht, S., Cipriani, A., Spineli, L., Mavridis, D., Örey, D., Richter, F., Samara, M., Barbui, C., Engel, R. R., Geddes, J. R., et al. (2013) · The Lancet · doi:10.1016/s0140-6736(13)60733-3 · Cited by 2,561 · see citing works
Plea for routinely presenting prediction intervals in meta-analysis.
IntHout, J., Ioannidis, J. P. A., Rovers, M. M., & Goeman, J. J. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010247 · Cited by 2,264 · see citing works
Self-citationThe case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · see citing works
Self-citationShea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.009 · Cited by 1,755 · see citing works
Better Reporting of Harms in Randomized Trials: An Extension of the CONSORT Statement
Ioannidis, J. P. A., Evans, S. J. W., Gøtzsche, P. C., O'Neill, R. T., Altman, D. G., Schulz, K., Moher, D., & for the CONSORT Group* (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-10-200411160-00009 · Cited by 1,593 · see citing works
Self-citationThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · see citing works
Self-citationUncertainty in heterogeneity estimates in meta-analyses
Ioannidis, J. P. A., Patsopoulos, N. A., & Evangelou, E. (2007) · BMJ · doi:10.1136/bmj.39343.408449.80 · Cited by 1,254 · see citing works
Self-citationTheodoratou, E., Tzoulaki, I., Zgaga, L., & Ioannidis, J. P. A. (2014) · BMJ · doi:10.1136/bmj.g2035 · Cited by 956 · see citing works
Self-citationAn exploratory test for an excess of significant findings
Ioannidis, J. P., & Trikalinos, T. A. (2007) · Clinical Trials · doi:10.1177/1740774507079441 · Cited by 751 · see citing works
Self-citationIoannidis, J. P. A. (2009) · Canadian Medical Association Journal · doi:10.1503/cmaj.081086 · Cited by 716 · see citing works
Self-citationThe Proposal to Lower P Value Thresholds to .005
Ioannidis, J. P. A. (2018) · JAMA · doi:10.1001/jama.2018.1536 · Cited by 655 · see citing works
Self-citationPRISMA harms checklist: improving harms reporting in systematic reviews
Zorzela, L., Loke, Y. K., Ioannidis, J. P., Golder, S., Santaguida, P., Altman, D. G., Moher, D., Vohra, S., & PRISMA harms group (2016) · BMJ · doi:10.1136/bmj.i157 · Cited by 557 · see citing works
Self-citationDietary fiber and health outcomes: an umbrella review of systematic reviews and meta-analyses
Veronese, N., Solmi, M., Caruso, M. G., Giannelli, G., Osella, A. R., Evangelou, E., Maggi, S., Fontana, L., Stubbs, B., & Tzoulaki, I. (2018) · The American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqx082 · Cited by 536 · see citing works
Self-citationSolmi, M., Murru, A., Pacchiarotti, I., Undurraga, J., Veronese, N., Fornaro, M., Stubbs, B., Monaco, F., Vieta, E., Seeman, M., et al. (2017) · Therapeutics and Clinical Risk Management · doi:10.2147/tcrm.s117321 · Cited by 438 · see citing works
Self-citationEnvironmental risk factors and Parkinson's disease: An umbrella review of meta-analyses
Bellou, V., Belbasis, L., Tzoulaki, I., Εvangelou, Ε., & Ioannidis, J. P. A. (2015) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2015.12.008 · Cited by 429 · see citing works
Self-citationEnhancing reproducibility for computational methods
Stodden, V., Mcnutt, M. K., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P. A., & Taufer, M. (2016) · Science · doi:10.1126/science.aah6168 · Cited by 414 · see citing works
Self-citationLi, X., Meng, X., Timofeeva, M. N., Tzoulaki, I., Tsilidis, K. K., Ioannidis, J. P. A., Campbell, H. J., & Τheodoratou, E. (2017) · BMJ · doi:10.1136/bmj.j2376 · Cited by 402 · see citing works
Self-citationReproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310 · see citing works
Self-citationWhen Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment
Szucs, D., & Ioannidis, J. P. A. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00390 · Cited by 298 · see citing works
Self-citationPrespecified Falsification End Points
Prasad, V. K., & Jena, A. B. (2013) · JAMA · doi:10.1001/jama.2012.96867 · Cited by 284 · see citing works
Adverse Events in Randomized Trials
Ioannidis, J. P. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.313 · Cited by 243 · see citing works
Self-citationComparison of evidence on harms of medical interventions in randomized and nonrandomized studies
Papanikolaou, P. N. (2006) · Canadian Medical Association Journal · doi:10.1503/cmaj.050873 · Cited by 229 · see citing works
Naudet, F., Sakarovitch, C., Janiaud, P., Cristea, I., Fanelli, D., Moher, D., & Ioannidis, J. P. A. (2018) · BMJ · doi:10.1136/bmj.k400 · Cited by 227 · see citing works
Self-citationCorrell, C. U., Rubio, J. M., Inczedy-Farkas, G., Birnbaum, M. L., Kane, J. M., & Leucht, S. (2017) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2017.0624 · Cited by 214 · see citing works
Self-citationIoannidis, J. P. A. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2017-097621 · Cited by 189 · see citing works
Self-citationReanalyses of Randomized Clinical Trial Data
Ebrahim, S., Sohani, Z. N., Montoya, L., Agarwal, A., Thorlund, K., Mills, E. J., & Ioannidis, J. P. A. (2014) · JAMA · doi:10.1001/jama.2014.9646 · Cited by 186 · see citing works
Self-citationPollock, M., Fernandes, R. M., & Hartling, L. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0325-5 · Cited by 107 · see citing works
Belbasis, L., Bellou, V., & Εvangelou, Ε. (2016) · Neuroepidemiology · doi:10.1159/000443146 · Cited by 103 · see citing works
Availability of large-scale evidence on specific harms from systematic reviews of randomized trials.
Papanikolaou, P. N., & Ioannidis, J. P. A. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.04.026 · Cited by 79 · see citing works
Self-citationCan quality of clinical trials and meta-analyses be quantified?
Ioannidis, J. P. A., & Lau, J. (1998) · The Lancet · doi:10.1016/s0140-6736(98)22034-4 · Cited by 74 · see citing works
Self-citationPlacebo - Efficacy and Adverse Effects in Controlled Clinical Trials
Weihrauch, T. R., & Gauler, T. C. (2011) · Arzneimittelforschung · doi:10.1055/s-0031-1300432 · Cited by 51 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Barbui, C., Purgato, M., Abdulmalik, J., Acarturk, C., Eaton, J., Gastaldon, C., Gureje, O., Hanlon, C., Jordans, M., Lund, C., et al. (2020) · The Lancet Psychiatry · doi:10.1016/s2215-0366(19)30511-5 · Cited by 349 · see citing works
Solmi, M., Fornaro, M., Ostinelli, E. G., Zangani, C., Croatto, G., Monaco, F., Krinitski, D., Fusar‐Poli, P., & Correll, C. U. (2020) · World Psychiatry · doi:10.1002/wps.20765 · Cited by 324 · see citing works
Self-citationAssociation of Antidepressant Use With Adverse Health Outcomes: A Systematic Umbrella Review
Dragioti, E., Solmi, M., Favaro, A., Fusar-Poli, P., Dazzan, P., Thompson, T., Stubbs, B., Firth, J., Fornaro, M., Tsartsalis, D., et al. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.2859 · Cited by 215 · see citing works
Self-citationUmbrella reviews: what they are and why we need them
Papatheodorou, S. (2019) · European Journal of Epidemiology · doi:10.1007/s10654-019-00505-6 · Cited by 201 · see citing works
Barbui, C., Purgato, M., Abdulmalik, J., Caldas-de-Almeida, J. M., Eaton, J., Gureje, O., Hanlon, C., Nosè, M., Ostuzzi, G., Saraceno, B., et al. (2021) · The British Journal of Psychiatry · doi:10.1192/bjp.2020.144 · Cited by 148 · see citing works
- Authors
- Solmi, M., Correll, C. U., Carvalho, A. F., & Ioannidis, J. P. A.
- Affiliations
- University of Padua (IT)
- Northwell Health (US)
- Centre for Addiction and Mental Health (CA)
- Stanford Medicine (US)
- Corresponding author
- Solmi, M.
- Venue
- Epidemiology and Psychiatric Sciences
- Publisher
- Cambridge University Press (CUP)
- Published
- 2018-07-16
- Volume
- 27
- Issue
- 6
- Pages
- 537-542
- ISSN
- 2045-7960 (ISSN-L)
- 2045-7979
- PMC
- PMC6999005
- OpenAlex
- W2884467301
- Cited by
- 44
- References
- 40
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 4 · 2023: 10 · 2022: 5 · 2021: 9 · 2020: 6 · 2019: 4 · 2018: 1
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Epidemiology and Psychiatric Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Padua Research Archive (University of Padova) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- psychotropic drugs, drug safety, meta-analysis, umbrella review, observational studies, randomized controlled trials, AMSTAR, GRADE, confounding, heterogeneity, small-study effects, non-maleficence
- MeSH terms
- Humans, Psychopharmacology, Psychotropic Drugs, Research Design, Meta-Analysis as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
