Quality of reporting in systematic reviews of adverse events: systematic review
Open accessAbstract
OBJECTIVES: To examine the quality of reporting of harms in systematic reviews, and to determine the need for a reporting guideline specific for reviews of harms. DESIGN: Systematic review. DATA SOURCES: Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effects (DARE). REVIEW METHODS: Databases were searched for systematic reviews having an adverse event as the main outcome, published from January 2008 to April 2011. Adverse events included an adverse reaction, harms, or complications associated with any healthcare intervention. Articles with a primary aim to investigate the complete safety profile of an intervention were also included. We developed a list of 37 items to measure the quality of reporting on harms in each review; data were collected as dichotomous outcomes ("yes" or "no" for each item). RESULTS: Of 4644 reviews identified, 309 were systematic reviews or meta-analyses primarily assessing harms (13 from CDSR; 296 from DARE). Despite a short time interval, the comparison between the years of 2008 and 2010-11 showed no difference on the quality of reporting over time (P=0.079). Titles in fewer than half the reviews (proportion of reviews 0.46 (95% confidence interval 0.40 to 0.52)) did not mention any harm related terms. Almost one third of DARE reviews (0.26 (0.22 to 0.31)) did not clearly define the adverse events reviewed, nor did they specify the study designs selected for inclusion in their methods section. Almost half of reviews (n=170) did not consider patient risk factors or length of follow-up when reviewing harms of an intervention. Of 67 reviews of complications related to surgery or other procedures, only four (0.05 (0.01 to 0.14)) reported professional qualifications of the individuals involved. The overall, unweighted, proportion of reviews with good reporting was 0.56 (0.55 to 0.57); corresponding proportions were 0.55 (0.53 to 0.57) in 2008, 0.55 (0.54 to 0.57) in 2009, and 0.57 (0.55 to 0.58) in 2010-11. CONCLUSION: Systematic reviews compound the poor reporting of harms data in primary studies by failing to report on harms or doing so inadequately. Improving reporting of adverse events in systematic reviews is an important step towards a balanced assessment of an intervention.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Meditation-based clinical trials are not required to report adverse occurrences outside serious adverse events, which is likely to lead to under-reporting of MAEs.
“they are not required to report other types of adverse occurrences that fall outside of this list, which is likely to lead to an under-reporting of MAEs -a general problem found with randomized controlled trials (23).”
From Adverse events in meditation practices and meditation‐based therapies: a systematic review (2020) · cites this paper as [23]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859 · see citing works
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048 · see citing works
The environment and disease: association or causation?
Hill, A. B. (2015) · Journal of the Royal Society of Medicine · doi:10.1177/0141076814562718 · Cited by 7,302 · see citing works
The Environment and Disease: Association or Causation?
Hill, A. B. (1965) · Proceedings of the Royal Society of Medicine · doi:10.1177/003591576505800503 · Cited by 6,974 · see citing works
Cochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670 · see citing works
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476 · see citing works
Consort 2010 statement: extension to cluster randomised trials
Campbell, M. K., Piaggio, G., Elbourne, D. R., Altman, D. G., & for the CONSORT Group (2012) · BMJ · doi:10.1136/bmj.e5661 · Cited by 3,510 · see citing works
Piaggio, G., Elbourne, D. R., Pocock, S. J., Evans, S. J. W., Altman, D. G., & CONSORT Group, F. T. (2012) · JAMA · doi:10.1001/jama.2012.87802 · Cited by 2,428 · see citing works
Improving the reporting of pragmatic trials: An extension of the CONSORT statement
Zwarenstein, M., Treweek, S., Gagnier, J. J., Altman, D. G., Tunis, S., Haynes, B., Oxman, A. D., Moher, D., & groups, F. T. C. A. P. T. I. H. (. (2008) · BMJ · doi:10.1136/bmj.a2390 · Cited by 1,813 · see citing works
Better Reporting of Harms in Randomized Trials: An Extension of the CONSORT Statement
Ioannidis, J. P. A., Evans, S. J. W., Gøtzsche, P. C., O'Neill, R. T., Altman, D. G., Schulz, K., Moher, D., & for the CONSORT Group* (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-10-200411160-00009 · Cited by 1,593 · see citing works
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437 · see citing works
MacPherson, H., Altman, D. G., Hammerschlag, R., Li, Y., Tai-xiang, W., White, A. R., Moher, D., & on behalf of the STRICTA Revision Group (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000261 · Cited by 1,409 · see citing works
Reporting of Noninferiority and Equivalence Randomized Trials: An Extension of the CONSORT Statement
Piaggio, G., Elbourne, D. R., Altman, D. G., Pocock, S. J., Evans, S. J. W., & CONSORT Group, F. T. (2006) · JAMA · doi:10.1001/jama.295.10.1152 · Cited by 1,153 · see citing works
Epidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · see citing works
Reporting Randomized, Controlled Trials of Herbal Interventions: An Elaborated CONSORT Statement
Gagnier, J. J., Boon, H., Rochon, P., Moher, D., Barnes, J., Bombardier, C., & for the CONSORT Group* (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-5-200603070-00013 · Cited by 608 · see citing works
Boutron, I., Moher, D., Altman, D. G., Schulz, K. F., Ravaud, P., & for the CONSORT Group* (2008) · Annals of Internal Medicine · doi:10.7326/0003-4819-148-4-200802190-00008-w1 · Cited by 523 · see citing works
Uncertain Effects of Rosiglitazone on the Risk for Myocardial Infarction and Cardiovascular Death
Diamond, G. A., Bax, L., & Kaul, S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00182 · Cited by 288 · see citing works
Reporting of Safety Results in Published Reports of Randomized Controlled Trials
(2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.306 · Cited by 287 · see citing works
Systematic reviews of adverse effects: framework for a structured approach
Loke, Y. K., Price, D., Herxheimer, A., & the Cochrane Adverse Effects Methods Group (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-32 · Cited by 253 · see citing works
Self-citationAdverse Events in Randomized Trials
Ioannidis, J. P. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.313 · Cited by 243 · see citing works
Drug safety assessment in clinical trials: methodological challenges and opportunities
Singh, S., & Loke, Y. K. (2012) · Trials · doi:10.1186/1745-6215-13-138 · Cited by 228 · see citing works
Self-citationMoseley, A. M., Elkins, M. R., Herbert, R. D., Maher, C. G., & Sherrington, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.018 · Cited by 202 · see citing works
Chou, R., Aronson, N., Atkins, D. C., Ismaila, A., Santaguida, P. L., Smith, D. H., Whitlock, E. P., Wilt, T. J., & Moher, D. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.007 · Cited by 195 · see citing works
Challenges in Systematic Reviews That Assess Treatment Harms
Chou, R., & Helfand, M. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-12_part_2-200506211-00009 · Cited by 145 · see citing works
Reporting of adverse drug reactions in randomised controlled trials – a systematic survey
Loke, Y. K., & Derry, S. (2001) · BMC Clinical Pharmacology · doi:10.1186/1472-6904-1-3 · Cited by 128 · see citing works
Self-citationDerry, S., Loke, Y. K., & Aronson, J. K. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-7 · Cited by 110 · see citing works
Self-citationShuster, J. J., Jones, L. S., & Salmon, D. A. (2007) · Statistics in Medicine · doi:10.1002/sim.3060 · Cited by 105 · see citing works
Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects
Golder, S., Loke, Y. K., & McIntosh, H. M. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.06.005 · Cited by 99 · see citing works
Self-citationAssessment of therapeutic safety in systematic reviews: literature review
Ernst, E., & Pittler, M. H. (2001) · BMJ · doi:10.1136/bmj.323.7312.546 · Cited by 94 · see citing works
Availability of large-scale evidence on specific harms from systematic reviews of randomized trials.
Papanikolaou, P. N., & Ioannidis, J. P. A. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.04.026 · Cited by 79 · see citing works
Room for improvement? A survey of the methods used in systematic reviews of adverse effects
Golder, S., Loke, Y. K., & McIntosh, H. M. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-3 · Cited by 73 · see citing works
Self-citationGolder, S., & Loke, Y. K. (2008) · British Journal of Clinical Pharmacology · doi:10.1111/j.1365-2125.2008.03272.x · Cited by 71 · see citing works
Self-citationAssessing harmful effects in systematic Reviews
McIntosh, H. M., Woolacott, N. F., & Bagnall, A. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-19 · Cited by 69 · see citing works
Reporting of adverse events in systematic reviews can be improved: survey results
Hopewell, S., Wolfenden, L., & Clarke, M. J. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.10.005 · Cited by 64 · see citing works
Cornelius, V. R., Perrio, M. J., Shakir, S., & Smith, L. A. (2009) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1844 · Cited by 54 · see citing works
Adverse drug reactions: keeping up to date
Aronson, J. K., Derry, S., & Loke, Y. K. (2002) · Fundamental and Clinical Pharmacology · doi:10.1046/j.1472-8206.2002.00066.x · Cited by 50 · see citing works
Self-citationLoke, Y. K., Golder, S. P., & Vandenbroucke, J. P. (2011) · Therapeutic Advances in Drug Safety · doi:10.1177/2042098611401129 · Cited by 49 · see citing works
Self-citationHernández, A. V., Walker, E., Ioannidis, J. P. A., & Kattan, M. W. (2008) · American Heart Journal · doi:10.1016/j.ahj.2008.03.002 · Cited by 39 · see citing works
Warren, F. C., Abrams, K. R., Golder, S., & Sutton, A. J. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-64 · Cited by 34 · see citing works
Self-citationPilkington, K., & Boshnakova, A. (2011) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2011.10.002 · Cited by 34 · see citing works
Self-citationGolder, S., Loke, Y. K., & Zorzela, L. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.013 · Cited by 34 · see citing works
Self-citationFebrile Pancytopenia Associated With Clopidogrel
Andrès, E., Perrin, A.-E., Alt, M., et al. (2001) · Archives of Internal Medicine · doi:10.1001/archinte.161.1.125 · Cited by 32 · see citing works
Identifying systematic reviews of the adverse effects of health care interventions
Golder, S., McIntosh, H. M., & Loke, Y. K. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-22 · Cited by 26 · see citing works
Self-citationSome concerns about adverse event reporting in randomized clinical trials.
Yazıcı, Y. (2008) · PubMed · Cited by 26 · see citing works
Meta-analyses of safety data: a comparison of exact versus asymptotic methods
Vandermeer, B., Bialy, L. M., Hooton, N., Hartling, L., Klassen, T. P., Johnston, B. C., & Wiebe, N. (2008) · Statistical Methods in Medical Research · doi:10.1177/0962280208092559 · Cited by 24 · see citing works
Self-citationHammad, T. A., Neyarapally, G. A., Pinheiro, S. P., Iyasu, S., Rochester, G., & Pan, G. J. D. (2013) · Clinical Trials · doi:10.1177/1740774513479467 · Cited by 22 · see citing works
Systematic Reviews Neglect Safety Issues
Ernst, E., & Pittler, M. H. (2001) · Archives of Internal Medicine · doi:10.1001/archinte.161.1.125-a · Cited by 15 · see citing works
Completeness of Safety Reporting in Randomized Trials: An Evaluation of 7 Medical Areas
Ioannidis, J. P. A., & Lau, J. (2001) · JAMA · doi:10.1001/jama.285.4.437 · Cited by 3 · see citing works
Detecting and describing heterogeneity in meta-analysis
Hardy, R. J., & Thompson, S. G. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980430)17:8
Flagged first, then by citations
Research works printed but not linked: 1
- Newcombe, Robert G. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med 1998;17:857-72.
Cited by
Sorted by citations
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · see citing works
PRISMA harms checklist: improving harms reporting in systematic reviews
Zorzela, L., Loke, Y. K., Ioannidis, J. P., Golder, S., Santaguida, P., Altman, D. G., Moher, D., Vohra, S., & PRISMA harms group (2016) · BMJ · doi:10.1136/bmj.i157 · Cited by 557 · see citing works
Self-citationSide-effects associated with ketamine use in depression: a systematic review
Short, B., Fong, J., Galvez, V., Shelker, W., & Loo, C. K. (2018) · The Lancet Psychiatry · doi:10.1016/s2215-0366(17)30272-9 · Cited by 558 · see citing works
Ryan, R. E., Santesso, N., Lowe, D., Hill, S., Grimshaw, J. M., Prictor, M., Kaufman, C., Cowie, G., & Taylor, M. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007768.pub3 · Cited by 273 · see citing works
Bougioukas, K. I., Liakos, A., Tsapas, A., Ntzani, E., & Haidich, A.-B. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.10.002 · Cited by 251 · see citing works
Adverse events in meditation practices and meditation‐based therapies: a systematic review
Farias, M., Maraldi, E., Wallenkampf, K. C., & Lucchetti, G. (2020) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.13225 · Cited by 262 · see citing works
“These regulations are relevant for the study of MAEs in two ways: meditation-based clinical trials across the EU and the USA are subjected to these regulations and must report all SAEs; on the other hand, they are not required to report other types of adverse occurrences that fall outside of this list, which is likely…”
Saini, P., Loke, Y. K., Gamble, C., Altman, D. G., Williamson, P. R., & Kirkham, J. J. (2014) · BMJ · doi:10.1136/bmj.g6501 · Cited by 209 · see citing works
Self-citationBafeta, A., Koh, M., Riveros, C., & Ravaud, P. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0343 · Cited by 162 · see citing works
Reproducible and replicable pain research: a critical review
Lee, H., Lamb, S. E., Bagg, M. K., Toomey, E., Cashin, A. G., & Moseley, G. L. (2018) · Pain · doi:10.1097/j.pain.0000000000001254 · Cited by 150 · see citing works
A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies
Xu, C., Furuya-Kanamori, L., Zorzela, L., Lin, L., & Vohra, S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.02.012 · Cited by 117 · see citing works
Self-citationKoffel, J. B., & Rethlefsen, M. L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0163309 · Cited by 107 · see citing works
Meert, D., Torabi, N., & Costella, J. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.4.004 · Cited by 108 · see citing works
Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study
Xu, C., Yu, T., Furuya-Kanamori, L., Lin, L., Zorzela, L., Zhou, X., Dai, H., Loke, Y., & Vohra, S. (2022) · BMJ · doi:10.1136/bmj-2021-069155 · Cited by 82 · see citing works
Self-citationHarms in Systematic Reviews Paper 1: An introduction to research on harms
Qureshi, R., Mayo-Wilson, E., & Li, T. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.10.023 · Cited by 63 · see citing works
Xu, C., Zhou, X., Zorzela, L., Ju, K., Furuya-Kanamori, L., Lin, L., Lu, C., Musa, O. A. H., & Vohra, S. (2021) · BMC Medicine · doi:10.1186/s12916-021-02008-2 · Cited by 43 · see citing works
Self-citationMeursinge Reynders, R., Ladu, L., & Di Girolamo, N. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.07.001 · Cited by 18 · see citing works
- Authors
- Zorzela, L., Golder, S., Liu, Y., Pilkington, K., Hartling, L., Joffe, A., Loke, Y., & Vohra, S.
- Affiliations
- University of Alberta (CA)
- University of York (GB)
- Lanzhou University (CN)
- University of Westminster (GB)
- University of East Anglia (GB)
- Corresponding author
- Vohra, S.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2014-01-08
- Volume
- 348
- Issue
- jan08 1
- Pages
- f7668-f7668
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC3898583
- OpenAlex
- W2169090881
- Cited by
- 217
- References
- 51
- Citations by year
- 2026: 7 · 2025: 7 · 2024: 24 · 2023: 22 · 2022: 15 · 2021: 26 · 2020: 11 · 2019: 16 · 2018: 11 · 2017: 21 · 2016: 21 · 2015: 15 · 2014: 20
- FWCI
- 7.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- BMJ (publishedVersion, cc-by-nc)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- UEA Digital Repository (University of East Anglia) (acceptedVersion)
- Lanzhou University Institutional Repository (submittedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- harm, adverse events, reporting guidelines, patient risk factors, follow-up duration, surgical complications, Cochrane Database of Systematic Reviews, meta-analysis, risk of bias assessment, ROBIS
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Meta-Analysis as Topic, Randomized Controlled Trials as Topic, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
