CoChat

When beauty is but skin deep: dealing with problematic studies in systematic reviews

Open access
Boughton, Stephanie L, Wilkinson, Jack, Bero, Lisa · 2021 · Cochrane Database of Systematic Reviews

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

Systematic reviews provide high‐quality assessment of the evidence for healthcare interventions, but this assessment is predicated on the trustworthiness of the studies they include. The presence of untrustworthy, or 'problematic', studies in the literature poses challenges for systematic reviewers and is a threat to the reliability of systematic reviews. This is compounded by the absence of an agreed definition of what constitutes a problematic study and the lack of a validated method to identify them. The most visible problematic studies are retracted studies. The number of retractions has risen rapidly,[1, 2, 3] suggesting a rise in problematic studies in the literature or an increase in awareness of them. Articles may be retracted for a number of different reasons, including error, naïve mistakes, or research misconduct,[4] and estimates vary as to the percentage of retractions that are due to misconduct versus honest error. One study found 67.4% of retractions were due to misconduct (comprised of fraud, suspected fraud, duplicate publication, and plagiarism).[5] The MECIR standards (Methodological Expectations of Cochrane Intervention Reviews; community.cochrane.org/mecir-manual) are clear that retracted articles should not be included in systematic reviews, but retracted studies are likely to represent only the tip of the iceberg. A systematic review found that 2% of scientists admitted having fabricated, falsified, or modified data or results, while 14% reported having witnessed a colleague doing so,[6] suggesting the scale of problematic studies is far greater than the number of articles retracted. In an assessment by John Carlisle of 526 randomized controlled trials submitted to the journal Anaesthesia between February 2017 and March 2020, he believed 8% to be fatally flawed due to containing false data.[7] For those where he had access to individual patient data, this rose to 44% containing false data, and 26% being classified as critically flawed. It is unclear how many problematic studies go undetected during typical journal editorial processes, which do not usually include the sort of forensic investigation undertaken by Carlisle. Even studies that are eventually retracted have downstream effects on the healthcare literature, including on systematic reviews, due to the time lag between their initial publication and subsequent retraction. A study of all retractions indexed in PubMed up to May 2012 found that the average time‐to‐retraction was 32.91 months, with a range of less than 1 month to 304 months.[8] Another study of retracted obstetrics and gynaecology articles up to June 2018 found a median time to retraction of two years, with one study being retracted 18 years after publication.[3] During this time there may be no indication to readers or those conducting a systematic review for which it is eligible for inclusion that there are any concerns about the study, though use of Expressions of Concern to alert readers to potential issues is increasing.[9] Cochrane has developed a new policy to address potentially problematic studies in the context of systematic reviews. The Cochrane policy for managing potentially problematic studies and accompanying implementation guidance set out how Cochrane Review authors and Cochrane Review Groups (CRGs) should manage situations involving potentially untrustworthy, or 'problematic' studies in the context of Cochrane Reviews. It provides steps to follow when an included study is retracted or has a formal Expression of Concern published, and also how to deal with scenarios where there is no formal post‐publication amendment but review authors or the CRG have concerns about the trustworthiness of the data in an included study, or study eligible for inclusion in a review. While this new policy provides guidance for review authors and editors where they have identified serious concerns about a study, there is an urgent need for a validated method to identify problematic studies reliably and fairly. Existing tools for assessing included studies, such as tools for assessing risk of bias, assume that the data are real. A variety of methods to identify problematic studies have been proposed. One example is the REAPPRAISED checklist for evaluation of publication integrity.[10] Suggested methods range from testing for statistical anomalies, searching for inconsistencies in how data are reported (for example different sample sizes for the same study reported in a publication, or across several publications), searching for awkward or repetitive phrasing, analyzing images, and scrutinizing all publications from authors who have previously published retracted or problematic studies. However, many of these methods involve subjective evaluation of data, and none have been fully evaluated. Use of unvalidated methods risks over‐ or under‐detection of problematic studies, and caution is needed as misclassification of a genuine study as problematic could result in erroneous

Overview

Integrity signals

Registry

Crossref · 10.1002/14651858.ed000152

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 12 references

Cited by

See all 32

Metadata

Authors
Boughton, S. L., Wilkinson, J., & Bero, L.
Affiliations
  • Cochrane Canada (CA)
  • Manchester Academic Health Science Centre (GB)
  • University of Colorado Anschutz (US)
Venue
Cochrane Database of Systematic Reviews
Publisher
Wiley
Published
2021-06-03
Volume
6
Pages
ED000152-ED000152
ISSN
  • 1361-6137 (ISSN-L)
  • 1465-1858
OpenAlex
W3165758371
Cited by
32
References
12
Citations by year
2026: 2 · 2025: 5 · 2024: 11 · 2023: 7 · 2022: 5 · 2021: 2
FWCI
0.26
Type
journal article
Language
en
Open access
Locations
  • Cochrane Database of Systematic Reviews (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Research Explorer (The University of Manchester) (publishedVersion, other-oa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Education › Academic integrity and plagiarism
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
retracted study, research misconduct, research integrity, systematic review methodology, study quality assessment, duplicate publication, plagiarism, data fabrication, individual patient data, research funding conflicts of interest
MeSH terms
Systematic Reviews as Topic, Humans, Retraction of Publication as Topic, Time Factors, Scientific Misconduct, Checklist

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: When beauty is but skin deep: dealing with problematic studies in systematic reviews. https://app.cochat.ai/papers/10.1002/14651858.ed000152
The paper
Boughton, S. L., Wilkinson, J., & Bero, L. (2021). When beauty is but skin deep: dealing with problematic studies in systematic reviews. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.ed000152

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.