When beauty is but skin deep: dealing with problematic studies in systematic reviews
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Systematic reviews provide high‐quality assessment of the evidence for healthcare interventions, but this assessment is predicated on the trustworthiness of the studies they include. The presence of untrustworthy, or 'problematic', studies in the literature poses challenges for systematic reviewers and is a threat to the reliability of systematic reviews. This is compounded by the absence of an agreed definition of what constitutes a problematic study and the lack of a validated method to identify them. The most visible problematic studies are retracted studies. The number of retractions has risen rapidly,[1, 2, 3] suggesting a rise in problematic studies in the literature or an increase in awareness of them. Articles may be retracted for a number of different reasons, including error, naïve mistakes, or research misconduct,[4] and estimates vary as to the percentage of retractions that are due to misconduct versus honest error. One study found 67.4% of retractions were due to misconduct (comprised of fraud, suspected fraud, duplicate publication, and plagiarism).[5] The MECIR standards (Methodological Expectations of Cochrane Intervention Reviews; community.cochrane.org/mecir-manual) are clear that retracted articles should not be included in systematic reviews, but retracted studies are likely to represent only the tip of the iceberg. A systematic review found that 2% of scientists admitted having fabricated, falsified, or modified data or results, while 14% reported having witnessed a colleague doing so,[6] suggesting the scale of problematic studies is far greater than the number of articles retracted. In an assessment by John Carlisle of 526 randomized controlled trials submitted to the journal Anaesthesia between February 2017 and March 2020, he believed 8% to be fatally flawed due to containing false data.[7] For those where he had access to individual patient data, this rose to 44% containing false data, and 26% being classified as critically flawed. It is unclear how many problematic studies go undetected during typical journal editorial processes, which do not usually include the sort of forensic investigation undertaken by Carlisle. Even studies that are eventually retracted have downstream effects on the healthcare literature, including on systematic reviews, due to the time lag between their initial publication and subsequent retraction. A study of all retractions indexed in PubMed up to May 2012 found that the average time‐to‐retraction was 32.91 months, with a range of less than 1 month to 304 months.[8] Another study of retracted obstetrics and gynaecology articles up to June 2018 found a median time to retraction of two years, with one study being retracted 18 years after publication.[3] During this time there may be no indication to readers or those conducting a systematic review for which it is eligible for inclusion that there are any concerns about the study, though use of Expressions of Concern to alert readers to potential issues is increasing.[9] Cochrane has developed a new policy to address potentially problematic studies in the context of systematic reviews. The Cochrane policy for managing potentially problematic studies and accompanying implementation guidance set out how Cochrane Review authors and Cochrane Review Groups (CRGs) should manage situations involving potentially untrustworthy, or 'problematic' studies in the context of Cochrane Reviews. It provides steps to follow when an included study is retracted or has a formal Expression of Concern published, and also how to deal with scenarios where there is no formal post‐publication amendment but review authors or the CRG have concerns about the trustworthiness of the data in an included study, or study eligible for inclusion in a review. While this new policy provides guidance for review authors and editors where they have identified serious concerns about a study, there is an urgent need for a validated method to identify problematic studies reliably and fairly. Existing tools for assessing included studies, such as tools for assessing risk of bias, assume that the data are real. A variety of methods to identify problematic studies have been proposed. One example is the REAPPRAISED checklist for evaluation of publication integrity.[10] Suggested methods range from testing for statistical anomalies, searching for inconsistencies in how data are reported (for example different sample sizes for the same study reported in a publication, or across several publications), searching for awkward or repetitive phrasing, analyzing images, and scrutinizing all publications from authors who have previously published retracted or problematic studies. However, many of these methods involve subjective evaluation of data, and none have been fully evaluated. Use of unvalidated methods risks over‐ or under‐detection of problematic studies, and caution is needed as misclassification of a genuine study as problematic could result in erroneous
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“There is however no broad consensus on how to deal with potentially fraudulent studies 6 .”
From Fraudulent studies are undermining the reliability of systematic reviews: on the prevalence of problematic images in preclinical depression studies (2025) · cites this paper as [6]
In the citing paper
“Cochrane's research integrity department points out that the Cochrane RoB tool operates on the basis that the data are true. 47”
From Identifying and managing problematic trials: A research integrity assessment tool for randomized controlled trials in evidence synthesis (2023) · cites this paper as [47]
Citations
Cites
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911
Misconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232
Why has the number of scientific retractions increased?
Steen, R. G., Casadevall, A., & Fang, F. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068397 · Cited by 415
Retractions in the scientific literature: is the incidence of research fraud increasing?
Steen, R. G. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2010.040923 · Cited by 405
Why and how do journals retract articles? An analysis of Medline retractions 1988–2008
Wager, E., & Williams, P. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2010.040964 · Cited by 263
Guidelines for retracting articles
Barbour, V., Kleinert, S., Wager, E., & Yentis, S. (2009) · doi:10.24318/cope.2019.1.4 · Cited by 185
False individual patient data and zombie randomised controlled trials submitted to Anaesthesia
Carlisle, J. B. (2021) · Anaesthesia · doi:10.1111/anae.15263 · Cited by 173
Byrne, J. A., & Christopher, J. (2020) · FEBS Letters · doi:10.1002/1873-3468.13747 · Cited by 133
Check for publication integrity before misconduct
Grey, A., Bolland, M. J., Avenell, A., Klein, A. A., & Gunsalus, C. K. (2020) · Nature · doi:10.1038/d41586-019-03959-6 · Cited by 114
Chambers, L., Michener, C., & Falcone, T. (2019) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.15689 · Cited by 99
Cited by
Weibel, S., Popp, M., Reis, S., Skoetz, N., Garner, P., & Sydenham, E. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1599 · Cited by 93
“Cochrane's research integrity department points out that the Cochrane RoB tool operates on the basis that the data are true.(47) Research integrity of the trials is acknowledged as being a separate issue that should be handled prior to RoB assessment.”
MethodMedicine is plagued by untrustworthy clinical trials. How many studies are faked or flawed?
Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-02299-w · Cited by 87
Assessing the certainty of the evidence in systematic reviews: importance, process, and use
Brignardello‐Petersen, R., & Guyatt, G. H. (2024) · American Journal of Epidemiology · doi:10.1093/aje/kwae332 · Cited by 112
“Systematic reviewers should be aware of these potential issues and make all possible efforts to detect problematic studies 20 and not include them in the body of evidence.”
Wilkinson, J., Heal, C., Antoniou, G. A., Flemyng, E., Alfirevic, Z., Avenell, A., Barbour, G., Brown, N. J. L., Carlisle, J., Clarke, M., et al. (2024) · BMJ Open · doi:10.1136/bmjopen-2024-084164 · Cited by 45
“The Cochrane policy on Managing Potentially Problematic Studies 10 11 defines a problematic study as ‘any published or unpublished study where there are serious questions about the trustworthiness of the data or findings, regardless of whether the study has been formally retracted’.”
BackgroundSelf-citationParker, L., Boughton, S., Lawrence, R., & Bero, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.07.006 · Cited by 45
Self-citation
Metadata
- Authors
- Boughton, S. L., Wilkinson, J., & Bero, L.
- Affiliations
- Cochrane Canada (CA)
- Manchester Academic Health Science Centre (GB)
- University of Colorado Anschutz (US)
- Venue
- Cochrane Database of Systematic Reviews
- Publisher
- Wiley
- Published
- 2021-06-03
- Volume
- 6
- Pages
- ED000152-ED000152
- ISSN
- 1361-6137 (ISSN-L)
- 1465-1858
- PMC
- PMC10285350
- OpenAlex
- W3165758371
- Cited by
- 32
- References
- 12
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 11 · 2023: 7 · 2022: 5 · 2021: 2
- FWCI
- 0.26
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Cochrane Database of Systematic Reviews (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- retracted study, research misconduct, research integrity, systematic review methodology, study quality assessment, duplicate publication, plagiarism, data fabrication, individual patient data, research funding conflicts of interest
- MeSH terms
- Systematic Reviews as Topic, Humans, Retraction of Publication as Topic, Time Factors, Scientific Misconduct, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When beauty is but skin deep: dealing with problematic studies in systematic reviews. https://app.cochat.ai/papers/10.1002/14651858.ed000152
- The paper
- Boughton, S. L., Wilkinson, J., & Bero, L. (2021). When beauty is but skin deep: dealing with problematic studies in systematic reviews. Cochrane Database of Systematic Reviews. https://doi.org/10.1002/14651858.ed000152
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.