Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 77 of the 78 references it cites were found in a registry or an index; 5 of the 27 sources its claims cite were checked for support.

The ability of different peer review procedures to flag problematic publications

Open access
Horbach, S. P. J. M., Halffman, W. · 2019 · Scientometrics

Abstract

There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review's ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.

Overview

CoChat Summary

TL;DR

Using survey data on peer review procedures for 361 journals and Retraction Watch data, this paper compares retraction rates across peer review models. It finds some procedures are associated with fewer retractions, but stresses that retractions are ambivalent indicators of both problematic research and willingness to correct it.

Key findings

  • Across 361 journals, 833,172 articles published since 2000 were identified, of which 670 were retracted.
  • Pre-submission review was associated with fewer retractions: 7.6% of all articles went through pre-submission review, versus 4.8% of retractions (Λ(3)=18.899, p<0.001).
  • Journals using novelty and anticipated impact as review criteria had more retractions; author blinding was associated with fewer retractions, especially in Social Sciences and Humanities, where 79% of articles went through double-blind review but only 13% of retractions did.
  • Plagiarism-detection software introduction was followed by fewer plagiarism/duplication retractions: journals published 70,097 articles before introduction with 38 retractions, 11 for plagiarism/duplication, and 41,043 articles after with 19 retractions, only 1 for plagiarism/duplication.

Methods

  • Retracted journal articles were collected from the Retraction Watch database on December 11, 2017, then matched to Web of Science; this yielded 7,861 retractions, and the final sample covered 833,172 articles and 670 retractions.
  • A 12-question survey on peer review attributes was sent to journal editors; 361 useful responses were obtained, a 6.12% response rate, and responses were used to assign each article to the review procedure it went through since 2000.
  • Analyses used log likelihood ratio tests on retracted/non-retracted ratios, binary logistic regression interaction tests for research areas, and ANOVA/multinomial logistic regression for retraction reasons, in SPSS 25.0.

Limitations

  • The paper states selection and response bias: only Web of Science-indexed journals were sampled, and journals more attentive to or innovative in peer review were more likely to respond.
  • The paper states that retractions are an imperfect proxy: many problematic articles are never retracted, the non-retracted control group may contain problematic papers, and manuscripts rejected during review cannot be observed.
  • The paper states that survey self-reports may misclassify procedures or not reflect actual review practice; the text also makes evident that the design is observational/correlational and cannot establish causal mechanisms.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11192-018-2969-2

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

13 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Figures

Fig. 1 Overview of the retractions data. Starting with the RetractionWatch database, we omitted a large number of retracted conference proceedings by IEEE. The final set consists of the resulting retracted articles that are contained in the WoS database
Fig. 1 Overview of the retractions data. Starting with the RetractionWatch database, we omitted a large number of retracted conference proceedings by IEEE. The final set consists of the resulting retracted articles that are contained in the WoS databaseCC BY · PubMed Central
Fig. 2 Cumulative distribution of retractions over journals, either in the entire population, in our sample, or outside of our sample
Fig. 2 Cumulative distribution of retractions over journals, either in the entire population, in our sample, or outside of our sampleCC BY · PubMed Central

Verification

Claims the paper makes

5 Valid · 3 Not valid · 12 Not checked · 124 Own results and interpretations

53 more claims

Create a free account to see all 144 claims and to verify the 22 references still open.

See all claims — free account
  • Probably not a substantial claim. 30
  • It may state two separate claims as one. 64
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

2 of 2

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upF(3, 1262) = 2.839reported p = .784 · recomputed p = .037The two fall on opposite sides of .05

    “F(3,1262) = 2.839, p = .784”

  • Doesn’t add upt(280) = -3.043reported p = .016 · recomputed p = .003

    “t(280) = − 3.043, p = .016”

  • ConsistentF(1, 1266) = 5.409reported p = .02 · recomputed p = .020

    “F(1,1266) = 5.409, p = .020”

  • ConsistentF(3, 4665) = 1.22reported p = .301 · recomputed p = .301

    “F(3,4665) = 1.220, p = .301”

  • ConsistentF(4, 2782) = 10.538reported p < .001 · recomputed p < .001

    “F(4,2782) = 10.538, p < .001”

5 of 13 · the rest aren’t loaded

8 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

1 retracted

Show all 78 references

Research works printed but not linked: 15

  1. Amsen, E. (2014). What is open peer review. F1000 Researchblog (Vol. 2017).
  2. BMJ Open. (2018). Reviewer guidelines. http://bmjopen.bmj.com/pages/reviewerguidelines/ .
  3. Davis, P. (2010). Cascading peer-review: The future of open access publishing? The Scholarly Kitchen.
  4. EMBO Press. (2017). About: Cross peer review. http://emboj.embopress.org/about#Cross_peer_review .
  5. Frontiers. (2014). Frontiers is a community-rooted, open-access academic publisher. https://www.frontiersin.org/about/reviewsystem .
  6. Goodman, S. N. (2017). Statistical review in biomedical journals: Can we get SMARTA? In Eight international congress on peer review and scientific publication, Chicago.
  7. Hiney, M. (2015). What it means, why it is important and how we might protect it. Brussels: Science Europe, working group research integrity.
  8. Mellor, D. (2016). Registered reports: Peer review before results are known to align scientific values and practices. https://cos.io/rr/ .
  9. Pattinson, D., & Prater, C. (2017). Assessment of the prevalence of integrity issues in submitted manuscripts. In Paper presented at the eighth international congress on peer review and scientific publication, Chicago.
  10. PLOS. (2018). Guidelines for reviewers. http://journals.plos.org/plosone/s/reviewer-guidelines2018 .
  11. Research Square. (2017). Editorial checks and badges. https://www.researchsquare.com/publishers/badges .
  12. Sage Open. (2018). More information: Description. https://uk.sagepub.com/en-gb/eur/journal/sage-open#description2018 .
  13. Scheman, R., & Bennett, C. N. (2017) Assessing the outcomes of introducing a digital image quality control review into the publication process for research articles in physiology journals. In International congress on peer review and scientific publication, Chicago.
  14. Van Buggenhout, M., & Christiaens, J. (2016). Deviance in science: A criminological analysis. Brussels: Free University of Brussels.
  15. Van Leeuwen, T., Reyes Elizondo, A., & De Rijcke, S. (2017). Analysis of papers retracted in the Web of Science. In PRINTEGER (Ed.). Leiden.

Cited by

See all 73

Metadata

Authors
Horbach, S. P. J. M., & Halffman, W.
Affiliations
  • Leiden University (NL)
  • Radboud University Nijmegen (NL)
Corresponding author
Horbach, S. P. J. M.
Venue
Scientometrics
Published
2018-11-29
Volume
118
Issue
1
Pages
339-373
ISSN
  • 0138-9130 (ISSN-L)
  • 1588-2861
OpenAlex
W2902356319
Cited by
73
References
78
Citations by year
2026: 8 · 2025: 13 · 2024: 11 · 2023: 3 · 2022: 12 · 2021: 12 · 2020: 13 · 2019: 1
FWCI
31.55
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/4.0 (tdm)
Locations
  • Scientometrics (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Radboud Repository (Radboud University) (submittedVersion, other-oa)
Funders
ZonMw (445001001)
Topics
  • Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
peer review procedures, retraction rates, research misconduct, scientific fraud, Retraction Watch, journal editorial policy, disciplinary differences, retraction reasons

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The ability of different peer review procedures to flag problematic publications. https://app.cochat.ai/papers/10.1007/s11192-018-2969-2
The paper
Horbach, S. P. J. M., & Halffman, W. (2019). The ability of different peer review procedures to flag problematic publications. Scientometrics. https://doi.org/10.1007/s11192-018-2969-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.