The DOI matches its Crossref record; 77 of the 78 references it cites were found in a registry or an index; 5 of the 27 sources its claims cite were checked for support.
The ability of different peer review procedures to flag problematic publications
Open accessAbstract
There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review's ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.
Overview
CoChat Summary
TL;DR
Using survey data on peer review procedures for 361 journals and Retraction Watch data, this paper compares retraction rates across peer review models. It finds some procedures are associated with fewer retractions, but stresses that retractions are ambivalent indicators of both problematic research and willingness to correct it.
Key findings
- Across 361 journals, 833,172 articles published since 2000 were identified, of which 670 were retracted.
- Pre-submission review was associated with fewer retractions: 7.6% of all articles went through pre-submission review, versus 4.8% of retractions (Λ(3)=18.899, p<0.001).
- Journals using novelty and anticipated impact as review criteria had more retractions; author blinding was associated with fewer retractions, especially in Social Sciences and Humanities, where 79% of articles went through double-blind review but only 13% of retractions did.
- Plagiarism-detection software introduction was followed by fewer plagiarism/duplication retractions: journals published 70,097 articles before introduction with 38 retractions, 11 for plagiarism/duplication, and 41,043 articles after with 19 retractions, only 1 for plagiarism/duplication.
Methods
- Retracted journal articles were collected from the Retraction Watch database on December 11, 2017, then matched to Web of Science; this yielded 7,861 retractions, and the final sample covered 833,172 articles and 670 retractions.
- A 12-question survey on peer review attributes was sent to journal editors; 361 useful responses were obtained, a 6.12% response rate, and responses were used to assign each article to the review procedure it went through since 2000.
- Analyses used log likelihood ratio tests on retracted/non-retracted ratios, binary logistic regression interaction tests for research areas, and ANOVA/multinomial logistic regression for retraction reasons, in SPSS 25.0.
Limitations
- The paper states selection and response bias: only Web of Science-indexed journals were sampled, and journals more attentive to or innovative in peer review were more likely to respond.
- The paper states that retractions are an imperfect proxy: many problematic articles are never retracted, the non-retracted control group may contain problematic papers, and manuscripts rejected during review cannot be observed.
- The paper states that survey self-reports may misclassify procedures or not reflect actual review practice; the text also makes evident that the design is observational/correlational and cannot establish causal mechanisms.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
5 Valid · 3 Not valid · 12 Not checked · 124 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“are not reproducible (Open Science Collaboration 2015)”
§ The ability of different peer review procedures to flag problematic publications > Introduction
What the sources say
- Partly supportedcites Open Science Collaboration 2015 sourceExperimental — accuracy not yet measured
The Discussion (p. 12) reports that a large portion of replications of 100 published psychology findings did not reproduce evidence supporting the original results. That supports concern about reproducibility, but the claim extends the finding to published articles generally and states categorically that they are not reproducible. Insights on Reproducibility (p. 13) cautions that failure to replicate does not necessarily mean an original result was a false positive. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“involve questionable research practices, or even outright misconduct (e.g. Horbach and Halffman 2017b; Nuijten et al. 2016)”
§ The ability of different peer review procedures to flag problematic publications > Introduction
What the sources say
- Not checked yetcites Horbach and Halffman 2017b
- Partly supportedcites Nuijten et al. 2016 sourceExperimental — accuracy not yet measured
Nuijten et al. (2016) support the 'large proportion ... questionable research practices' side only in a limited form: Results ('General prevalence of inconsistencies') report that 49.6% of articles with NHST results in eight psychology journals contained at least one inconsistency and 12.9% at least one gross inconsistency, and the Abstract/Introduction frames such inconsistencies as 'honest mistakes' that have 'also been categorized as one of several fairly common questionable research practices' (citing John et al., 2012) as a possible indicator of other QRPs. Nothing in our copy attributes 'outright misconduct' (fabrication/fraud) to these articles; on the contrary, the Discussion states that 'the large majority of inconsistencies are accidental' and that it remains important 'to know whether reporting inconsistencies are mere sloppiness or whether they are intentional'. The Discussion's limitations also explicitly avoid conclusions about psychology in general, restricting them to the APA-reported NHST results in the eight selected journals, so 'a large proportion of published articles' overstates the source's scope. (LLM, conf=0.82)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“scholars have demonstrated that a large proportion of published articles contains flaws (Ioannidis 2005)”
§ The ability of different peer review procedures to flag problematic publications > Introduction
What the sources say
- Partly supportedcites Ioannidis 2005 sourceExperimental — accuracy not yet measured
Ioannidis 2005 argues and simulates that most published research CLAIMS/FINDINGS are likely to be false (Abstract; 'Most Research Findings Are False for Most Research Designs and for Most Fields'), not that a quantified large proportion of articles contain identifiable flaws — and he explicitly concedes 'there is no good large-scale empirical evidence' for the frequency of some problems (Bias section), so 'demonstrated' overstates an essay built on proofs and simulations. The paper does support the general tenor: Corollary 4 (p. ~Box 1) says 'even in the most stringent research designs, bias seems to be a major problem... selective outcome reporting... is a common problem even for randomized trials', and Corollary 5 says conflicts of interest 'are very common in biomedical research, and typically they are inadequately and sparsely reported'. The unsupplied part is the article-level, demonstrated prevalence of flaws; the prevalent-flaw evidence there is credited to other works (Chan et al. [25], Krimsky et al. [26], Papanikolaou et al. [27]) rather than established by this paper. (LLM, conf=0.70)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A quarter of retractions do not involve misconduct, but honest error (Fang et al. 2012)”
§ The ability of different peer review procedures to flag problematic publications > Theoretical framework > What are retractions?
What the sources say
- Supportedcites Fang et al. 2012 source
The Causes of Retraction section states outright that 'for articles in which the reason for retraction is known, three-quarters were retracted because of misconduct or suspected misconduct, and only onequarter was retracted for error,' with the full-sample figure given as 21.3% attributable to error (also in the abstract heading passage). The cited sentence's 'a quarter ... honest error' therefore reproduces the source's own proportion and its complementary framing to the 67.4%/three-quarters misconduct figure. The only nuance is that the exact one-quarter figure is stated for the known-cause subset while the unrestricted error rate is 21.3%, which is the same point at slightly different specificity rather than a discrepancy. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, relatively many journals with a high number of retractions have responded to our survey.”
§ The ability of different peer review procedures to flag problematic publications > Methods > Data analysis
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
53 more claims
Create a free account to see all 144 claims and to verify the 22 references still open.
- Probably not a substantial claim. 30
- It may state two separate claims as one. 64
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Evidence that peer review substantially improves reporting, or catches errors or questionable research practices, is limited [9, 10].”
From Is the future of peer review automated? (2022) · cites this paper as [9]
Experimental — accuracy not yet measured
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“F(3,1262) = 2.839, p = 0.784”
§ The ability of different peer review procedures to flag problematic publications > Results > Reviewer anonymity
Reported p = .784 · recomputed p = .037
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(280) = − 3.043, p = 0.016”
§ The ability of different peer review procedures to flag problematic publications > Results > Review criteria
Reported p = .016 · recomputed p = .003
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(3, 1262) = 2.839reported p = .784 · recomputed p = .037The two fall on opposite sides of .05
“F(3,1262) = 2.839, p = .784”
- Doesn’t add upt(280) = -3.043reported p = .016 · recomputed p = .003
“t(280) = − 3.043, p = .016”
- ConsistentF(1, 1266) = 5.409reported p = .02 · recomputed p = .020
“F(1,1266) = 5.409, p = .020”
- ConsistentF(3, 4665) = 1.22reported p = .301 · recomputed p = .301
“F(3,4665) = 1.220, p = .301”
- ConsistentF(4, 2782) = 10.538reported p < .001 · recomputed p < .001
“F(4,2782) = 10.538, p < .001”
8 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ The ability of different peer review procedures to flag problematic publications > Results > External sources
“Our data (Table 12), suggest no difference in retraction rates due to the usage of review reports from external sources (Λ(3) = 42.270, p < 0.0001).”
§ The ability of different peer review procedures to flag problematic publications > Results > Summary results
“Using review reports from partner journals is related to fewer retractions.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Interaction between reviewers is facilitated | 48 (9.0%) | 5.3–9.0 | 75,100 (9.0%) | 9.0–9.1 |”
§ The ability of different peer review procedures to flag problematic publications > Results > Interaction between actors
Reported 9 · recomputed 7.164179
Citations
Cites
1 retracted
Retraction rates are on the rise
Cokol, M., Ozbay, F., & Rodriguez‐Esteban, R. (2008) · The EMBO Reports · doi:10.1038/sj.embor.7401143 · Cited by 149
RetractedWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Mainly originating in the biomedical sciences, scholars have demonstrated that a large proportion of published articles contains flaws (Ioannidis 2005), are not reproducible (Open Science Collaboration 2015), involve questionable research practices, or even outright misconduct (e.g. Horbach and…”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Medication Errors and Adverse Drug Events in Pediatric Inpatients
Kaushal, R. (2001) · JAMA · doi:10.1001/jama.285.16.2114 · Cited by 1,917
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“As a result, the number of retractions has grown sharply over the last decades, which has led some scholars and journalists to use retractions as a window to study problematic research practices (e.g. Fanelli 2009; Fanelli et al.”
BackgroundThe oligopoly of academic publishers in the digital era
Larivière, V., Haustein, S., & Mongeon, P. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127502 · Cited by 1,240
Misconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232
Martinson, B. C., Anderson, M. S., & de Vries, R. (2005) · Nature · doi:10.1038/435737a · Cited by 1,096
Research works printed but not linked: 15
- Amsen, E. (2014). What is open peer review. F1000 Researchblog (Vol. 2017).
- BMJ Open. (2018). Reviewer guidelines. http://bmjopen.bmj.com/pages/reviewerguidelines/ .
- Davis, P. (2010). Cascading peer-review: The future of open access publishing? The Scholarly Kitchen.
- EMBO Press. (2017). About: Cross peer review. http://emboj.embopress.org/about#Cross_peer_review .
- Frontiers. (2014). Frontiers is a community-rooted, open-access academic publisher. https://www.frontiersin.org/about/reviewsystem .
- Goodman, S. N. (2017). Statistical review in biomedical journals: Can we get SMARTA? In Eight international congress on peer review and scientific publication, Chicago.
- Hiney, M. (2015). What it means, why it is important and how we might protect it. Brussels: Science Europe, working group research integrity.
- Mellor, D. (2016). Registered reports: Peer review before results are known to align scientific values and practices. https://cos.io/rr/ .
- Pattinson, D., & Prater, C. (2017). Assessment of the prevalence of integrity issues in submitted manuscripts. In Paper presented at the eighth international congress on peer review and scientific publication, Chicago.
- PLOS. (2018). Guidelines for reviewers. http://journals.plos.org/plosone/s/reviewer-guidelines2018 .
- Research Square. (2017). Editorial checks and badges. https://www.researchsquare.com/publishers/badges .
- Sage Open. (2018). More information: Description. https://uk.sagepub.com/en-gb/eur/journal/sage-open#description2018 .
- Scheman, R., & Bennett, C. N. (2017) Assessing the outcomes of introducing a digital image quality control review into the publication process for research articles in physiology journals. In International congress on peer review and scientific publication, Chicago.
- Van Buggenhout, M., & Christiaens, J. (2016). Deviance in science: A criminological analysis. Brussels: Free University of Brussels.
- Van Leeuwen, T., Reyes Elizondo, A., & De Rijcke, S. (2017). Analysis of papers retracted in the Web of Science. In PRINTEGER (Ed.). Leiden.
Cited by
The limitations to our understanding of peer review
Tennant, J. P., & Ross-Hellauer, T. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00092-1 · Cited by 384
Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-020-00703-8 · Cited by 286
Pandemic publishing: Medical journals strongly speed up their publication process for COVID-19
Horbach, S. P. J. M. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00076 · Cited by 268
Self-citationOpen science saves lives: lessons from the COVID-19 pandemic
Besançon, L., Peiffer-Smadja, N., Segalas, C., Jiang, H., Masuzzo, P., Smout, C., Billy, E., Deforet, M., & Leyrat, C. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01304-y · Cited by 249
Bryanov, K., & Vziatysheva, V. (2021) · PLOS ONE · doi:10.1371/journal.pone.0253717 · Cited by 211
Metadata
- Authors
- Horbach, S. P. J. M., & Halffman, W.
- Affiliations
- Leiden University (NL)
- Radboud University Nijmegen (NL)
- Corresponding author
- Horbach, S. P. J. M.
- Venue
- Scientometrics
- Published
- 2018-11-29
- Volume
- 118
- Issue
- 1
- Pages
- 339-373
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- PMC
- PMC6404393
- OpenAlex
- W2902356319
- Semantic Scholar
- e38dfec9a644c3dc3a07eb1bffb5ffd37fb9d56d
- Cited by
- 73
- References
- 78
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 11 · 2023: 3 · 2022: 12 · 2021: 12 · 2020: 13 · 2019: 1
- FWCI
- 31.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- Locations
- Scientometrics (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- Funders
- ZonMw (445001001)
- Topics
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Education › Academic integrity and plagiarism
- Keywords
- peer review procedures, retraction rates, research misconduct, scientific fraud, Retraction Watch, journal editorial policy, disciplinary differences, retraction reasons
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The ability of different peer review procedures to flag problematic publications. https://app.cochat.ai/papers/10.1007/s11192-018-2969-2
- The paper
- Horbach, S. P. J. M., & Halffman, W. (2019). The ability of different peer review procedures to flag problematic publications. Scientometrics. https://doi.org/10.1007/s11192-018-2969-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.