Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
The case of Dr. Hwang Woo Suk, the South Korean stem-cell researcher, is arguably the highest profile case in the history of research misconduct. The discovery of Dr. Hwang’s fraud led to fierce criticism of the peer review process (at Science). To find answers to the question of why the journal peer review system did not detect scientific misconduct (falsification or fabrication of data) not only in the Hwang case but also in many other cases, an overview is needed of the criteria that editors and referees normally consider when reviewing a manuscript. Do they at all look for signs of scientific misconduct when reviewing a manuscript? We conducted a quantitative content analysis of 46 research studies that examined editors’ and referees’ criteria for the assessment of manuscripts and their grounds for accepting or rejecting manuscripts. The total of 572 criteria and reasons from the 46 studies could be assigned to nine main areas: (1) ‘relevance of contribution,’ (2) ‘writing / presentation,’ (3) ‘design / conception,’ (4) ‘method / statistics,’ (5) ‘discussion of results,’ (6) ‘reference to the literature and documentation,’ (7) ‘theory,’ (8) ‘author’s reputation / institutional affiliation,’ and (9) ‘ethics.’ None of the criteria or reasons that were assigned to the nine main areas refers to or is related to possible falsification or fabrication of data. In a second step, the study examined what main areas take on high and low significance for editors and referees in manuscript assessment. The main areas that are clearly related to the quality of the research underlying a manuscript emerged in the analysis frequently as important: ‘theory,’ ‘design / conception’ and ‘discussion of results.’
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Martinson, B. C., Anderson, M. S., & de Vries, R. (2005) · Nature · doi:10.1038/435737a · Cited by 1,096
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Real Science: What it Is and What it Means
Ziman, J. (2000) · doi:10.1017/cbo9780511541391 · Cited by 760
Cited by
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567
Self-citationImpact factor: outdated artefact or stepping-stone to journal certification?
Vanclay, J. K. (2012) · Scientometrics · doi:10.1007/s11192-011-0561-0 · Cited by 370
Korpela, K. M. (2010) · Current Medical Research and Opinion · doi:10.1185/03007991003603804 · Cited by 58
“It is known that the peer review system is not geared toward detecting scientific misconduct or fraud but is largely based on trust 17 .”
BackgroundClosed versus open reviewing of journal manuscripts: how far do comments differ in language use?
Bornmann, L., Wolf, M., & Daniel, H.-D. (2012) · Scientometrics · doi:10.1007/s11192-011-0569-5 · Cited by 49
Self-citation
Metadata
- Authors
- Bornmann, L., Nast, I., & Daniel, H.-D.
- Venue
- Scientometrics
- Published
- 2008-09-17
- Volume
- 77
- Issue
- 3
- Pages
- 415-432
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W2049033896
- Cited by
- 66
- References
- 3
- FWCI
- 9.54
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. https://app.cochat.ai/papers/10.1007/s11192-007-1950-2
- The paper
- Bornmann, L., Nast, I., & Daniel, H.-D. (2008). Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication. Scientometrics. https://doi.org/10.1007/s11192-007-1950-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.