Peer Review Quality and Transparency of the Peer-Review Process in Open Access and Subscription Journals
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Recent controversies highlighting substandard peer review in Open Access (OA) and traditional (subscription) journals have increased the need for authors, funders, publishers, and institutions to assure quality of peer-review in academic journals. I propose that transparency of the peer-review process may be seen as an indicator of the quality of peer-review, and develop and validate a tool enabling different stakeholders to assess transparency of the peer-review process. METHODS AND FINDINGS: Based on editorial guidelines and best practices, I developed a 14-item tool to rate transparency of the peer-review process on the basis of journals' websites. In Study 1, a random sample of 231 authors of papers in 92 subscription journals in different fields rated transparency of the journals that published their work. Authors' ratings of the transparency were positively associated with quality of the peer-review process but unrelated to journal's impact factors. In Study 2, 20 experts on OA publishing assessed the transparency of established (non-OA) journals, OA journals categorized as being published by potential predatory publishers, and journals from the Directory of Open Access Journals (DOAJ). Results show high reliability across items (α = .91) and sufficient reliability across raters. Ratings differentiated the three types of journals well. In Study 3, academic librarians rated a random sample of 140 DOAJ journals and another 54 journals that had received a hoax paper written by Bohannon to test peer-review quality. Journals with higher transparency ratings were less likely to accept the flawed paper and showed higher impact as measured by the h5 index from Google Scholar. CONCLUSIONS: The tool to assess transparency of the peer-review process at academic journals shows promising reliability and validity. The transparency of the peer-review process can be seen as an indicator of peer-review quality allowing the tool to be used to predict academic quality in new journals.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bohannon, J. (2013) · Science · doi:10.1126/science.342.6154.60 · Cited by 934
Peer review in a changing world: An international study measuring the attitudes of researchers
Mulligan, A., Hall, L., & Raphael, E. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22798 · Cited by 393 · cited 1×
“This raises questions about potential sampling bias, as is often the case in surveys about peer review among researchers (where response rates are often low [31]).”
BackgroundGodlee, F., Gale, C. R., & Martyn, C. N. (1998) · JAMA · doi:10.1001/jama.280.3.237 · Cited by 333 · cited 1×
“Peer-review reports often show low inter-reviewer reliability [17], reviewers do not always spot all errors in a manuscript [18], and review quality is often hard to assess [19].”
BackgroundBornmann, L., Mutz, R., & Daniel, H.-D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0014331 · Cited by 290 · cited 1×
“Peer-review reports often show low inter-reviewer reliability [17], reviewers do not always spot all errors in a manuscript [18], and review quality is often hard to assess [19].”
BackgroundMeasuring the quality of editorial peer review.
Jefferson, T. (2002) · JAMA · doi:10.1001/jama.287.21.2786 · Cited by 230
Author Perception of Peer Review: Impact of Review Quality and Acceptance on Satisfaction
Weber, E. J., Katz, P. P., Waeckerle, J. F., & Callaham, M. L. (2002) · JAMA · doi:10.1001/jama.287.21.2790 · Cited by 140 · cited 3×
“These averages are somewhat higher than those found in previous ratings of the quality of peer-review [23], supposedly because of the fact that the current survey only covered peer review experiences with a happy outcome for the authors.”
BackgroundResultReliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
Callaham, M. L. (1998) · JAMA · doi:10.1001/jama.280.3.229 · Cited by 122 · cited 1×
“Peer-review reports often show low inter-reviewer reliability [17], reviewers do not always spot all errors in a manuscript [18], and review quality is often hard to assess [19].”
Background
Cited by
How Open Science Helps Researchers Succeed
McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., McDougall, D., Nosek, B. A., Ram, K., Soderberg, C. K., et al. (2016) · eLife · doi:10.7554/elife.16800 · Cited by 798
The Academic, Economic and Societal Impacts of Open Access: An Evidence-Based Review
Tennant, J. P., Waldner, F., Jacques, D. C., Masuzzo, P., Collister, L. B., & Hartgerink, C. H. J. (2016) · F1000Research · doi:10.12688/f1000research.8460.3 · Cited by 560
A multi-disciplinary perspective on emergent and future innovations in peer review
Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12037.1 · Cited by 289
The effect of publishing peer review reports on referee behavior in five scholarly journals
Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B., & Squazzoni, F. (2019) · Nature Communications · doi:10.1038/s41467-018-08250-2 · Cited by 273
Peer review analyze: A novel benchmark resource for computational analysis of peer reviews
Ghosal, T., Kumar, S., Bharti, P. K., & Ekbal, A. (2022) · PLOS ONE · doi:10.1371/journal.pone.0259238 · Cited by 66
Metadata
- Authors
- Wicherts, J. M.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-01-29
- Volume
- 11
- Issue
- 1
- Pages
- e0147913
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W2276801815
- Cited by
- 165
- References
- 7
- FWCI
- 106.44
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.