Quantifying professionalism in peer review
Open accessAbstract
Background: The process of peer-review in academia has attracted criticism surrounding issues of bias, fairness, and professionalism; however, frequency of occurrence of such comments is unknown. Methods: We evaluated 1491 sets of reviewer comments from the fields of “Ecology and Evolution” and “Behavioural Medicine,” of which 920 were retrieved from the online review repository Publons and 571 were obtained from six early career investigators. Comment sets were coded for the occurrence of “unprofessional comments” and “incomplete, inaccurate or unsubstantiated critiques” using an a-prior rubric based on our published research. Results are presented as absolute numbers and percentages. Results: Overall, 12% (179) of comment sets included at least one unprofessional comment towards the author or their work, and 41% (611) contained incomplete, inaccurate of unsubstantiated critiques (IIUC). Conclusions: The large number of unprofessional comments, and IIUCs observed could heighten psychological distress among investigators, particularly those at an early stage in their career. We suggest that development and adherence to a universally agreed upon reviewer code of conduct is necessary to improve the quality and professional experience of peer review.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Toxic reviews may hinder scientific progress.
“This, in turn, may hinder scientific progress (Beaumont 2019;Gerwing et al. 2020).”
From Benchmark on Peer Review Toxic Detection: A Challenging Task with a New Dataset (2025) · cites this paper as Gerwing et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bohannon, J. (2013) · Science · doi:10.1126/science.342.6154.60 · Cited by 934 · see citing works
Peer review in a changing world: An international study measuring the attitudes of researchers
Mulligan, A., Hall, L., & Raphael, E. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22798 · Cited by 393 · see citing works
Open peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261 · see citing works
Tools used to assess the quality of peer review reports: A methodological systematic review
Superchi, C., González, J. A., Solà, I., Cobo, E., Hren, D., & Boutron, I. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0688-x · Cited by 110 · cited 2× · see citing works
“Finally, a variety of tools have been created to assess the quality of peer reviews, refer to Superchi, González [12] for a detailed review.”
Background
Flagged first, then by citations
Research works printed but not linked: 12
- [1]Schwartz SJ Zamboanga BL The peer-review and editorial system: ways to fix something that might be broken Perspect Psychol Sci 2009 4 1 54 61 10.1111/j.1745-6924.2009.01106.x 26158834 · DOI
- [3]Silbiger NJ Stubler AD Unprofessional peer reviews disproportionately harm underrepresented groups in STEM PeerJ 2019 7 e8247 10.7717/peerj.8247 31844596 PMC6911688 · DOI · Paper page
- [5]Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews. 2006;(1). 10.1002/14651858.MR000016.pub2. PMC8973931 17443635 · DOI
- [6]Rothwell PM Martyn CN Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain 2000 123 9 1964 1969 10.1093/brain/123.9.1964 10960059 · DOI
- [7]Beaumont LJ. Peer reviewers need a code of conduct too. Nature. 2019;572(439). 10.1038/d41586-019-02492-w. 31431744 · DOI · Paper page
- [8]Gerwing TG Rash JA Constructive and collegial peer-review as a necessary precursor to data-driven environmental policy Mar Policy 2020 111 103721 10.1016/j.marpol.2019.103721 · DOI
- [9]Hyland K, Jiang FK. “This work is antithetical to the spirit of research”: an anatomy of harsh peer reviews. J Engl Acad Purp. 2020;46. https://www.sciencedirect.com/science/article/abs/pii/S1475158519304370?via%3Dihub.
- [10]Editors N The mental health of PhD researchers demands urgent attention Nature 2019 575 257 31723298
- [13]Brown HM Peer review should not be anonymous BMJ 2003 326 7393 824 10.1136/bmj.326.7393.824/b PMC1125732 12689993 · DOI
- [14]van Rooyen S Delamothe T Evans SJW Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial BMJ 2010 341 c5729 10.1136/bmj.c5729 21081600 PMC2982798 · DOI
- [16]Bravo G Grimaldo F López-Iñesta E Mehmani B Squazzoni F The effect of publishing peer review reports on referee behavior in five scholarly journals Nat Commun 2019 10 1 1 8 10.1038/s41467-018-08250-2 30659186 PMC6338763 · DOI
- [17]Schroter S Black N Evans S Carpenter J Godlee F Smith R Effects of training on quality of peer review: randomised controlled trial BMJ 2004 328 7441 673 10.1136/bmj.38023.700775.AE 14996698 PMC381220 · DOI
Cited by
Sorted by citations
Proceed with caution: The need to raise the publication bar for microplastics research
Provencher, J. F., Covernton, G. A., Moore, R. C., Horn, D. A., Conkle, J. L., & Lusher, A. L. (2020) · Science of The Total Environment · doi:10.1016/j.scitotenv.2020.141426 · Cited by 133 · see citing works
Examining uncertainty in journal peer reviewers’ recommendations: a cross-sectional study
Barnett, A., Allen, L., Aldcroft, A., Lash, T. L., & McCreanor, V. (2024) · Royal Society Open Science · doi:10.1098/rsos.240612 · Cited by 7 · see citing works
“This commonly used system has been criticized for lacking evidence of its effectiveness [2,3], being slow [3,4], costly [3,4], conservative [5], unable to detect errors [6–8], lacking transparency [7,9], frequently unprofessional [10] and prone to bias [3,11–13].”
MethodBenchmark on Peer Review Toxic Detection: A Challenging Task with a New Dataset
Luo, M., Peterson, B., Gan, R., Ramalingame, H., Gangrade, N., Dimarogona, A., Banerjee, I., & Howard, P. (2025) · arXiv · doi:10.48550/arxiv.2502.01676 · Cited by 1 · see citing works
- Authors
- Gerwing, T. G., Allen Gerwing, A. M., Avery-Gomm, S., Choi, C.-Y., Clements, J. C., & Rash, J. A.
- Affiliations
- University of Victoria (CA)
- Southern University of Science and Technology (CN)
- Norwegian University of Science and Technology (NO)
- Memorial University of Newfoundland (CA)
- Corresponding author
- Gerwing, Travis G.
- Venue
- Research Integrity and Peer Review
- Published
- 2020-07-24
- Volume
- 5
- Issue
- 1
- Pages
- 9-9
- ISSN
- 2058-8615 (ISSN-L)
- PMC
- PMC7379804
- OpenAlex
- W3045465353
- Cited by
- 65
- References
- 4
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 12 · 2023: 10 · 2022: 13 · 2021: 9 · 2020: 3
- FWCI
- 31.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Research Integrity and Peer Review (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- peer review, professionalism, reviewer comments, early-career researchers, psychological distress, Publons, ecology and evolution, behavioral medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
