The DOI matches its Crossref record; 111 of the 115 references it cites were found in a registry or an index.
The limitations to our understanding of peer review
Open accessAbstract
Peer review is embedded in the core of our knowledge generation systems, perceived as a method for establishing quality or scholarly legitimacy for research, while also often distributing academic prestige and standing on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We identify core themes including editorial responsibility, the subjectivity and bias of reviewers, the function and quality of peer review, and the social and epistemic implications of peer review. The high-priority gaps are focused around increased accountability and justification in decision-making processes for editors and developing a deeper, empirical understanding of the social impact of peer review. Addressing this at the bare minimum will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Because of the increasing complexity of the knowledge being created and increased specialization, the system relies on the good-faith assumption of genuine contributions by all participants (7–10).”
From The entities enabling scientific fraud at scale are large, resilient, and growing rapidly (2025) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Peer review remains central to ensuring research quality, yet it is constrained by reviewer fatigue and human bias.”
From Evaluation of Large Language Models for Peer Review in Transplantation Research: Algorithm Validation Study (2026) · cites this paper as [1]
The cited source says what the claim attributes to it.
In the citing paper
“a lack of incentives for reviewers has been introduced as a hurdle to rigorous testing or validation of studies under review [5]”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“This is perhaps one of the main reasons why fraudulent behaviour, or questionable research practices, still enter the scholarly record at high proportions, even though peer review occurs [15, 92].”
BackgroundLee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22784 · Cited by 1,000
Peer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605
What is open peer review? A systematic review
Ross-Hellauer, T. (2017) · F1000Research · doi:10.12688/f1000research.11369.2 · Cited by 449
Self-citationvan Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations
McKiernan, E. C., Schimanski, L. A., Muñoz Nieves, C., Matthias, L., Niles, M. T., & Alperin, J. P. (2019) · eLife · doi:10.7554/elife.47338 · Cited by 347
The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future
Schimanski, L. A., & Alperin, J. P. (2018) · F1000Research · doi:10.12688/f1000research.16493.1 · Cited by 342
Research works printed but not linked: 5
- Rennie D. Editorial peer review: its development and rationale. Peer Rev Health Sci. 2003;2:1–13.
- Overbeke J, Wager E. 3: The state of evidence: what we know and what we don’t know about journal peer review. JAMA. 2011;272:79–174.
- L. A. Barba, Terminologies for reproducible research. arXiv:1802.03311 [cs] (2018) (available at http://arxiv.org/abs/1802.03311). · arXiv
- S. Pranić, B. Mehmani, S. Marušić, M. Malički, A. Marušić, in New Frontiers of Peer Review (PEERE), European Cooperation in Science and Technology (COST) (2017).
- S. Hindle, Saderi, PREreview — a new resource for the collaborative review of preprints (2017; https://elifesciences.org/labs/57d6b284/prereview-a-new-resource-for-the-collaborative-review-of-preprints).
Cited by
Alqahtani, T., Badreldin, H. A., Alrashed, M., Alshaya, A. I., Alghamdi, S. S., bin Saleh, K., Alowais, S. A., Alshaya, O. A., Rahman, I., Al Yami, M. S., et al. (2023) · Research in Social and Administrative Pharmacy · doi:10.1016/j.sapharm.2023.05.016 · Cited by 565
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 328
Guidance for Authors, Peer Reviewers, and Editors on Use of AI, Language Models, and Chatbots
Flanagin, A., Kendall-Taylor, J., & Bibbins-Domingo, K. (2023) · JAMA · doi:10.1001/jama.2023.12500 · Cited by 143
The entities enabling scientific fraud at scale are large, resilient, and growing rapidly
Richardson, R. A. K., Hong, S. S., Byrne, J. A., Stoeger, T., & Amaral, L. A. N. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2420092122 · Cited by 129
The crisis of peer review: Part of the evolution of science
Horta, H., & Jung, J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12511 · Cited by 96
Metadata
- Authors
- Tennant, J. P., & Ross-Hellauer, T.
- Affiliations
- Institute for Globally Distributed Open Research and Education (SE)
- Venue
- Research Integrity and Peer Review
- Published
- 2020-04-30
- Volume
- 5
- Issue
- 1
- Pages
- 6-6
- ISSN
- 2058-8615 (ISSN-L)
- PMC
- PMC7191707
- OpenAlex
- W3021454583
- Cited by
- 384
- References
- 115
- Citations by year
- 2026: 58 · 2025: 79 · 2024: 85 · 2023: 71 · 2022: 43 · 2021: 32 · 2020: 15
- FWCI
- 24.81
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Research Integrity and Peer Review (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- SocArXiv (OSF Preprints) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by)
- Topics
- Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Keywords
- peer review, editorial responsibility, reviewer bias, academic prestige, peer review standards, research data infrastructure, research integrity, scholarly publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The limitations to our understanding of peer review. https://app.cochat.ai/papers/10.1186/s41073-020-00092-1
- The paper
- Tennant, J. P., & Ross-Hellauer, T. (2020). The limitations to our understanding of peer review. Research Integrity and Peer Review. https://doi.org/10.1186/s41073-020-00092-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.