Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
The debate on whether the peer-review system is in crisis has been heated recently. A variety of alternative systems have been proposed to improve the system and make it sustainable. However, we lack sufficient evidence and data related to these issues. Here we used a previously developed agent-based model of the scientific publication and peer-review system calibrated with empirical data to compare the efficiency of five alternative peer-review systems with the conventional system. We modelled two systems of immediate publication, with and without online reviews (crowdsourcing), a system with only one round of reviews and revisions allowed (re-review opt-out) and two review-sharing systems in which rejected manuscripts are resubmitted along with their past reviews to any other journal (portable) or to only those of the same publisher but of lower impact factor (cascade). The review-sharing systems outperformed or matched the performance of the conventional one in all peer-review efficiency, reviewer effort and scientific dissemination metrics we used. The systems especially showed a large decrease in total time of the peer-review process and total time devoted by reviewers to complete all reports in a year. The two systems with immediate publication released more scientific information than the conventional one but provided almost no other benefit. Re-review opt-out decreased the time reviewers devoted to peer review but had lower performance on screening papers that should not be published and relative increase in intrinsic quality of papers due to peer review than the conventional system. Sensitivity analyses showed consistent findings to those from our main simulations. We recommend prioritizing a system of review-sharing to create a sustainable scientific publication and peer-review system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bohannon, J. (2013) · Science · doi:10.1126/science.342.6154.60 · Cited by 934
Peer review in a changing world: An international study measuring the attitudes of researchers
Mulligan, A., Hall, L., & Raphael, E. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22798 · Cited by 393 · cited 3×
“For estimating the two time-related measures, we used the respective distributions from an international survey of 4000 participants (Mulligan et al. 2013).”
BackgroundMethodInfluentialPeer review: still king in the digital age
NICHOLAS, D., WATKINSON, A., JAMALI, H. R., HERMAN, E., TENOPIR, C., VOLENTINE, R., ALLARD, S., & LEVINE, K. (2015) · Learned Publishing · doi:10.1087/20150104 · Cited by 242 · cited 1×
“…number of papers submitted for publication, which puts pressure on the system (Bohannon 2013; Hopewell et al. 2014; Arns 2014; Jennings 2006; Mulligan et al. 2013; Nicholas et al. 2015; Rennie 2016; Sense About Science 2012; Siler et al. 2015; Walker and Rocha da Silva 2015b; Kovanis et al. 2016b).”
BackgroundBruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016) · BMC Medicine · doi:10.1186/s12916-016-0631-5 · Cited by 218 · cited 1×
“Until 2016, only 22 randomized controlled trials had been conducted to assess peer-review interventions such as double-blind peer review and the addition of a statistical reviewer (Bruce et al. 2016).”
BackgroundSelf-citationFormalizing the role of agent-based modeling in causal inference and epidemiology
Marshall, B. D. L., & Galea, S. (2015) · American Journal of Epidemiology · doi:10.1093/aje/kwu274 · Cited by 192 · cited 1×
“Because of multiple interactions of many heterogeneous and independent agents (e.g., authors, editors, reviewers, papers), this sort of systemic thinking and detailed microscopic modelling was necessary (Galea et al. 2010; Vespignani 2012; Bonabeau 2002; Marshall and Galea 2015).”
BackgroundLet’s make peer review scientific
Rennie, D. (2016) · Nature · doi:10.1038/535031a · Cited by 111 · cited 2×
“…number of papers submitted for publication, which puts pressure on the system (Bohannon 2013; Hopewell et al. 2014; Arns 2014; Jennings 2006; Mulligan et al. 2013; Nicholas et al. 2015; Rennie 2016; Sense About Science 2012; Siler et al. 2015; Walker and Rocha da Silva 2015b; Kovanis et al. 2016b).”
BackgroundHopewell, S., Collins, G. S., Boutron, I., Yu, L.-M., Cook, J., Shanyinde, M., Wharton, R., Shamseer, L., & Altman, D. G. (2014) · BMJ · doi:10.1136/bmj.g4145 · Cited by 103 · cited 2×
“…the sustainability of the system given the growing number of papers submitted for publication, which puts pressure on the system (Bohannon 2013; Hopewell et al. 2014; Arns 2014; Jennings 2006; Mulligan et al. 2013; Nicholas et al. 2015; Rennie 2016; Sense About Science 2012; Siler et al.…”
BackgroundPeer review for biomedical publications: we can improve the system
Stahel, P. F., & Moore, E. E. (2014) · BMC Medicine · doi:10.1186/s12916-014-0179-1 · Cited by 103 · cited 1×
“…allow editors of journals within each respective group to discuss rejected manuscripts and to propose submission to another journal of the group (Walker and Rocha da Silva 2015b; Cals et al. 2013; Gura 2002; Houry et al. 2012; Patel 2014; Stahel and Moore 2014; van Rooyen et al. 1999; Ware 2013).”
BackgroundModelling the effects of subjective and objective decision making in scientific peer review
Park, I.-U., Peacey, M. W., & Munafò, M. R. (2014) · Nature · doi:10.1038/nature12786 · Cited by 103 · cited 2×
“objectivity and subjectivity in reviewers’ decisions macroscopically bias peer review (Park et al. 2014) or estimated the level of bias necessary to affect peer review in grant applications (Day 2015).”
BackgroundPatel, J. (2014) · BMC Medicine · doi:10.1186/s12916-014-0128-z · Cited by 75 · cited 1×
“…allow editors of journals within each respective group to discuss rejected manuscripts and to propose submission to another journal of the group (Walker and Rocha da Silva 2015b; Cals et al. 2013; Gura 2002; Houry et al. 2012; Patel 2014; Stahel and Moore 2014; van Rooyen et al. 1999; Ware 2013).”
Background
Cited by
The limitations to our understanding of peer review
Tennant, J. P., & Ross-Hellauer, T. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00092-1 · Cited by 384
A multi-disciplinary perspective on emergent and future innovations in peer review
Tennant, J. P., Dugan, J. M., Graziotin, D., Jacques, D. C., Waldner, F., Mietchen, D., Elkhatib, Y., B. Collister, L., Pikas, C. K., Crick, T., et al. (2017) · F1000Research · doi:10.12688/f1000research.12037.1 · Cited by 289
Arbitrariness in the peer review process
Brezis, E. S., & Birukou, A. (2020) · Scientometrics · doi:10.1007/s11192-020-03348-1 · Cited by 82
Can editors save peer review from peer reviewers?
D’Andrea, R., & O’Dwyer, J. P. (2017) · PLOS ONE · doi:10.1371/journal.pone.0186111 · Cited by 80
“Recent tests indicate that alternatives such as bidding and review-sharing may outperform classical peer review in speed of publication and quality control [30, 34].”
BackgroundDoes My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1129 · Cited by 48
“Birukou et al. (2011) and Kovanis et al. (2017) discuss the shortcomings of the review process in general, such as its inability to detect major flaws in papers (Godlee et al., 1998) and its ineffectiveness in selecting papers that will have high citation counts in the future (Ragone et al., 2013).”
Background
Metadata
- Authors
- Kovanis, M., Trinquart, L., Ravaud, P., & Porcher, R.
- Venue
- Scientometrics
- Published
- 2017-04-03
- Volume
- 113
- Issue
- 1
- Pages
- 651-671
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W2604669265
- Cited by
- 70
- References
- 10
- FWCI
- 29.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication. https://app.cochat.ai/papers/10.1007/s11192-017-2375-1
- The paper
- Kovanis, M., Trinquart, L., Ravaud, P., & Porcher, R. (2017). Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication. Scientometrics. https://doi.org/10.1007/s11192-017-2375-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.