The impact of peer review on the contribution potential of scientific papers
Open accessAbstract
The peer-reviewing process has long been regarded as an indispensable tool in ensuring the quality of a scientific publication. While previous studies have tried to understand the process as a whole, not much effort has been devoted to investigating the determinants and impacts of the content of the peer review itself. This study leverages open data from nearly 5,000 PeerJ publications that were eventually accepted. Using sentiment analysis, Latent Dirichlet Allocation (LDA) topic modeling, mixed linear regression models, and logit regression models, we examine how the peer-reviewing process influences the acceptance timeline and contribution potential of manuscripts, and what modifications were typically made to manuscripts prior to publication. In an open review paradigm, our findings indicate that peer reviewers’ choice to reveal their names in lieu of remaining anonymous may be associated with more positive sentiment in their review, implying possible social pressure from name association. We also conduct a taxonomy of the manuscript modifications during a revision, studying the words added in response to peer reviewer feedback. This study provides insights into the content of peer reviews and the subsequent modifications authors make to their manuscripts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MOPRD has 20% more data from a wider variety of disciplines than the data used in Matsui et al. (2021).
“MOPRD has 20% more data from a wider variety of disciplines than the data used in this study.”
From MOPRD: A multidisciplinary open peer review dataset (2022) · cites this paper as Matsui et al. (2021)
The Data > PeerJ data section (page unavailable) identifies the disciplines included in Matsui et al.’s dataset and gives article counts, supporting the baseline for the comparison. It does not describe MOPRD, so it cannot establish that MOPRD has 20% more data or covers a wider variety of disciplines. (LLM, conf=0.80)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
An index to quantify an individual’s scientific research output
Hirsch, J. E. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507655102 · Cited by 11,669 · see citing works
(2009) · Cited by 1,900 · see citing works
Fortunato, S., Bergstrom, C. T., Börner, K., Evans, J. A., Helbing, D., Milojević, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., et al. (2018) · Science · doi:10.1126/science.aao0185 · Cited by 1,497 · see citing works
Evaluation methods for topic models
Wallach, H., Murray, I., Salakhutdinov, R., & Mimno, D. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553515 · Cited by 893 · see citing works
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605 · see citing works
Bornmann, L. (2011) · Annual Review of Information Science and Technology · doi:10.1002/aris.2011.1440450112 · Cited by 567 · see citing works
Gender diversity leads to better science
Nielsen, M. W., Alegria, S., Börjeson, L., Etzkowitz, H., Falk-Krzesinski, H. J., Joshi, A., Leahey, E., Smith-Doerr, L., Woolley, A. W., & Schiebinger, L. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1700616114 · Cited by 557 · see citing works
Exploring Topic Coherence over Many Models and Many Topics
Stevens, K., Kegelmeyer, P., Andrzejewski, D., & Buttler, D. J. (2012) · Cited by 473 · see citing works
Comparison and benchmark of name-to-gender inference services
Santamaría, L., & Mihaljević, H. (2018) · PeerJ Computer Science · doi:10.7717/peerj-cs.156 · Cited by 405 · see citing works
Are Referees and Editors in Economics Gender Neutral?*
Card, D., DellaVigna, S., Funk, P., & Iriberri, N. (2019) · The Quarterly Journal of Economics · doi:10.1093/qje/qjz035 · Cited by 297 · see citing works
The effect of publishing peer review reports on referee behavior in five scholarly journals
Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B., & Squazzoni, F. (2019) · Nature Communications · doi:10.1038/s41467-018-08250-2 · Cited by 286 · see citing works
Does Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators (1998) · JAMA · doi:10.1001/jama.280.3.240 · Cited by 278 · see citing works
Effects of training on quality of peer review: randomised controlled trial
Schroter, S., Black, N. A., Evans, S. J. W., Carpenter, J. R., Godlee, F., & Smith, R. W. (2004) · BMJ · doi:10.1136/bmj.38023.700775.ae · Cited by 268 · see citing works
Kovanis, M., Porcher, R., Ravaud, P., & Trinquart, L. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166387 · Cited by 246 · see citing works
Firm action needed on predatory journals
Clark, J., & Smith, R. (2015) · BMJ · doi:10.1136/bmj.h210 · Cited by 242 · see citing works
Implementing peer review on the net: scientific quality control in scholarly electronic journals
Harnad, S. R. (1996) · Archipel (Université du Québec à Montréal) · Cited by 176 · see citing works
Author-Reviewer Homophily in Peer ReviewPreprint
Murray, D., Siler, K., Larivière, V., Chan, W. M., Collings, A., Raymond, J. L., & Sugimoto, C. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/400515 · Cited by 176 · see citing works
How the entire scientific community can confront gender bias in the workplace
Grogan, K. E. (2018) · Nature Ecology & Evolution · doi:10.1038/s41559-018-0747-4 · Cited by 172 · see citing works
Wicherts, J. M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0147913 · Cited by 169 · see citing works
NIH funding longevity by gender
Hechtman, L. A., Moore, N. P., Schulkey, C. E., Miklos, A. C., Calcagno, A. M., Aragon, R., & Greenberg, J. H. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1800615115 · Cited by 151 · see citing works
Predatory Publishing, Questionable Peer Review, and Fraudulent Conferences
Bowman, J. D. (2014) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe7810176 · Cited by 128 · see citing works
Tools used to assess the quality of peer review reports: A methodological systematic review
Superchi, C., González, J. A., Solà, I., Cobo, E., Hren, D., & Boutron, I. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0688-x · Cited by 110 · see citing works
The Shift in Academic Quality Control
Hemlin, S., & Rasmussen, S. B. (2006) · Science Technology & Human Values · doi:10.1177/0162243905283639 · Cited by 98 · see citing works
Dimensions: redescubriendo el ecosistema de la información científica
Orduña‐Malea, E., & López‐Cózar, E. D. (2018) · El Profesional de la Informacion · doi:10.3145/epi.2018.mar.21 · Cited by 71 · see citing works
Are Referees and Editors in Economics Gender Neutral?
Card, D., DellaVigna, S., Funk, P., et al. (2019) · National Bureau of Economic Research · doi:10.3386/w25967 · Cited by 68 · see citing works
Fragments of peer review: A quantitative analysis of the literature (1969-2015)
Grimaldo, F., Marušić, A., & Squazzoni, F. (2018) · PLoS ONE · doi:10.1371/journal.pone.0193148 · Cited by 55 · see citing works
Henderson, M. (2010) · BMJ · doi:10.1136/bmj.c1409 · Cited by 36 · see citing works
Large-scale language analysis of peer review reports
Buljan, I., García-Costa, D., Grimaldo, F., Squazzoni, F., & Marušić, A. (2020) · eLife · doi:10.7554/elife.53249 · Cited by 35 · see citing works
Peer Reviewing in Political Science: New Survey Results
Djupe, P. A. (2015) · PS Political Science & Politics · doi:10.1017/s1049096514002315 · Cited by 34 · see citing works
Quantifying the Volunteer Effort of Scientific Peer Reviewing
Golden, M. P., & Schultz, D. M. (2011) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-11-00129.1 · Cited by 28 · see citing works
The influence of first author sex on acceptance rates of submissions to Anaesthesia Cases
Laycock, H. C., & Bailey, C. R. (2019) · Anaesthesia · doi:10.1111/anae.14797 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Greene WH Econometric analysis 2003 London Pearson Education
Cited by
Sorted by citations
MOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · see citing works
An Analysis of Tasks and Datasets in Peer Reviewing
Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A. (2024) · Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024) · doi:10.18653/v1/2024.sdp-1.24 · Cited by 6 · see citing works
- Authors
- Matsui, A., Chen, E., Wang, Y., & Ferrara, E.
- Affiliations
- University of Southern California (US)
- Corresponding author
- Matsui, Akira
- Venue
- PeerJ
- Publisher
- PeerJ
- Published
- 2021-09-15
- Volume
- 9
- Pages
- e11999
- ISSN
- 2167-8359 (ISSN-L)
- PMC
- PMC8459734
- OpenAlex
- W3200959306
- Cited by
- 21
- References
- 32
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 7 · 2023: 5 · 2022: 2
- FWCI
- 14.47
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PeerJ (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- DARPA (D16AP00115 and W911NF-17-C-0094)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- peer review, open peer review, reviewer anonymity, manuscript revision, editorial decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
