Reviewing the reviewers: A study of author perception on peer reviews in computer science
Open accessAbstract
Peer reviewing is an important form of collaborative work that is used for quality assurance in science and in other domains like software development and knowledge management. Review ratings by authors have potential to improve the quality of peer reviews, by giving way to remuneration of good revi
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior papers [19-23] all find that in authors' evaluations of reviews on their own papers, the reviewer's accept-or-reject decision is highly correlated with the evaluation rating given by the authors.
“The papers [19–23] all find that in authors’ evaluations of reviews on their own papers, the decision of accept or reject given by the reviewer is highly correlated with evaluation rating given by the authors.”
From Peer reviews of peer reviews: A randomized controlled trial and other experiments (2025) · cites this paper as [20]
In “Influence of Acceptance Status on Ratings” (page not supplied), the paper reports correlations between acceptance and authors’ mean ratings of .06 to .253 and says this effect is weaker than that of review scores. The Discussion likewise says authors’ assessments depend only weakly on acceptance status, contrary to the claimed high correlation. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reliability of Editors' Subjective Quality Ratings of Peer Reviews of Manuscripts
Callaham, M. L. (1998) · JAMA · doi:10.1001/jama.280.3.229 · Cited by 122 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- M. L. Callaham, R. K. Knopp, and E. J. Gallagher, "Effect of Written Feedback by Editors on Quality of Reviews: Two Randomized Trials," lAMA, vol. 287, no. 21, 2002.
- "Nature's peer review debate," http://www.nature.com/nature/ peerreview/debate.
- D. Prelec, "A Bayesian truth serum for subjective data," Science, vol. 306, no. 5695, 2004.
- N. Miller, P. Resnick, and R. Zeckhauser, "Eliciting informative feed back: The peer-prediction method," Manage. Sci., vol. 51, no. 9, 2005.
- G. Marks and N. Miller, "Ten years of research on the false-consensus effect: An empirical and theoretical review," Psychological Bulletin, no. 102, 1987.
- E. J. Weber, P. P. Katz, J. F. Waeckerle, and et aI., "Author Perception of Peer Review: Impact of Review Quality and Acceptance on Satisfaction," lAMA, vol. 287, no. 21, 2002.
- M. Gibson et aI., "Author perception of peer review." Obstetrics and gynecology, vol. 112, no. 3, September 2008.
- J. Garfunkel et aI., "Effect of acceptance or rejection on the author's evaluation of peer review of medical manuscripts," lAMA, vol. 263, no. 10, 1990.
- " Can 'open peer review' work for biologist? biology direct is hopeful." http://www.nature.comlnature/peerreview/debate/opl.html.
- c. Bauer et aI., "The student view on online peer reviews," in ITiCSE'09. Paris, France: ACM, 2009.
- "How can we get the best out of peer review? a recipe for good peer review," http://www.nature.com/nature/peerreview/debate/ nature04995.html.
- D. Galin, Software Quality Assurance: From Theory to Implementation. Harlow, UK: Pearson Education, 2004.
- K. E. Wiegers, Peer Reviews in Software: A Practical Guide. Boston, MA.: Addison-Wesley, 2002.
- N. F. Noy, R. V. Guha, and M. A. Musen, "User ratings of ontologies: Who will rate the raters?" in Proceedings of the AAAI 2005 Spring Symposium on Knowledge Collection from Volunteer Contributors, 2005.
- C. HUtter, C. KUhne, and K. Bohm, "Peer production of structured knowledge -: an empirical study of ratings and incentive mechanisms," in CIKM '08: Proceeding of the 17th ACM conference on Information and knowledge management. New York, NY, USA: ACM, 2008, pp. 827-842.
- " CASES 2009 -international conference on compilers, architecture, and synthesis for embedded systems," 2009.
- D. Cosley et aI., "Is seeing believing?: how recommender system interfaces affect users' opinions," in CHI '03. New York, NY, USA: ACM, 2003.
Cited by
Sorted by citations
Peer reviews of peer reviews: A randomized controlled trial and other experiments
Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., & Shah, N. B. (2025) · PLOS ONE · doi:10.1371/journal.pone.0320444 · Cited by 18 · see citing works
- Authors
- Kühne, C., Böhm, K., & Yue, J. Z.
- Affiliations
- Karlsruhe Institute of Technology (DE)
- Corresponding author
- Yue, Jing Zhi
- ORCID
- Böhm, Klemens
- Venue
- Proceedings of the 6th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing
- Publisher
- IEEE
- Published
- 2010
- OpenAlex
- W1982531395
- Cited by
- 8
- References
- 1
- Citations by year
- 2025: 2 · 2018: 2 · 2017: 1 · 2014: 1 · 2013: 2
- FWCI
- 12.06
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom) (publishedVersion)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- peer review, author perceptions, computer science, quality assurance, collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
