CoChat
Sign Up
Cited by 25Meta-analysis and systematic reviewsPublishing and Scholarly CommunicationPsychology Research and Bibliometrics

Peer reviews of peer reviews: A randomized controlled trial and other experiments

Open access
Goldberg, Alexander, Stelmakh, Ivan, Cho, Kyunghyun, Oh, Alice, Agarwal, Alekh, Belgrave, Danielle, Shah, Nihar B·2025·PLOS ONE
DOI 10.1371/journal.pone.0320444PMC PMC11964232
Open this paper in CoChat — free account

Abstract

Is it possible to reliably evaluate the quality of peer reviews? We study this question driven by two primary motivations – incentivizing high-quality reviewing using assessed quality of reviews and measuring changes to review quality in experiments. We conduct a large scale study at the NeurIPS 2022 conference, a top-tier conference in machine learning, in which we invited (meta)-reviewers and authors to voluntarily evaluate reviews given to submitted papers. First, we conduct a randomized controlled trial to examine bias due to the length of reviews. We generate elongated versions of reviews by adding substantial amounts of non-informative content. Participants in the control group evaluate the original reviews, whereas participants in the experimental group evaluate the artificially lengthened versions. We find that lengthened reviews are scored (statistically significantly) higher quality than the original reviews. Additionally, in analysis of observational data we find that authors are positively biased towards reviews recommending acceptance of their own papers, even after controlling for confounders of review length, quality, and different numbers of papers per author. We also measure disagreement rates between multiple evaluations of the same review of 28% – 32%, which is comparable to that of paper reviewers at NeurIPS. Further, we assess the amount of miscalibration of evaluators of reviews using a linear model of quality scores and find that it is similar to estimates of miscalibration of paper reviewers at NeurIPS. Finally, we estimate the amount of variability in subjective opinions around how to map individual criteria to overall scores of review quality and find that it is roughly the same as that in the review of papers. Our results suggest that the various problems that exist in reviews of papers – inconsistency, bias towards irrelevant factors, miscalibration, subjectivity – also arise in reviewing of reviews.

AI snapshot

Written from the paper’s full text.

TL;DR

At NeurIPS 2022, evaluations of review quality showed biases, inconsistency, miscalibration, and subjectivity similar to those seen in paper reviewing. In particular, uselessly lengthened reviews were rated significantly higher, and authors rated reviews recommending acceptance of their own papers more favorably.

Key findings

  • In the randomized controlled trial, uselessly elongated reviews received higher overall quality scores than original reviews: τ = 0.64, 95% CI [0.60, 0.69], p < 0.0001, with mean scores 4.29 vs. 3.73 and a difference in means of 0.56. Effects were significant for reviewers (τ = 0.65) and meta-reviewers (τ = 0.61), and for all criteria after Holm–Bonferroni correction, strongest for Coverage (τ = 0.71, difference 0.83).
  • Authors were positively biased toward reviews recommending acceptance: across 418 matched pairs, τ = 0.82, p < 0.0001, and accept reviews scored 1.406 points higher on the 7-point scale than reject reviews; criteria scores were also higher for accept reviews.
  • Inter-evaluator disagreement rates were approximately 0.3, with the abstract reporting 28%–32%, comparable to paper-review disagreement at NeurIPS 2016 (0.25–0.3).
  • Miscalibration was estimated with author offset variance 0.780 versus about 0.45 for other evaluator types; subjectivity/commensuration loss was about 0.45 for review evaluations versus 0.402 for NeurIPS 2022 paper reviews after renormalization.

Methods

  • Opt-in study at NeurIPS 2022: 7,740 evaluators (493 meta-reviewers, 2,395 paper reviewers, 3,429 authors, 1,423 external reviewers) rated 9,870 paper reviews, yielding 24,638 evaluations; overall quality was rated on a 7-point scale and four criteria on a 5-point scale.
  • Randomized controlled trial: 10 papers/reviews and 458 external evaluators were randomly assigned to short original or uselessly elongated review versions; analysis used Mann–Whitney U-type statistics with bootstrap confidence intervals and Fisher permutation tests.
  • Observational analyses included matched author evaluations of accept vs. reject reviews, inter-evaluator agreement following prior NeurIPS 2016 methods, a Gaussian process miscalibration model, and a learned monotone mapping from criteria to overall scores for commensuration bias.

Limitations

  • Stated in the paper: possible Hawthorne effects because participants knew they were in an experiment; opt-in participation may create selection bias; the length experiment used only reviewers/meta-reviewers who had not seen the original reviews; rating scales differed between review evaluations (7-point) and paper reviews (10-point), requiring renormalization; and dishonest behavior such as lone-wolf manipulation or collusion rings could not be studied.
  • Evident from the text: the evaluations had no downstream consequences for reviewers or papers, so behavior may differ when review-quality evaluations carry real stakes such as awards or promotion.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0320444

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 34 cited sources checked · 4 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Prior papers [19-23] all find that in authors' evaluations of reviews on their own papers, the reviewer's accept-or-reject decision is highly correlated with the evaluation rating given by the authors.

    “The papers [19–23] all find that in authors’ evaluations of reviews on their own papers, the decision of accept or reject given by the reviewer is highly correlated with evaluation rating given by the authors.”
    Quote in the paperSubstantial claim1 of 5 sources checked
    • cites [19]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [20] C Kühne, 2010, Reviewing the reviewers: A study of author perception on peer reviews in computer science. 6th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2010) source

      In “Influence of Acceptance Status on Ratings” (page not supplied), the paper reports correlations between acceptance and authors’ mean ratings of .06 to .253 and says this effect is weaker than that of review scores. The Discussion likewise says authors’ assessments depend only weakly on acceptance status, contrary to the claimed high correlation. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [21] K Papagiannaki, 2007, Author feedback experiment at PAM 2007, SIGCOMM Comput Commun Rev source
    • cites [22] EJ Weber, 2002, Author perception of peer review: Impact of review quality and acceptance on satisfaction, JAMA source
    • cites [23] SM Pranić, 2020, Is the quality of reviews reflected in editors’ and authors’ satisfaction with peer review? A cross‐sectional study in 12 journals across four research fields, Learn Pub source
  • Premisecitation

    Peer review of papers is plagued with well-documented problems such as bias, subjectivity, fraud, miscalibration, and low effort.

    “While the peer review of papers is widespread, it is plagued with well-documented problems like bias, subjectivity, fraud, miscalibration, and low effort, among others (see [1] for a survey).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [1] NB Shah, 2022, Challenges, experiments, and computational solutions in peer review, Commun ACM source
  • Premisecitation

    Dishonest behavior is a prominent problem in reviews of papers that they were unable to study in the context of reviewing reviews.

    “There is one prominent problem which exists in reviews of papers which we are unable to study in the context of reviewing reviews—dishonest behavior.”
    Quote in the paperNot a substantial claim1 of 6 sources checked
    • cites [41] S Balietti, 2016, Peer review and competition in the art exhibition game, Proc Natl Acad Sci source

      The paper’s section on review fairness reports that competition led participants in the Art Exhibition Game to give unfairly low scores to direct competitors. That supports dishonest or self-interested reviewing as a concern, but the Discussion says the experiment used abstracted art exhibitions and that its results may be transferable to scholarly peer review; it does not establish the behavior’s prevalence in reviews of papers. The citing authors’ inability to study it is their own framing, not a finding attributed to this source. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [42] Y Xu, 2019, On strategyproof conference review, IJCAI

      Not checked in the free run.

    • cites [43]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [44]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [45] ML Littman, 2021, Collusion rings threaten the integrity of computer science research, Commun ACM source
    • cites [46]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Higher loss in the aggregate function indicates more variability in how evaluators map criteria to overall scores, suggesting higher commensuration bias.

    “Higher loss indicates that there is more variability in how evaluators map criteria to overall scores, suggesting higher commensuration bias.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [40] R Noothigattu, 2021, Loss functions, axioms, and peer review, JAIR source

      The Introduction (page not available) defines commensuration bias as reviewers differing in how they weight criteria, and Section 4.2 defines reviewer loss as the difference between an overall score and the aggregate mapping’s output. The paper does not establish that higher loss specifically measures greater variability in reviewers’ mappings or, therefore, higher commensuration bias. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Findingown data

    In the randomized controlled trial, the mean overall quality score for the long condition was 4.29 compared to 3.73 for the short condition.

    “Overall, the mean score for the long condition group was 4 . 29 compared to 3 . 73 for the short condition.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 25
  • May combine two claims. 21

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The criteria used in the evaluation questionnaire were chosen based on proposed Review Quality Indicators for peer reviews.

    “We chose these criteria for the questionnaire based on proposed Review Quality Indicators (RQIs) for peer reviews [31,32], additionally tailoring the questions to suit our needs of being concise and relevant to papers in the domain of machine learning.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [31] S van Rooyen, 1999, Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts, J Clin Epidemiol source
    • cites [32] C Superchi, 2019, Tools used to assess the quality of peer review reports: A methodological systematic review, BMC Med Res Methodol source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    At ICML 2020, newly recruited and trained reviewers were evaluated as higher quality than reviewers in the standard reviewer pool according to a number of metrics, including meta-reviewers' evaluations of reviews.

    “Similarly, a study at the ICML 2020 conference [13] designed a special process to recruit new paper reviewers and asked meta-reviewers to evaluate the review quality from this group and from the standard group of reviewers. They found that their newly recruited and trained reviewers were evaluated as higher quality than reviewers in the standard reviewer pool according to a number of metrics which”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [13] I Stelmakh, 2021, A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences, AAAI source

  • Classified as probably not a substantial claim.

    Premisenone given

    Past works on incentivizing high quality paper review content ([2,3]) have assumed that evaluators of review quality report 'true quality.'

    “Some past works on incentivizing high quality paper review content ([2,3]) have assumed that evaluators of review quality report “true quality.””
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [2]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [3] Y Xiao, 2018, Incentive design in peer review: Rating and repeated endogenous matching, IEEE Trans Netw Sci Eng source

  • May combine two claims.

    Findingown data

    In the randomized controlled trial, length had a positive effect on all four criteria scores, significant after Holm-Bonferroni correction at level 0.05, with the strongest effect on Coverage.

    “As shown in Table 2, we also find a positive effect of length on the criteria scores. In particular, after Holm–Bonferroni correction, results are significant at level 0 . 05 for all the criteria, with the strongest effect on Coverage.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the paper's observational data, authors show a clear positive trend in which reviews recommending strong accepts receive higher evaluations than reviews recommending strong rejects, while meta-reviewers, reviewers, and external reviewers do not show a strong trend.

    “While meta-reviewers, reviewers and external reviewers do not show a strong trend in how the positivity of review score correlates with review quality assessments, for authors there is a clear positive trend with reviews recommending strong accepts receiving higher evaluations than reviews recommending strong rejects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The authors' positive trend holds both for the overall review quality score and for assessments of specific criteria.

    “This trend holds both for the overall review quality score and for assessments of specific criteria.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The normalized variance of author offset is 0.780, which is much higher than the variance of evaluator offset for other types of approximately 0.45.

    “First, observe that the (normalized) variance of author offset of 0 . 780 is much higher than the variance of evaluator offset for other types of approximately 0 . 45”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The overall quality score given by evaluator j to review i is given by yij = fi + bj + εi,j, where fi is objective quality, bj is evaluator offset, and εi,j is idiosyncrasy.
    • The authors state this as an assumption: The objective quality fi of review i is drawn from a normal distribution with mean μ and variance αf.
    • The authors state this as an assumption: The evaluator offset bj is drawn from a separate per-type normal distribution with mean 0 and variance αb,t for each evaluator type t.
    • The authors state this as an assumption: The idiosyncrasy εi,j associated with evaluation of review i by evaluator j is drawn from a normal distribution with mean 0 and variance σ².

  • May combine two claims.

    Findingown data

    The combined type function achieved a train loss of 0.456 and a test loss of 0.457, while the per-type functions achieved a train loss of 0.448 and a test loss of 0.465.

    “Repeating this procedure 5 times, we find that the combined type function achieves a train loss of 0 . 456 and a test loss of 0 . 457, while the per-type functions achieves a train loss of 0 . 448 and a test loss of 0 . 465.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the evaluations, 29% of pairs of overall score evaluations were ties, while for each of the criteria scores 37% to 40% of pairs were ties.

    “For the overall score, 29% of pairs of evaluations were ties, while for each of the criteria scores 37% to 40% of pairs were ties.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisereasoning

    Both designing incentive mechanisms for high-quality reviewing and experiments measuring the efficacy of interventions in the peer-review process depend on reliable evaluations of the quality of reviews.

    “Both of these approaches depend on reliable evaluations of the quality of reviews.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    In the randomized controlled trial, each participant was assigned uniformly at random to either the long or short condition.

    “Additionally, each participant was assigned uniformly at random to either a “long” or “short” condition.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Participants in the experiment knew they were providing evaluations for an experiment, which may result in Hawthorne effects.

    “First, participants in the experiment knew they were providing evaluations for an experiment, which may result in “Hawthorne” effects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Participants in the randomized controlled trial were not informed about the specific goal of the experiment or the length confounder.

    “Participants were not informed about the specific goal of this additional experiment: we only notified them that the data they contributed would be used to gain insights about the review quality evaluation practice, but did not specifically mention the length confounder.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    In the length experiment, they were only able to use reviewers/meta-reviewers who did not themselves review the paper, since original reviewers had seen the actual reviews.

    “Third, a limitation in the length experiment is that we were only able to use reviewers/meta-reviewers who did not themselves review the paper, since original reviewers had seen the actual reviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In Elsevier journal reviews from 2014 across medicine, science, and computer science, there is a correlation between author satisfaction with the review and whether the review recommended acceptance.

    “In a cross-sectional study of journals in multiple disciplines, the study [23] analyzed authors’ and editors’ evaluations of review quality in Elsevier journal reviews from 2014 across medicine, science, and computer science. They found correlation between author satisfaction with the review and whether the review recommended acceptance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] SM Pranić, 2020, Is the quality of reviews reflected in editors’ and authors’ satisfaction with peer review? A cross‐sectional study in 12 journals across four research fields, Learn Pub source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    There may be other effects in the use of different scales that are not accounted for.

    “While we re-normalize so that metrics from different domains share the same scale, there may be other effects in the use of different scales that are not accounted for.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A recent paper [29] found a correlation between the length of reviews given to accepted journal articles and the future citations received by those articles, suggesting that review length may be associated with review quality.

    “A recent paper [29] analyzed whether length of reviews seems to capture review quality. They found a correlation between the length of reviews given to accepted journal articles and the future citations received by these articles, suggesting that review length may be associated with review quality.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study was conducted on an opt-in basis and was not compulsory.

    “Second, our study was conducted on an opt-in basis and was not compulsory.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study was approved by the Carnegie Mellon University Institutional Review Board, and participants gave written documentation of informed consent.

    “The participants gave written documentation of informed consent and this study was approved by the Carnegie Mellon University Institutional Review Board (IRB).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Evaluations of review quality are rife with issues like biases, inconsistency, subjectivity, and miscalibration.

    “In particular, our results suggest that evaluations of review quality are rife with issues like biases, inconsistency, subjectivity, and miscalibration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Evaluation of review quality is somewhat correlated with the number of flaws reported by reviewers, indicating that assessment of review quality may capture some objective qualities that make a review useful.

    “Subsequent work [28] tested the efficacy of evaluating reviews by generating a fictitious manuscript with known flaws, obtaining peer reviews of the manuscript and then asking editors to evaluate quality of the peer reviews. They found that evaluation of review quality is somewhat correlated with number of flaws reported by the reviewers, indicating that assessment of review quality may in fact”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28] ML Callaham, 1998, Reliability of editors’ subjective quality ratings of peer reviews of manuscripts, JAMA source

  • May combine two claims.

    Premisecitation

    In an experiment at NeurIPS 2021, the rate of disagreement between co-authors of multiple jointly authored papers about the contribution of their own papers is 0.32, and that between authors of papers and the review process is 0.34.

    “In fact, an experiment at NeurIPS 2021 found that the rate of disagreement between co-authors of multiple jointly authored papers about the contribution of their own papers is 0.32, and that between authors of papers and the review process is 0.34 [34].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [34]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One recent study [15] found that LLM-generated reviews were rated as more helpful than some human-generated reviews.

    “They found that LLM-generated reviews were rated as more helpful than some human-generated reviews.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductown data

    The external reviewers in the randomized controlled trial had not seen the original reviews on the selected papers, allowing the reviews to be manipulated without their knowledge of the treatment.

    “Importantly, unlike (meta)-reviewers and authors, the participants from this group of external reviewers had not seen the original reviews on these papers, allowing us to manipulate the reviews without their knowledge of the treatment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    There may be selection bias in which authors, reviewers, and meta-reviewers chose to participate in evaluating reviews.

    “There may be selection bias in which authors, reviewers, and meta-reviewers chose to participate in evaluating reviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    Separately analyzing the four types of evaluators accounts for selection bias in which types decided to opt-in, but there may still be selection biases within each type.

    “In many of our experiments, we separately analyze the four types of evaluators, which accounts for selection bias in which types decided to opt-in, but there still may be selection biases within each type.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    If the LLM was writing uselessly longer reviews, then uselessly elongated review bias could lead to false positive conclusions in this study.

    “Furthermore, if the LLM was writing uselessly longer reviews (e.g., the LLM adds more filler sentences), then uselessly elongated review bias could lead to false positive conclusions in this study.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Evaluator miscalibration is an issue in peer review of papers, defined as the tendency for evaluators to exhibit idiosyncrasies such as giving especially lenient or harsh reviews.

    “Another issue in peer review of papers is evaluator miscalibration, that is the tendency for evaluators to exhibit idiosyncrasies such as giving especially lenient or harsh reviews [35–37].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [36]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [37] J Wang, 2019, Your 2 is My 1

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Our work pinpoints a number of specific pitfalls in evaluating review quality, which may negatively impact downstream applications that use these evaluations.

    “In conclusion, our work pinpoints a number of specific pitfalls in evaluating review quality, which may negatively impact downstream applications that use these evaluations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Commensuration bias is a source of subjectivity in paper review, arising from reviewers having differing opinions about the relative importance of various criteria in determining overall quality of a paper.

    “In the context of paper review, one source of subjectivity arises from reviewers having differing opinions about the relative importance of various criteria in determining overall quality of a paper, a phenomenon referred to as “commensuration bias” [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] CJ Lee, 2015, Commensuration bias in peer review, Philos Sci source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The author-bias analysis was restricted to reviews where the non-author evaluators agreed in their quality evaluation.

    “We restrict our analysis to reviews where the non-author evaluators agree in their quality evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    In the NeurIPS 2014 reviews of papers, αf = 1.28, αb/αf = 0.19, and σ2/αf = 1.01.

    “For the NeurIPS 2014 reviews of papers, it was found that αf=1.28, αb∕αf=0.19 and σ²∕αf=1.01.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [38]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    At NeurIPS 2020, invited reviewers were not rated higher than reviewers recruited from the author pool, and less experienced reviewers were rated slightly higher on review quality than more experienced reviewers.

    “At NeurIPS 2020, the program chairs asked meta-reviewers to rate whether paper reviewers met their expectations [25]. They found that invited reviewers to the conference were not rated any higher than reviewers recruited from among the author pool. Additionally, they found that less experienced reviewers were actually rated slightly higher on review quality than more experienced reviewers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] HT Lin, 2020, What we learned from NeurIPS 2020 reviewing process

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    In the randomized controlled trial, the original reviews were roughly 200-300 words and the elongated reviews were roughly 600-850 words; the mean word count was 268 for original reviews and 755 for elongated reviews.

    “As shown in Fig 3, across the 10 reviews the original reviews were roughly 200-300 words long, while the elongated reviews were roughly 600-850 words long. The mean word count of the original reviews was 268 words, compared to a mean of 755 for elongated reviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Editors show strong agreement in their evaluations of peer reviews in medical journals as measured by the intraclass correlation coefficient.

    “The study [27] asked editors to evaluate the quality of peer reviews in medical journals and concluded that editors show strong agreement in their evaluations as measured by the intraclass correlation coefficient.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] ID Feurer, 1994, Evaluating peer reviews, Pilot testing of a grading instrument. JAMA source
35 of 35

  • The authors state this as an assumption.

    Non-author evaluations of review quality can serve as a proxy for the true quality of a paper review.

    “Since, we do not know the ground truth quality of a paper review, the evaluations from non-authors serve as a proxy for true quality.”

  • The paper relies on this without support.

    Textual similarity between an external reviewer's profile and a paper indicates familiarity with the paper's area.

    “in order to ensure that the external reviewers evaluated reviews on relevant papers, we chose papers so that “similarity” between the external reviewers and papers was high — here, similarity is defined as the similarity between the text of the paper and the text of the reviewers’ profile (past papers), which is used in NeurIPS 2022 and various other conferences to assign reviewers to papers in”

  • The authors state this as an assumption.

    The overall quality score given by evaluator j to review i is given by yij = fi + bj + εi,j, where fi is objective quality, bj is evaluator offset, and εi,j is idiosyncrasy.

    “The model assumes that the overall quality score given by evaluator j to review i, denoted as yij, is given by yij=fi+bj+ϵi,j,”

  • The authors state this as an assumption.

    The objective quality fi of review i is drawn from a normal distribution with mean μ and variance αf.

    “fi∼N(μ,αf) is an assumed “objective quality” of review i in the model, drawn from a normal distribution with mean μ and variance αf;”

  • The authors state this as an assumption.

    The evaluator offset bj is drawn from a separate per-type normal distribution with mean 0 and variance αb,t for each evaluator type t.

    “bj∼N(0,αb,g) is an “evaluator offset” capturing miscalibration of evaluator j. In order to capture differences in distributions of the four types of evaluators (meta-reviewers, reviewers, authors, and opt-in reviewers) we model the evaluator offset as a separate per-type normal distribution with mean 0 and variance αb,t for t ∈ { 1 , 2 , 3 , 4 } ;”

  • The authors state this as an assumption.

    The idiosyncrasy εi,j associated with evaluation of review i by evaluator j is drawn from a normal distribution with mean 0 and variance σ².

    “ϵi,j∼N(0,σ²) is the idiosyncrasy associated to this specific evaluation of review i by evaluator j.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Human evaluators of peer reviews were highly susceptible to bias from review length and paper score.

    “[41] found that human evaluators of peer reviews were highly susceptible to bias from review length and paper score”

    From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [41]

    Section 3.1.2 reports that reviewers and meta-reviewers gave significantly higher quality scores to uselessly elongated reviews. Section 3.2 finds a strong association with the score a review gave the paper for author evaluators, but says reviewers, meta-reviewers, and external reviewers did not show a strong trend. The claim overgeneralizes the paper-score bias to human evaluators as a whole. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Scoring peer review quality involves a significant degree of subjectivity.

    “since scoring peer review quality involves a significant degree of subjectivity (as also noted by Goldberg et al. (2025))”

    From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Goldberg et al. (2025)

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. HT Lin, 2020, What we learned from NeurIPS 2020 reviewing process
  2. Y Xu, 2019, On strategyproof conference review, IJCAI

Cited by

Sorted by citations

Related

Authors
Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., & Shah, N. B.
Affiliations
  • Carnegie Mellon University (US)
  • New Economic School (RU)
  • New York University (US)
  • Korea Advanced Institute of Science and Technology (KR)
  • Google (United States) (US)
  • GlaxoSmithKline Research & Development Limited (United Kingdom) (GB)
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2025-04-02
Volume
20
Issue
4
Pages
e0320444
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0320444
PMC
PMC11964232
OpenAlex
W4409103579
Semantic Scholar
17f71a45c69fb9665fc497d86438e10a857346e0
Cited by
18
References
27
Citations by year
2026: 8 · 2025: 5 · 2024: 5
FWCI
8.33
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
Carnegie Mellon University
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
peer review quality, author bias, subjectivity in assessment, NeurIPS, conference peer review
MeSH terms
Machine Learning, Humans, Peer Review, Peer Review, Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review