The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Open accessAbstract
Providing constructive feedback to paper authors is a core component of peer review. With reviewers increasingly having less time to perform reviews, automated support systems are required to ensure high reviewing quality, thus making the feedback in reviews useful for authors. To this end, we identify four key aspects of review comments (individual points in weakness sections of reviews) that drive the utility for authors: Actionability, Grounding & Specificity, Verifiability, and Helpfulness. To enable evaluation and development of models assessing review comments, we introduce the RevUtil dataset. We collect 1,430 human-labeled review comments and scale our data with 10k synthetically labeled comments for training purposes. The synthetic data additionally contains rationales, i.e., explanations for the aspect score of a review comment. Employing the RevUtil dataset, we benchmark fine-tuned models for assessing review comments on these aspects and generating rationales. Our experiments demonstrate that these fine-tuned models achieve agreement levels with humans comparable to, and in some cases exceeding, those of powerful closed models like GPT-4o. Our analysis further reveals that machine-generated reviews generally underperform human reviews on our four aspects.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper defines four aspects of peer-review comment utility for authors—Actionability, Grounding & Specificity, Verifiability, and Helpfulness—and introduces RevUtil, with 1,430 human-labeled and 10k synthetically labeled review comments. Fine-tuned open models match or exceed GPT-4o on aspect scoring, while machine-generated reviews generally underperform human reviews on utility.
Key findings
- RevUtil contains 1,430 human-labeled review comments, each annotated by three annotators, plus 10,000 synthetically labeled comments per aspect generated with GPT-4o. Human inter-annotator agreement was moderate overall: quadratic-weighted Cohen’s κ₂ was 0.614 for Actionability, 0.435 for Grounding & Specificity, 0.495 for Verifiability, and 0.511 for Helpfulness; on the Full+Majority subset, all aspects except Grounding & Specificity reached κ₂ > 0.6.
- Fine-tuned models achieved agreement comparable to or exceeding GPT-4o. For example, Llama-3.1-IT-8B improved Actionability on synthetic data from 0.103 zero-shot to 0.726 after fine-tuning, and on combined Full+Majority human data it reached κ₂ 0.550 for Actionability, 0.563 for Grounding & Specificity, 0.389 for Verifiability, and 0.554 for Helpfulness, versus GPT-4o’s 0.544, 0.517, 0.368, and 0.544, respectively; however, it underperformed GPT-4o on claim-detection F1 (0.476 vs. 0.533).
- Machine-generated reviews generally underperformed human reviews on the four aspects. Using the fine-tuned Llama-3.1-IT-8B model on 20 papers, human reviews scored higher than GPT-4 reviews on Actionability (2.34 ± 1.36 vs. 1.89 ± 0.91, p = 0.023), Grounding & Specificity (2.92 ± 1.53 vs. 2.43 ± 0.88, p = 0.017), and Verifiability (2.14 ± 1.24 vs. 1.66 ± 0.84, p = 0.009), while Helpfulness was slightly higher for GPT-4 reviews (2.96 ± 0.72 vs. 2.81 ± 1.06, p = 0.335).
- Synthetic labels showed fair-to-moderate agreement with human labels, with κ₂ ranging from 0.368 to 0.544 across aspects; rationale quality was rated highly, e.g., GPT-4o rationales averaged 4.64 Relevance and 4.16 Correctness for Actionability, and fine-tuned Llama-3.1-IT-8B rationales were on par with GPT-4o rationales in manual evaluation.
Methods
- The authors defined four aspects—Actionability, Grounding & Specificity, Verifiability, and Helpfulness—with 1–5 score rubrics, plus a binary claim-detection label for Verifiability. They segmented reviews into comments using rule-based string matching, extracting 207k comments from 60k reviews; manual validation of 100 random comments found 94% correctly segmented.
- Human annotations were collected via Prolific from PhD-level CS/IT native English speakers, with three annotators per sample. Data sources included NLPeer (ACL 2017, ARR 2022), Reviewer2 (NeurIPS 2016–2022, ICLR 2021–2023), and OpenReview (EMNLP 2023, ICLR 2024–2025). Synthetic annotations were generated with GPT-4o using in-context examples and rationales, producing 10,000 labeled comments per aspect.
- Models with ≤8B parameters were fine-tuned with LoRA using Alpaca-style prompts, in score-only and score-plus-rationale modes, and evaluated with quadratic-weighted Cohen’s κ₂, Spearman correlation, Krippendorff’s alpha, and F1. Utility of human vs. GPT-4 reviews was assessed on 20 papers from Du et al. (2024), yielding 95 human review comments and 47 LLM-generated comments, rated by the fine-tuned Llama-3.1-IT-8B model and by human evaluation, with Welch’s t-test for significance.
Limitations
- The paper states that RevUtil covers only NLP and ML peer reviews and only English, so generalization to other CS or scientific domains and languages is unclear. It also notes that the task uses only the review comment, not the full paper, though a 100-sample check found better alignment without full-paper context except for Verifiability; segmentation is imperfect; some comments target meta-reviewers rather than authors; and the task remains subjective, especially Helpfulness.
- The paper states that synthetic data generation requires development data to ensure quality, and that the four aspects are not always relevant to all extracted comments, such as trivial layout or typo comments.
- Evident from the text: human evaluation of rationales used only 25 test examples per aspect (100 total), and the human-vs-LLM review utility comparison used only 20 papers, limiting the strength of those conclusions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
17 of 34 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured
- Premisecitation§ 2.1 Review Evaluation · p. 2
Comparing LLM-generated reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments.
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
Quote in the paperSubstantial claim2 of 2 sources checked- cites Novikova et al., 2017 source
Section 8.2 (page unavailable) discusses how multiple human references affect word-based metric scores, and Section 7.1 (page unavailable) finds that no metric consistently achieves even moderate correlation with human ratings. The study evaluates generated restaurant and hotel utterances, not LLM-generated reviews, so it does not establish the claim specifically for reviews. (LLM, conf=0.85)
Experimental — accuracy not yet measured - cites Reiter, 2018 source
The Introduction describes BLEU as measuring word-based overlap with a reference text, and the Results report poor BLEU–human-judgment correlations for natural language generation. The paper does not establish that human reference reviews are diverse or that their diversity causes this limitation; it also does not specifically assess LLM-generated reviews. (LLM, conf=0.86)
Experimental — accuracy not yet measured
- Premisecitation§ Review Utility Evaluation · p. 1
Unprofessional reviews can impact productivity and cause emotional distress, particularly for early-career researchers and those from underrepresented groups.
“Furthermore, unprofessional reviews can impact productivity and cause emotional distress, particularly for early-career researchers and those from underrepresented groups (Silbiger and Stubler, 2019; Hyland and Jiang, 2020).”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites Silbiger and Stubler, 2019 source
The Results and Conclusions (page not provided) report perceived negative effects on productivity and scientific self-confidence, particularly among underrepresented groups. The paper did not assess emotional distress as such, and the Conclusions identify career stage as a subject for future investigation rather than finding that early-career researchers were particularly affected. (LLM, conf=0.88)
Experimental — accuracy not yet measured - cites Hyland and Jiang, 2020 source
In “Comment and criticism in peer reviews” (pp. 4–5), the paper says harsh reviews can be demoralizing, exacerbate negative feelings, and discourage first-time writers from revising and resubmitting. This supports emotional distress and a possible effect on publication activity, especially for novices. It does not establish an effect on productivity more broadly or identify researchers from underrepresented groups as particularly affected. (LLM, conf=0.86)
Experimental — accuracy not yet measured
- Premisecitation§ Review Utility Evaluation · p. 1
Reviewers have less time to complete reviews.
“Reviewers have less time to complete reviews; consequently, reviews suffer from poor quality and high variance (Rogers and Augenstein, 2020; Künzli et al., 2022).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Rogers and Augenstein, 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Künzli et al., 2022 source
- Findingown data§ 3.2.2 RevUtil Human · p. 5
When excluding annotations with no agreement, 72%-84% of original data is retained.
“Whenexcluding the annotations for which no agreement can be reached, we retain 72%-84% of our original data.”
Quote in the paperNot a substantial claim - Findingown data§ 3.2.2 RevUtil Human · p. 5
The low F1 value for Verifiability comes from divergent annotations from one annotator.
“Furthermore, the low value of the F1 comes from divergent annotations from one annotator (see Table 11 for the pair-wise agreements).”
Quote in the paperNot a substantial claim
- Findingown data§ 3.2.1 Peer Review Data · p. 4
The paper extracted 207k review comments from 60k reviews.
“Following this process, we extract 207k review comments from 60k reviews.”
Quote in the paperSubstantial claim - Findingown data§ 3.2.2 RevUtil Human · p. 5
The highest κ2 score (0.614) occurred for Actionability.
“The highest κ 2 score (0.614) occurs for Actionability”
Quote in the paperNot a substantial claim - Findingown data§ 3.2.2 RevUtil Human · p. 5
Grounding & Specificity showed a lower κ2 score (0.435).
“Grounding & Specificity shows a lower κ 2 score (0.435)”
Quote in the paperNot a substantial claim - Findingown data§ 2.1 Review Evaluation · p. 2
Open and general LLM-as-a-judge models do not perform well on the task of assessing review comments.
“For open and general LLM-as-a-judge models, our results show that they do not perform well on this task (see §5).”
Quote in the paperSubstantial claim - Findingown data§ Abstract · p. 1
Machine-generated reviews generally underperform human reviews on the four aspects.
“Our analysis further reveals that machinegenerated reviews generally underperform human reviews on our four aspects.”
Quote in the paperSubstantial claim - Findingown data§ 3.2.2 RevUtil Human · p. 5
Inter-annotator agreement was moderate across all four annotated aspects.
“Focusing on κ 2 , we observe moderate agreement levels across all aspects.”
Quote in the paperNot a substantial claim - Findingown data§ Abstract · p. 1
Fine-tuned models achieve agreement levels with humans comparable to, and in some cases exceeding, those of powerful closed models like GPT-4o.
“Our experiments demonstrate that these fine-tuned models achieve agreement levels with humans comparable to, and in some cases exceeding, those of powerful closed models like GPT-4o.”
- Also stated in: § Review Utility Evaluation · p. 2 · boosted
“Finetuning models on the collected data and show that they are equivalent or outperform closed models like GPT-4o, demonstrating that privacy-preserving models can provide useful review comment scoring and feedback.”
Quote in the paperSubstantial claim - Also stated in: § Review Utility Evaluation · p. 2 · boosted
- Findingown data§ 2.2 Review Aspects · p. 3
The vast majority of comments were appropriate in our review data.
“However, we found that the vast majority of comments were appropriate in our review data and therefore decided to omit this aspect.”
Quote in the paperSubstantial claim - Findingown data§ 3.2.1 Peer Review Data · p. 4
94% of review points were correctly segmented by the rule-based segmentation.
“We find that 94% of the points are correctly segmented.”
Quote in the paperSubstantial claim - Findingnone given§ 2.3 Human Preference Data · p. 3
RevUtil contains human preferences on review comments on multiple aspects.
“Similarly, RevUtil contains human preferences on review comments on multiple aspects.”
Quote in the paperSubstantial claim - Findingnone given§ Review Utility Evaluation · p. 2
The synthetic labeling method obtains good agreement with human annotations.
“using a method that obtains good agreement with human annotations.”
Quote in the paperNot a substantial claim - Findingown data§ O Paper Text Context · p. 31
In 89.25% of samples, on average, the two setups (with and without paper context) arrive at the same prediction or their labels differ by at most ±1.
“Table 22 shows that in 89.25% of samples, on average, the two setups arrive at the same prediction or their labels differ by at most ±1.”
Quote in the paperSubstantial claim - Findingown data§ F Manual Data Statistics · p. 21
Most labels are concentrated around scores '5' and '1'.
“We observe that most labels are concentrated around scores '5' and '1', indicating that extreme cases are relatively easier for annotators to agree on, while middle-range cases show lower agreement.”
Quote in the paperNot a substantial claim - Findingown data§ H Synthetic Data Statistics and Distribution · p. 23
Synthetic labels on the FullAgreement and MajorityAgreement datasets show high agreement with human labels.
“We note that we don't have ground truth for the synthetic labels; however, the synthetic labels on the FullAgreement and MajorityAgreement datasets show high agreement with human labels (see §3.2.3) while being generated using the same method used with the synthetic labels here.”
Quote in the paperNot a substantial claim - Findingown data§ 3.2.2 RevUtil Human · p. 5
On the majority-agreement subset, all aspects except Grounding & Specificity achieve substantial agreement (κ2 > 0.6).
“On this subset, we obtain a substantial agreement ( κ 2 > 0 . 6 ) for all aspects, except Grounding & Specificity”
Quote in the paperNot a substantial claim - Findingnone given§ Limitations · p. 9
RevUtil exclusively contains peer reviews from the NLP and ML community.
“RevUtil exclusively contains peer reviews from the NLP and ML community.”
Quote in the paperNot a substantial claim - Findingown data§ 6.1 Error Analysis · p. 8
Grounding & Specificity exhibited the lowest IAA (κ2=0.435).
“Since this aspect exhibited the lowest IAA ( κ 2 = 0 . 435 , cf. Table 1)”
Quote in the paperSubstantial claim - Findingreasoning§ I.1 Fine-Tuning · p. 23
Computing loss only on the completion tokens improved model performance.
“A key optimization that improved model performance was computing loss only on the completion tokens.”
Quote in the paperNot a substantial claim - Findingown data§ 6.1 Error Analysis · p. 9
In 90% of cases, the model assigns lower scores than humans for Actionability.
“In 90% of cases, the model assigns lower scores than humans.”
Quote in the paperNot a substantial claim - Findingnone given§ 5 Results · p. 7
Small, open models underperform GPT-4o without fine-tuning.
“Unsurprisingly, the small, open models underperform GPT-4o without fine-tuning.”
Quote in the paperNot a substantial claim - Findingown data§ 5 Results · p. 7
Llama-3.1-IT-8B achieves the highest agreement across all aspects.
“Fine-tuning greatly boosts performance compared to zero-shot, with Llama-3.1-IT8B achieving the highest agreement across all aspects (e.g., Actionability rising from 0.103 to 0.726).”
Quote in the paperSubstantial claim - Findingown data§ 6.1 Error Analysis · p. 9
The model frequently underrates comments referencing the overall paper, failing to recognize them as grounded.
“the model frequently underrates comments referencing the overall paper (e.g., writing, formatting), failing to recognize them as grounded”
Quote in the paperNot a substantial claim - Findingown data§ 6.2 Utility of Human vs. LLM Reviews · p. 9
Same trends repeat with human evaluation: human-written reviews are rated with higher utility than LLM-generated ones, albeit not statistically significant.
“The same trends repeat with the human evaluation, where human-written reviews are rated with higher utility than the LLM-generated ones, albeit not being statistically significant.”
Quote in the paperNot a substantial claim - Findingown data§ 5 Results · p. 7
Fine-tuning greatly boosts performance compared to zero-shot.
“Fine-tuning greatly boosts performance compared to zero-shot, with Llama-3.1-IT8B achieving the highest agreement across all aspects (e.g., Actionability rising from 0.103 to 0.726).”
Quote in the paperSubstantial claim - Findingown data§ 5 Results · p. 7
Even the tested LLM judges that are fine-tuned to rate texts according to provided score definitions severely lag behind GPT-4o.
“However, even the tested LLM judges that are finetuned to rate texts according to provided score definitions, severely lag behind GPT-4o.”
Quote in the paperSubstantial claim - Findingown data§ 5 Results · p. 7
In the zero-shot setting, GPT-4o demonstrates the strongest overall performance across most evaluation aspects, except for Verifiability.
“In the zero-shot setting, GPT-4o demonstrates the strongest overall performance across most evaluation aspects, except for Verifiability .”
Quote in the paperSubstantial claim - Findingown data§ 7 Conclusion · p. 9
Fine-tuned models can outperform zero-shot, closed-source models.
“We fine-tuned models on our newly collected data and showed that they can outperform zero-shot, closed-source models.”
Quote in the paperSubstantial claim - Findingown data§ 5 Results · p. 7
Training models to generate rationales yields only limited, inconsistent gains.
“Training models to generate rationales yields only limited, inconsistent gains, and sciencespecialized models (ScitLitLLM, DeepReviewer, SciTülu) underperform Llama-3.1-IT-8B.”
Quote in the paperSubstantial claim - Findingown data§ 7 Conclusion · p. 9
RevUtil contains 1,430 human and 10k synthetically annotated review comments for each of four aspects.
“We use these definitions to collect RevUtil , a dataset of 1,430 human, and 10k synthetically annotated review comments for each of our four aspects.”
Quote in the paperSubstantial claim - Findingown data§ 6.1 Error Analysis · p. 8
Disagreements often resulted from subtle differences in interpretation.
“Disagreements often resulted from subtle differences in interpretation:”
Quote in the paperNot a substantial claim - Findingreasoning§ 5 Results · p. 7
The fine-tuned model underperforms GPT-4o in claim detection F1.
“The fine-tuned model outperforms GPT-4o in κ 2 , but underperforms in claim detection F1, likely due to GPT-4o's two-step approach for Verifiability compared to the single-step setup.”
Quote in the paperSubstantial claim - Findingown data§ 6.1 Error Analysis · p. 8
Annotators vary in judging whether vague references are identifiable to authors.
“annotators vary in judging whether vague references ('The method is only tested on two datasets') are identifiable to authors.”
Quote in the paperNot a substantial claim - Findingown data§ 6.1 Error Analysis · p. 9
Most errors involve underestimation.
“Most errors involve underestimation”
Quote in the paperNot a substantial claim - Findingown data§ 6.1 Error Analysis · p. 8
Annotators disagree on the level of detail required for a comment to be deemed actionable.
“Annotators disagree on the level of detail required for a comment to be deemed actionable.”
Quote in the paperNot a substantial claim - Findingown data§ 5 Results · p. 7
Science-specialized models (SciLitLLM, DeepReviewer, SciTülu) underperform Llama-3.1-IT-8B.
“Training models to generate rationales yields only limited, inconsistent gains, and sciencespecialized models (ScitLitLLM, DeepReviewer, SciTülu) underperform Llama-3.1-IT-8B.”
Quote in the paperSubstantial claim - Findingown data§ O Paper Text Context · p. 31
Providing GPT-4o with the full paper text as extra context (excluding references) confused the model for all utility aspects except Verifiability.
“Results in Table 21 show that the extra context confused the model for all aspects except for Verifiability , where it leads to an increased performance.”
Quote in the paperNot a substantial claim - Findingown data§ M Aspects Correlation · p. 24
Verifiability shows relatively weak correlations with the other aspects.
“In contrast, Verifiability shows relatively weak correlations with the other aspects, indicating it captures a more distinct dimension of review comment utility.”
Quote in the paperSubstantial claim - Findingown data§ B Review Segmentation · p. 19
After applying the segmentation process, the paper obtains a total of 207k unique review comments from weaknesses and discussion sections of reviews.
“After applying the segmentation process, we obtain a total of 207k unique review comments from weaknesses and discussion sections of reviews.”
Quote in the paperSubstantial claim - Findingreasoning§ 5 Results · p. 6
Agreement scores on the human-annotated dataset are generally lower than on the synthetic test set.
“Overall, we observe that agreement scores on the human-annotated dataset are generally lower than on the synthetic test set.”
Quote in the paperNot a substantial claim - Findingown data§ Limitations · p. 10
On average, on 89.25% of samples, the two setups fully agree or deviate in the predicted label by at most ±1.
“on average, on 89.25% of samples, the two setups fully agree or deviate in the predicted label by at most ± 1 .”
Quote in the paperSubstantial claim - Findingown data§ 5 Results · p. 8
GPT-4o achieves the highest similarities with the reference rationales in the zero-shot setting.
“GPT-4o achieves the highest similarities with the reference rationales in the zero-shot setting.”
Quote in the paperNot a substantial claim - Findingown data§ 6.2 Utility of Human vs. LLM Reviews · p. 9
Human-written reviews outperform LLM-generated ones across 3 out of 4 aspects with statistically significant differences.
“The results show that human-written reviews outperform LLM-generated ones across 3 out of 4 aspects with statistically significant differences measured by Welch's t-test (Welch, 1947).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Welch, 1947 source
- Findingown data§ O Paper Text Context · p. 31
Providing GPT-4o with the full paper text as extra context (excluding references) increased its performance for Verifiability.
“Results in Table 21 show that the extra context confused the model for all aspects except for Verifiability , where it leads to an increased performance.”
Quote in the paperNot a substantial claim - Findingown data§ 3.2.3 RevUtil Synthetic · p. 6
Synthetic and human labels show fair to moderate agreement across all aspects, with κ2 scores ranging from 0.368 to 0.544.
“Across all aspects, the κ 2 scores range from 0.368 to 0.544, indicating fair to moderate agreement between synthetic and human labels.”
Quote in the paperSubstantial claim - Findingown data§ 5 Results · p. 8
The rationales generated by the best fine-tuned model are on par with the GPT-4o rationales.
“Results in Table 6 show that the rationales are on par with the GPT-4o rationales.”
Quote in the paperSubstantial claim- Rests on an assumption: The 25 test samples are representative of the synthetic RevUtil data.
- Rests on an assumption: The first author's manual evaluation is a valid measure of rationale quality.
- Findingown data§ Limitations · p. 10
The entire review is seldom required to understand a particular review comment.
“Preliminarily, we found that the entire review is seldom required to understand a particular review comment.”
Quote in the paperSubstantial claim - Findingown data§ 6.1 Error Analysis · p. 9
The model tends to treat reviewer questions as vague and non-actionable.
“It tends to treat reviewer questions as vague and non-actionable”
Quote in the paperNot a substantial claim - Findingreasoning§ 5 Results · p. 8
The metrics are higher when the associated aspect prediction is correct.
“Notably, the metrics are higher when the associated aspect prediction is correct”
Quote in the paperNot a substantial claim - Findingown data§ N Training Data Vs. Performance · p. 24
Models' performance starts to saturate after around 5k training examples.
“Results in Figure 9 show that the models' performance starts to saturate after around 5k training examples.”
Quote in the paperNot a substantial claim - Findingown data§ 5 Results · p. 7
The fine-tuned model outperforms GPT-4o in κ2.
“The fine-tuned model outperforms GPT-4o in κ 2 , but underperforms in claim detection F1, likely due to GPT-4o's two-step approach for Verifiability compared to the single-step setup.”
Quote in the paperSubstantial claim - Interpretationown data§ 3.2.2 RevUtil Human · p. 5
Grounding & Specificity has greater subjectivity than other aspects.
“indicating greater subjectivity in this aspect.”
Quote in the paperNot a substantial claim - Interpretationown data§ 3.2.2 RevUtil Human · p. 5
Annotators were relatively consistent when determining whether comments included actionable comments.
“suggesting that annotators were relatively consistent when determining whether the comments included actionable comments.”
Quote in the paperNot a substantial claim - Interpretationown data§ Review Utility Evaluation · p. 2
Privacy-preserving models can provide useful review comment scoring and feedback.
“demonstrating that privacy-preserving models can provide useful review comment scoring and feedback.”
Quote in the paperSubstantial claim - Interpretationown data§ 5 Results · p. 8
There is a link between accurate scoring and high-quality explanations.
“reinforcing the link between accurate scoring and high-quality explanations”
Quote in the paperNot a substantial claim - Interpretationown data§ 5 Results · p. 7
The synthetic rationales are of strong overall quality.
“indicating that the synthetic rationales are of strong overall quality”
Quote in the paperSubstantial claim- Rests on an assumption: The 25 test samples are representative of the synthetic RevUtil data.
- Interpretationown data§ O Paper Text Context · p. 31
GPT-4o rarely uses the paper context to produce a significantly different prediction.
“This demonstrates that GPT-4o rarely uses the paper context to produce a significantly different prediction.”
Quote in the paperSubstantial claim - Interpretationown data§ 6.1 Error Analysis · p. 8
The primary reason for disagreements stems from the subjective interpretation of implicit references to specific parts of the paper.
“our qualitative analysis of these samples suggests that the primary reason for these disagreements stems from the subjective interpretation of implicit references to specific parts of the paper.”
Quote in the paperNot a substantial claim - Interpretationown data§ 6.1 Error Analysis · p. 8
Even in the relatively small set of low agreement (13% of all data), a general trend of agreement persists.
“This shows that even in the relatively small set of low agreement (13% of all data), a general trend of agreement persists.”
Quote in the paperNot a substantial claim - Interpretationown data§ M Aspects Correlation · p. 24
Verifiability captures a more distinct dimension of review comment utility.
“In contrast, Verifiability shows relatively weak correlations with the other aspects, indicating it captures a more distinct dimension of review comment utility.”
Quote in the paperSubstantial claim - Premisecitation§ Limitations · p. 9
Reviewing guidelines of other conferences and journals mention our aspects.
“the reviewing guidelines of other conferences and journals also mention our aspects.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Nature, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites CVPR, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites PLOS, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Classified as probably not a substantial claim. 63
- May combine two claims. 30
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 2.2 Review Aspects · p. 3Prior work analyses peer reviews on different aspects, including harshness, thoroughness and helpfulness, completeness, substantiation, and politeness.
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
Quote in the paperNot a substantial claim1 of 3 sources checked- cites Verma et al., 2022 source
- cites Severin et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Yuan et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ Review Utility Evaluation · p. 2Thakkar et al. (2025) leveraged closed-source models to provide feedback due to their superior zero-shot capabilities.
“Besides, Thakkar et al. (2025) leveraged closedsource models to provide feedback due to their superior zero-shot capabilities.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Thakkar et al., 2025 source
Section 2.2 (p. 5) says the feedback agent used Claude Sonnet 3.5 as its backbone, supporting the use of a closed-source model. It says the model was chosen after a blind preference evaluation against GPT-4o and Gemini 1.5 Flash; it does not attribute the choice to superior zero-shot capabilities. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim. May combine two claims.
Premisecitation§ 1 Introduction · p. 1Peer review provides explanations and constructive feedback to authors to improve their scientific contribution, besides serving as a high-quality filter.
“Besides serving as a high-quality filter, it provides explanations and constructive feedback to authors to improve their scientific contribution (Jefferson et al., 2002; Ross-Hellauer, 2017).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Jefferson et al., 2002 source
- cites Ross-Hellauer, 2017 source
The Background section, “Problems with peer review,” describes reviewers’ feedback as helping improve manuscripts and their judgments as informing selection decisions. It does not establish that peer review is a consistently high-quality filter; the same section says its ability to filter the best work into the best journals is “found wanting.” No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measured
May combine two claims.
Premisecitation§ Review Utility Evaluation · p. 1Reviews suffer from poor quality and high variance.
“Reviewers have less time to complete reviews; consequently, reviews suffer from poor quality and high variance (Rogers and Augenstein, 2020; Künzli et al., 2022).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Rogers and Augenstein, 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Künzli et al., 2022 source
In the paper’s main text (page unavailable), the authors report that “I do not have time” is a leading reason for declining review invitations and describe a resulting shortage of reviewers. They do not say that completed reviews are of poor quality or highly variable, or establish that reduced time causes those outcomes. (LLM, conf=0.81)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S2 (checkable).
Interpretationcitation§ Review Utility Evaluation · p. 1Peer review would benefit from novel support systems to facilitate reviewing and increase review quality.
“Therefore, peer review would benefit from novel support systems to facilitate reviewing and increase review quality (Kuznetsov et al., 2024).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kuznetsov et al., 2024 source
- Rests on an assumption: Without high-quality feedback, authors lack guidance on how to improve their work, leading to resubmissions without substantial changes and consequent inefficiencies in the scientific process.
May combine two claims.
Premisecitation§ Review Utility Evaluation · p. 2ICLR 2025 found that reviewers updated their review after receiving feedback, and the revised feedback was preferred by paper authors in 89% of cases.
“They found that reviewers updated their review after receiving feedback, and the revised feedback was preferred by paper authors in 89% of cases, indicating that such feedback is accepted by reviewers and effective in improving the quality and utility of reviews for authors.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Thakkar et al., 2025 source
Section 3.1 (p. 7) reports that 26.6% of reviewers who received feedback updated their reviews. The preference evaluation (p. 10) found that modified reviews were preferred in 89 of 100 selected examples, but the evaluators were two blinded AI researchers, not the papers’ authors. The 89% figure therefore does not establish author preference. (LLM, conf=0.94)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.2.3 RevUtil Synthetic · p. 5Providing rationales along with examples is more effective than incorporating Chain-of-Thought reasoning in the prompt.
“We also tried to incorporate Chain-of-Thought reasoning (Wei et al., 2022) in our prompt, but found that providing rationales along with the examples is more effective.”
- Also stated in: § 4 Experiments · p. 6 · plain
“We observed that providing GPT-4o with rationales in the in-context examples improved the quality of generated synthetic labels”
Quote in the paperNot a substantial claim1 of 3 sources checked- cites Wei et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Ye et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Zheng et al., 2023 source
Section 3.4 (p. 6) reports that few-shot judgment examples improved GPT-4's consistency and that a chain-of-thought prompt could leave the judge making the same mistakes as the answers it graded. It does not say the examples included rationales or compare rationale-providing examples directly against chain-of-thought prompting, so the claimed superiority is not established. (LLM, conf=0.78)
Experimental — accuracy not yet measured
- Also stated in: § 4 Experiments · p. 6 · plain
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ 4 Experiments · p. 6LoRA achieves comparable performance to full fine-tuning while requiring significantly less time and compute.
“Previous research demonstrated that LoRA achieves comparable performance to full fine-tuning, while requiring significantly less time and compute (Dettmers et al., 2023; Ghosh et al., 2024).”
Quote in the paperNot a substantial claim2 of 2 sources checked- cites Dettmers et al., 2023 source
The paper reports that LoRA matches full fine-tuning performance for LLaMA 7B on Alpaca only when adapters are applied to all linear transformer layers; default LoRA settings did not match it (QLoRA-All section, page unavailable). The Low-rank Adapters section says LoRA reduces memory requirements, but the cited passages do not establish that LoRA requires significantly less time and compute than full fine-tuning. (LLM, conf=0.88)
Experimental — accuracy not yet measured - cites Ghosh et al., 2024 source
The Evaluation Datasets section (page unavailable) says LoRA fine-tuning trains fewer parameters, making it faster and more efficient than full-parameter fine-tuning. Finding 4 (page unavailable) reports that LoRA outperformed full fine-tuning on factuality and usefulness in the paper’s instruction-tuning comparison. The source does not quantify time or compute savings, and these results do not establish comparable performance across fine-tuning tasks generally. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitation§ 3.2.2 RevUtil Human · p. 4The data was split into batches of 100-200 samples to prevent annotator fatigue.
“To maintain the quality of the annotations, we split our data into batches of 100200 samples each, to ensure our annotators do not suffer from annotation fatigue (Kern et al., 2023).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Kern et al., 2023 source
The Discussion raises annotator fatigue as a possible cause of lower-quality annotations, but says fatigue effects have not been tested. Section 3.1 reports up to 50 tweets per annotator, not batches of 100–200 samples; the cited paper therefore supports the fatigue concern, not the stated batch size or its effectiveness in preventing fatigue. (LLM, conf=0.87)
Experimental — accuracy not yet measured
May combine two claims.
Findingcitation§ Limitations · p. 10Alignment with human data is better without the full paper context, except for Verifiability which could benefit from additional context.
“We find that the alignment with human data is better without the full paper context (likely due to the phenomenon that LLMs are easily distracted by too much irrelevant context (Shi et al., 2023)), except for Verifiability , which could benefit from additional context.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Shi et al., 2023 source
Probably not a substantial claim: may not pass S2 (checkable).
Premisecitation§ 3.2.2 RevUtil Human · p. 4Scoring peer review quality involves a significant degree of subjectivity.
“since scoring peer review quality involves a significant degree of subjectivity (as also noted by Goldberg et al. (2025))”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Goldberg et al. (2025) source
May combine two claims.
Findingown data§ Abstract · p. 1The RevUtil dataset contains 1,430 human-labeled review comments and 10k synthetically labeled comments.
“We collect 1,430 human-labeled review comments and scale our data with 10k synthetically labeled comments for training purposes.”
- Also stated in: § Review Utility Evaluation · p. 2 · plain
“We collect human annotations of 1,430 review comments for the introduced aspects, with each sample labeled by three different annotators, and scale the data collection with 10k synthetically labeled review comments, using a method that obtains good agreement with human annotations.”
Quote in the paperSubstantial claim- Rests on an assumption: Research into evaluating and improving systems for automated reviewing feedback is limited by a lack of evaluation data.
- Also stated in: § Review Utility Evaluation · p. 2 · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.2.2 RevUtil Human · p. 4The human-annotated dataset consists of 1,430 review comments.
“Our human-annotated dataset consists of 1,430 review comments.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ Abstract · p. 1The synthetic data contains rationales, i.e., explanations for the aspect score of a review comment.
“The synthetic data additionally contains rationales, i.e., explanations for the aspect score of a review comment.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone given§ 2.2 Review Aspects · p. 3Our annotations are on the comment level, which can consist of multiple sentences, while previous datasets provide phrase- or sentence-level annotations.
“Furthermore, our annotations are on the commentlevel, which can consist of multiple sentences, while previous datasets provide phrase- or sentencelevel annotations.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingown data§ 2.1 Review Evaluation · p. 2Our data provides more fine-grained annotations by evaluating individual review comments instead of whole reviews.
“Further, our data provides more fine-grained annotations by evaluating individual review comments instead of whole reviews.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Findingreasoning§ Segmentation of Reviews into Comments · p. 10The task remains somewhat subjective.
“the task remains somewhat subjective.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ B Review Segmentation · p. 19Table 8 reports the statistics of the raw data across five venues, from the NLP and ML domains, and over seven years.
“Table 8 reports the statistics of our raw data across the five venues, from the NLP and ML domains, and over seven years.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 6.1 Error Analysis · p. 8In 187 samples without majority agreement, pair-wise disagreements: 40.3% differ by 1, 36% differ by 2, 17.5% differ by 3, and 6.2% differ by 4.
“Quantitatively, we find that 40.3% only differ by 1, 36% differ by 2, 17.5% differ by 3, and 6.2% differ by 4.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data§ B Review Segmentation · p. 19The total number of reviews is 60,084 and the total number of extracted comments is 207,187.
“| Total | 60084 | 207187 | 341 | 40 |”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 3.2.2 RevUtil Human · p. 5Grounding & Specificity improves but remains in the moderate agreement range on the majority-agreement subset.
“which improves, but remains in the moderate agreement range.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingnone given§ 5 Results · p. 6Human agreement scores represent the average agreement against the three annotators, including LowAgreement cases where all three annotators provided different scores.
“This is expected, as the human agreement scores represent the average agreement against the three annotators, including LowAgreement cases where all three annotators provided different scores.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone given§ I.3 Training and Inference Times · p. 24For inference, to predict all aspect scores for a single review comment, it takes on average 0.67s for Llama 3B, 5.0s for Llama 8B, and 10.1s for Llama 70B.
“For inference, to predict all aspect scores for a single review comment, it takes on average 0.67s for Llama 3B, 5.0s for Llama 8B, and 10.1s for Llama 70B.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 5 Results · p. 8Across all four aspects, fine-tuned models outperform zero-shot ones in both lexical and semantic similarity.
“Across all four aspects, fine-tuned models outperform zero-shot ones in both lexical and semantic similarity.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Findingnone given§ I.3 Training and Inference Times · p. 24Fine-tuning the Llama-3B model for one epoch takes 90 minutes, while the 8B model took 120 minutes.
“Fine-tuning the Llama-3B model for one epoch takes 90 minutes, while the 8B model took 120 minutes.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ H Synthetic Data Statistics and Distribution · p. 23In total, the authors labeled 10,000 review comments across the four aspects.
“In total, we labeled 10,000 review comments across the four aspects.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ B Review Segmentation · p. 18In the dataset, the comment length statistics are Mean: 53.13, Std: 47.62, Min: 5.51, Max: 100.76 words.
“In our dataset, these statistics are: Mean: 53.13 Std: 47.62 Min: 5.51 Max: 100.76 words.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 5 Results · p. 7The average ratings for both Relevance and Correctness in Table 5 are high.
“Table 5 shows that the average ratings for both Relevance and Correctness are high”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ H Synthetic Data Statistics and Distribution · p. 23On average, each review contains about 1.33 labeled comments.
“On average, each review contains about 1.33 labeled comments.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ Segmentation of Reviews into Comments · p. 10Most review comments are directed toward authors.
“Although most review comments are directed toward authors”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data§ Segmentation of Reviews into Comments · p. 10The rule-based approach to segmenting review text into independent comments is mostly effective but imperfect.
“While our rule-based approach to segmenting review text into independent comments is mostly effective, it remains imperfect.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 6.1 Error Analysis · p. 9Disagreement often stems from claim detection; the model either overlooks claims or misinterprets neutral statements as claims.
“Disagreement often stems from claim detection. The model either overlooks claims or misinterprets neutral statements as claims.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data§ B Review Segmentation · p. 19Table 8 reports the number of reviews and extracted individual review comments across venues and years, and the average number of words per review and per comment.
“Table 8: Number of reviews ( #R ) and extracted, individual review comments ( #P ) across venues and years. The |R| and |P| columns report the average number of words, rounded to the nearest integer.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ E Data License · p. 21The authors release their dataset with the CC-BY-NC-SA 4.0 open license to allow future use and utilization.
“We also release our dataset with CC-BY-NC-SA 4.0, an open license, to allow future use and utilization of our dataset.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ H Synthetic Data Statistics and Distribution · p. 23Synthetic label distributions appear more uniform than human-labeled data.
“Compared to human-labeled data, the synthetic label distributions appear more uniform, suggesting a more balanced representation across the different categories, which shows some divergence from the human judgments.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingreasoning§ Segmentation of Reviews into Comments · p. 10The four aspects are not always relevant to all extracted comments.
“the four aspects are not always relevant to all extracted comments”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ Limitations · p. 9All reviews in RevUtil are in English only.
“Finally, all reviews in RevUtil are in English only.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 6.2 Utility of Human vs. LLM Reviews · p. 9On the Helpfulness aspect, GPT-4 scores are slightly higher, but the difference is not significant.
“On the Helpfulness aspect, where GPT-4 scores are slightly higher, the difference in the reviews is not significant.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ Segmentation of Reviews into Comments · p. 10Some review comments are intended for meta-reviewers.
“we recognize that some are intended for meta-reviewers.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingown data§ 3.2.3 RevUtil Synthetic · p. 5The paper generated 10,000 labeled review comments for each aspect.
“In total, we generate 10,000 labeled review comments for each aspect.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown data§ 5 Results · p. 8Among the fine-tuned models, Llama-3.1-IT-8B achieves the highest scores across all metrics, followed closely by SciLitLLM.
“Among the fine-tuned models, Llama-3.1-IT-8B achieves the highest scores across all metrics, followed closely by SciLitLLM”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ 6.1 Error Analysis · p. 8Most cases (80%) of Helpfulness follow the three-consecutive-score pattern.
“Most cases (80%) follow the three-consecutive-score pattern.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 6.1 Error Analysis · p. 8Disagreement arises when annotators interpret borderline comments as either facts or claims.
“Disagreement arises when annotators interpret borderline comments as either facts or claims.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingreasoning§ Segmentation of Reviews into Comments · p. 10Subjectivity is more prominent in the Helpfulness aspect.
“This becomes more prominent in the Helpfulness aspect.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ Segmentation of Reviews into Comments · p. 10Some comments were incorrectly split, resulting in fragments that no longer formed coherent units.
“Some were incorrectly split, resulting in fragments that no longer formed coherent units.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 2.1 Review Evaluation · p. 2LLMs are increasingly used to generate peer reviews.
“Increasingly, LLMs are also used to generate peer reviews (Liang et al., 2024a).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Liang et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisereasoning§ Review Utility Evaluation · p. 2Providing paper drafts to external services can violate the ethics policies of conferences.
“However, this raises ethical concerns as paper drafts are private, and providing them to external services can violate the ethics policies of conferences.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Cahill et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone given§ 2.2 Review Aspects · p. 3Severin et al.'s helpfulness definition relates to our Actionability and Verifiability aspects.
“On the other hand, their helpfulness definition relates to our Actionability and Verifiability aspects.”
Quote in the paperNot a substantial claimMay combine two claims.
Premisecitation§ 2.2 Review Aspects · p. 3Severin et al. only collect binary labels on a sentence level, employing a single annotator.
“However, they only collect binary labels on a sentence level, employing a single annotator.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Severin et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S4 (based).
Interpretationreasoning§ 2.2 Review Aspects · p. 3A comment is the natural level of granularity as it provides the minimal, but necessary context to judge the proposed aspects.
“A comment is the natural level of granularity as it provides the minimal, but necessary context to judge the proposed aspects, however, it relies on reviewers structuring their review accordingly.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone given§ Review Utility Evaluation · p. 2The four aspects (Actionability, Verifiability, Grounding & Specificity, Helpfulness) assess a review comment's utility for authors.
“Introducing and defining four key aspects to assess a review comment's utility for authors, namely: Actionability , Verifiability , Grounding & Specificity , and Helpfulness .”
- Also stated in: § Abstract · p. 1 · plain
“we identify four key aspects of review comments (individual points in weakness sections of reviews) that drive the utility for authors: Actionability , Grounding & Specificity , Verifiability , and Helpfulness .”
Quote in the paperSubstantial claim- Also stated in: § Abstract · p. 1 · plain
Classified as probably not a substantial claim.
Interpretationreasoning§ 2.2 Review Aspects · p. 3Severin et al.'s thoroughness aspect is out of scope for judging the utility of the review for authors.
“While this is interesting for a comprehensive review evaluation, it is out of scope for judging the utility of the review for authors.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationreasoning§ 2.1 Review Evaluation · p. 2Our fine-tuned models are specialized in assessing an important aspect of review quality, namely the utility for authors, and can therefore be potentially used as evaluation metrics for automated review generation models.
“Conversely, our fine-tuned models are specialized in assessing an important aspect of review quality, namely the utility for authors, and can therefore be potentially used as evaluation metrics for automated review generation models.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationcitation§ Review Utility Evaluation · p. 2Such feedback is accepted by reviewers and effective in improving the quality and utility of reviews for authors.
“They found that reviewers updated their review after receiving feedback, and the revised feedback was preferred by paper authors in 89% of cases, indicating that such feedback is accepted by reviewers and effective in improving the quality and utility of reviews for authors.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationown data§ 2.3 Human Preference Data · p. 3Our data can be viewed as a human preference collection.
“Our data can also be viewed as a human preference collection, which has recently gained popularity for post-training LLMs to align them with humans (Ouyang et al., 2022; Bai et al., 2022).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisereasoning§ E Data License · p. 21The data sources reported in section 3.2.1 are with open licenses.
“The data sources reported in 3.2.1 are with open licenses.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ Segmentation of Reviews into Comments · p. 10Some assessments may not apply to comments aimed at meta-reviewers rather than authors.
“some assessments may not apply to comments aimed at metareviewers rather than authors.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown data§ 3.2.2 RevUtil Human · p. 4Annotator performance was evaluated against a gold-standard dataset labeled independently by the first two authors, with disagreements discussed and resolved.
“To evaluate annotator performance, we used a gold-standard dataset labeled independently by the first two authors. Any disagreements between their annotations were subsequently discussed and resolved to ensure consistency.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationown data§ H Synthetic Data Statistics and Distribution · p. 23The more uniform synthetic label distribution suggests a more balanced representation across categories and shows some divergence from human judgments.
“Compared to human-labeled data, the synthetic label distributions appear more uniform, suggesting a more balanced representation across the different categories, which shows some divergence from the human judgments.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ E Data License · p. 21NLPeer is distributed under the CC-BY-NC-SA 4.0 license.
“Specifically, NLPeer is distributed under CC-BY-NC-SA 4.0 license.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ 5 Results · p. 6For the Prometheus-2 model under the Score+Rationale setting, only 71% of samples had extractable outputs.
“For example, for the Prometheus2 model under the Score+Rationale setting, we could only extract outputs for 71% of the samples.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.
Interpretationreasoning§ Segmentation of Reviews into Comments · p. 10Trivial comments about layout or typos, or comments formulated as questions, might not always require a high verifiability.
“trivial comments about the layout or typos, or comments that are formulated as questions, might not always require a high verifiability.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone given§ 3.2.2 RevUtil Human · p. 4Three preliminary studies were conducted to identify high-performing annotators, recruiting 48 individuals.
“We conducted three preliminary studies to identify annotators with the highest performance on the task. In total, we recruited 48 individuals across these studies.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 3.2.2 RevUtil Human · p. 4All samples in the human dataset were annotated by three annotators: two recruited and the main author.
“For our human dataset, all samples are annotated by three annotators: two recruited and the main author of this work.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationcitation§ Limitations · p. 9Our developed aspects are core aspects for peer review, and our framework is transferable to other domains.
“we believe they are core aspects for peer review, and our framework is transferable to other domains.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Nature, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites CVPR, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites PLOS, 2025
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ H Synthetic Data Statistics and Distribution · p. 23The authors do not have ground truth for the synthetic labels.
“We note that we don't have ground truth for the synthetic labels; however, the synthetic labels on the FullAgreement and MajorityAgreement datasets show high agreement with human labels (see §3.2.3) while being generated using the same method used with the synthetic labels here.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisenone given§ E Data License · p. 21EMNLP 2023 and ICLR 2024-2025 reviews have the CC-BY 4.0 license.
“EMNLP 2023, and ICLR 2024-2025 reviews have CC-BY 4.0 license.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning§ 3.2.2 RevUtil Human · p. 4Annotators were selected who had obtained or were enrolled in a PhD in CS or IT to ensure familiarity with peer reviews, and only native English speakers were selected.
“We select annotators who have obtained or are currently enrolled in a PhD in CS or IT to ensure their familiarity with our peer reviews. Further, we select only native English speakers.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ B Review Segmentation · p. 19All the scripts and filters used are included in the code release.
“All the scripts and filters we used are included in our code release.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationreasoning§ 5 Results · p. 7The fine-tuned model's underperformance in claim detection F1 is likely due to GPT-4o's two-step approach for Verifiability compared to the single-step setup.
“The fine-tuned model outperforms GPT-4o in κ 2 , but underperforms in claim detection F1, likely due to GPT-4o's two-step approach for Verifiability compared to the single-step setup.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ 5 Results · p. 6Some models in the zero-shot evaluation failed to produce valid outputs for all samples.
“Some of the models in the zero-shot evaluation failed to produce valid outputs (outputs for all aspects, and with valid scores) for all samples.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ Segmentation of Reviews into Comments · p. 10Human labels are heavily dependent on the annotator.
“This makes the human labels heavily dependent on their annotator.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ 3.2.2 RevUtil Human · p. 4The two highest-performing annotators were selected for the main task.
“Annotator agreement was then measured against this finalized gold-standard, and the two highest-performing annotators were selected for the main task.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ F.1 Detailed Inter Annotator Agreements · p. 22Low agreement with Annotator Y on the F1 only leads to underestimating the agreement scores.
“We note that the low agreement with Annotator Y, on the F1, only leads to underestimating the agreement scores.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone given§ E Data License · p. 21Reviewer2 is distributed under an open Creative Commons license.
“Reviewer2 is distributed under an open Creative Commons license.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoning§ 3.2.2 RevUtil Human · p. 4Additional definitions for each score from 1-5 were provided to guide annotators.
“we provide additional definitions for each score from 1-5 to provide annotators with further guidance”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ F Manual Data Statistics · p. 21Extreme cases are relatively easier for annotators to agree on, while middle-range cases show lower agreement.
“We observe that most labels are concentrated around scores '5' and '1', indicating that extreme cases are relatively easier for annotators to agree on, while middle-range cases show lower agreement.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Interpretationreasoning§ M Aspects Correlation · p. 24The high correlation of Helpfulness with other aspects is expected because Helpfulness acts as a collective measurement for the other aspects.
“This high correlation is expected, as the Helpfulness aspect acts as a collective measurement for the other aspects.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data§ I.2 Evaluation · p. 24Due to occasional faulty outputs, evaluation results were computed based only on successfully parsed scores.
“Due to occasional faulty outputs (e.g., missing scores or failure to parse predictions correctly), there were cases where we could not extract all generated scores. In such instances, we computed evaluation results based only on the successfully parsed scores.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Without high-quality feedback, authors lack guidance on how to improve their work, leading to resubmissions without substantial changes and consequent inefficiencies in the scientific process.
“Yet, without high-quality feedback, authors lack the guidance on how to improve their work, leading to resubmissions without substantial changes and consequent inefficiencies in the scientific process.”
The paper relies on this without support.
Research into evaluating and improving systems for automated reviewing feedback is limited by a lack of evaluation data.
“While this experiment is promising, research into evaluating and improving systems for automated reviewing feedback is limited by a lack of evaluation data.”
The paper relies on this without support.
The data from prior review quality studies is not publicly available, so it cannot be used to develop or evaluate models for automatic review assessment.
“However, since the data from these studies is not publicly available, it cannot be used to develop or evaluate models for automatic review assessment.”
The paper relies on this without support.
LLM-generated reviews are often generic and flawed by hallucinations.
“While review generation can serve as a benchmark for LLMs, such reviews are often generic and flawed by hallucinations.”
The paper relies on this without support.
A major challenge in the development of LLM-generated reviews is evaluating their quality.
“A major challenge in their development is evaluating the quality of LLM-generated reviews.”
The paper relies on this without support.
Using closed-source models as judges is expensive and has limited reproducibility.
“However, using closed-source models as judges is expensive and has limited reproducibility.”
The paper relies on this without support.
We are the first to include rationales explaining our aspect scores.
“Albeit generated by GPT-4o, we are the first to include rationales explaining our aspect scores.”
The paper relies on this without support.
Review sections other than weaknesses and discussion, such as summary and strengths, hold limited value for authors to improve their manuscript.
“Other sections, such as summary and strengths, are primarily intended for meta-reviewers or chairs and hold limited value for the authors to improve their manuscript.”
The paper relies on this without support.
Assigning a single score to an entire review is challenging and prone to noise.
“assigning a single score to an entire review is challenging and prone to noise”
The paper relies on this without support.
Different review comments may vary with respect to the defined aspects.
“different review comments may vary with respect to our defined aspects (for example, some comments can be actionable, while others are not).”
The paper relies on this without support.
Collecting human annotations is costly.
“As collecting human annotations is costly”
The paper relies on this without support.
The capacity of the manual data is not enough to perform model fine-tuning.
“the capacity of our manual data is not enough to perform model fine-tuning”
The paper relies on this without support.
Manuscript drafts are confidential and should not be sent to potentially non-privacy-preserving services.
“Manuscript drafts are confidential and should therefore not be sent to potentially non-privacy-preserving services.”
The paper relies on this without support.
Sending manuscript drafts to non-privacy-preserving services violates ARR guidelines.
“This violates, for example, the ARR guidelines.”
The paper relies on this without support.
Using third-party services can be expensive, especially given high volumes of submissions at conferences.
“Moreover, using third-party services can be expensive, especially given high volumes of submissions at conferences.”
The paper relies on this without support.
Chat templates are primarily designed for multi-turn conversations, whereas the task involves single-turn response generation.
“However, these chat templates are primarily designed for multi-turn conversations, whereas our task involves single-turn response generation.”
The paper relies on this without support.
The 25 test samples are representative of the synthetic RevUtil data.
“Table 5: Human evaluation scores of the rationales of 25 test samples from the synthetic RevUtil data.”
“we manually evaluate the generated rationales by our best model on the same 25 test samples.”
The paper relies on this without support.
The first author's manual evaluation is a valid measure of rationale quality.
“8 Human evaluation conducted by the first author.”
The paper relies on this without support.
Actionability is important to authors because they seek clear guidance to improve their draft.
“It is important to authors as they seek clear guidance to help them improve their draft.”
The paper relies on this without support.
Grounding is crucial for authors because it helps them identify which section of their paper needs revision.
“This is crucial for authors, as it helps them identify which section of their paper needs revision.”
The paper relies on this without support.
Excluding cases with faulty outputs does not bias the evaluation results.
“Due to occasional faulty outputs (e.g., missing scores or failure to parse predictions correctly), there were cases where we could not extract all generated scores. In such instances, we computed evaluation results based only on the successfully parsed scores.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Claim of highest fine-tuned agreement across all aspects is too broad
MinorThe stated across-aspects ranking does not hold for every aspect. For example, Table 3 reports synthetic Actionability κ2 of 0.733 for fine-tuned DeepSeek-R1-7B in S+R mode, exceeding Llama-3.1-IT-8B's 0.726 in that mode.
“Fine-Tuning Fine-tuning greatly boosts performance compared to zero-shot, with Llama-3.1-IT8B achieving the highest agreement across all aspects (e.g., Actionability rising from 0.103 to 0.726).”
“| DeepSeek-R1-7B | S+R | Inst. | 0.733 | 0.453”
Cites
Part of this paper’s reference list could not be split into single entries.
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“We provide the aspect definition and in-context examples (Brown et al., 2020) in the prompt.”
Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit
Cohen, J. (1968) · Psychological Bulletin · doi:10.1037/h0026256 · Cited by 8,736 · cited 1× · see citing works
“Following previous work with a similar annotation scheme (Wang et al., 2024a), we use quadratic-weighted Kappa ( κ 2 ) (Cohen, 1968), an extension of Cohen’s Kappa that accounts for the magnitude of disagreements, making it suitable for ordinal data.”
THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED
WELCH, B. L. (1947) · Biometrika · doi:10.1093/biomet/34.1-2.28 · Cited by 4,135 · cited 1× · see citing works
“The results show that human-written reviews outperform LLM-generated ones across 3 out of 4 aspects with statistically significant differences measured by Welch’s t-test (Welch, 1947).”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2× · see citing works
“We conjecture that the increased performance on this aspect stems from the training regime of GPT-4, where Helpfulness has likely been incorporated during post-training, as has been reported for its predecessor InstructGPT (Ouyang et al., 2022).”
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 1× · see citing works
“Table 6 presents the results of this automatic evaluation using Rouge-L (Lin, 2004) and BERTScore (Zhang et al., 2020).”
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1× · see citing works
“We observed that providing GPT-4o with rationales in the in-context examples improved the quality of generated synthetic labels, which aligns with previous findings (Ye et al., 2024; Zheng et al., 2023).”
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817 · cited 1× · see citing works
“Table 18: Samples from the manually evaluated review rationales, along with their scores for Relevance, and Correctness.”
QLoRA: efficient finetuning of quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-0441 · Cited by 1,814 · see citing works
George, M. (1998) · Nursing Standard · doi:10.7748/ns.12.26.24.s36 · Cited by 1,615
Gemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.19786 · Cited by 1,156
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · cited 1× · see citing works
“Alternatively, closed-source models such as GPT-4 (Liu et al., 2023) or specialized LLM-as-a-judge models (Kim et al., 2024) can rate generated reviews.”
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928 · cited 1× · see citing works
“To enable multi-GPU support, we incorporate both the HF Accelerate library 13 and DeepSpeed-ZeRo (Rajbhandari et al., 2020 fine-tuning runs were conducted for one epoch us-ing the ‘bfloat16‘ datatype (Kalamkar et al., 2019).”
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 538 · see citing works
Bornmann, L., Haunschild, R., & Mutz, R. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00903-w · Cited by 532 · cited 1× · see citing works
“However, with the rapidly growing number of submissions to conferences and journals (Fire and Guestrin, 2019; Bornmann et al., 2020), the peer review process is Figure 1: An example from our RevUtil dataset.”
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · cited 1× · see citing works
“Previous research demonstrated that LoRA achieves comparable performance to full fine-tuning, while requiring significantly less time and compute (Dettmers et al., 2023; Ghosh et al., 2024).”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · cited 1× · see citing works
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
A Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386 · cited 1× · see citing works
“Comparing such reviews to human reference reviews using lexical overlap metrics is limited, since the references are diverse and the metrics do not align well with human judgments (Novikova et al., 2017; Reiter, 2018).”
Over-optimization of academic publishing metrics: observing Goodhart's Law in action
Fire, M., & Guestrin, C. (2019) · GigaScience · doi:10.1093/gigascience/giz053 · Cited by 351 · cited 1× · see citing works
“However, with the rapidly growing number of submissions to conferences and journals (Fire and Guestrin, 2019; Bornmann et al., 2020), the peer review process is Figure 1: An example from our RevUtil dataset.”
Effects of Editorial Peer Review: A Systematic Review
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002) · JAMA · doi:10.1001/jama.287.21.2784 · Cited by 347 · cited 1× · see citing works
“Besides serving as a high-quality filter, it provides explanations and constructive feedback to authors to improve their scientific contribution (Jefferson et al., 2002; Ross-Hellauer, 2017).”
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill SetsPreprint
Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., & Seo, M. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2307.10928 · Cited by 253 · cited 1× · see citing works
“We observed that providing GPT-4o with rationales in the in-context examples improved the quality of generated synthetic labels, which aligns with previous findings (Ye et al., 2024; Zheng et al., 2023).”
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 243 · cited 1× · see citing works
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
What is open peer review? A systematic review
Ross-Hellauer, T. (2017) · F1000Research · doi:10.12688/f1000research.11369.1 · Cited by 232 · cited 1× · see citing works
“Besides serving as a high-quality filter, it provides explanations and constructive feedback to authors to improve their scientific contribution (Jefferson et al., 2002; Ross-Hellauer, 2017).”
Unprofessional peer reviews disproportionately harm underrepresented groups in STEM
Silbiger, N. J., & Stubler, A. D. (2019) · PeerJ · doi:10.7717/peerj.8247 · Cited by 189 · cited 1× · see citing works
“Furthermore, unprofessional reviews can impact productivity and cause emotional distress, particularly for early-career researchers and those from underrepresented groups (Silbiger and Stubler, 2019; Hyland and Jiang, 2020).”
A Study of BFLOAT16 for Deep Learning Training
Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D., Banerjee, K., Avancha, S., Vooturi, D. T., Jammalamadaka, N., Huang, J., Yuen, H., et al. (2019) · arXiv · doi:10.48550/arxiv.1905.12322 · Cited by 169 · cited 2× · see citing works
“This high correlation is expected, as the Helpfulness aspect acts as a collective measurement for the other aspects.”
HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM
Wang, Z., Dong, Y., Zeng, J.-Q., et al. (2023) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2311.09528 · Cited by 161
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143 · cited 2× · see citing works
“To accelerate training, we employed FlashAtten-tion2 (Dao, 2024).”
DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process
Zhu, M.-J., Weng, Y., Yang, L.-Y., & Zhang, Y. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2503.08569 · Cited by 137 · cited 3×
“1-8B-Instruct (Dubey et al., 2024), DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-70B (DeepSeek-AI et al., 2025), Gemma-3-27b-IT (Kamath et al., 2025) representing generalist models, and SciTülu-7B (Wadden et al., 2024), SciLitLLM (Li et al., 2025), Deep-Review (Zhu et al., 2025), representing models speciali…”
Large Language Models Can Be Easily Distracted by Irrelevant Context
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2302.00093 · Cited by 134 · cited 1× · see citing works
“We find that the alignment with human data is better without the full paper context (likely due to the phenomenon that LLMs are easily distracted by too much irrelevant context (Shi et al., 2023)), except for Verifiability , which could benefit from additional context.”
Calderon, N., Reichart, R., & Dror, R. (2025) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2501.10970 · Cited by 127
Zephyr: Direct Distillation of LM AlignmentPreprint
Tunstall, L., Beeching, E., Lambert, N., Rajani, N. F., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16944 · Cited by 80 · see citing works
Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07183 · Cited by 73 · cited 1× · see citing works
“Increasingly, LLMs are also used to generate peer reviews (Liang et al., 2024a).”
“This work is antithetical to the spirit of research”: An anatomy of harsh peer reviews
Hyland, K., & Jiang, F. (. (2020) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2020.100867 · Cited by 66 · cited 1× · see citing works
“Furthermore, unprofessional reviews can impact productivity and cause emotional distress, particularly for early-career researchers and those from underrepresented groups (Silbiger and Stubler, 2019; Hyland and Jiang, 2020).”
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
Wadden, D., Shi, K., Morrison, J. D., Li, A., Naik, A., Singh, S., Barzilay, N., Lo, K., Hope, T., Soldaini, L., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2406.07835 · Cited by 50 · cited 2×
“1-8B-Instruct (Dubey et al., 2024), DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-70B (DeepSeek-AI et al., 2025), Gemma-3-27b-IT (Kamath et al., 2025) representing generalist models, and SciTülu-7B (Wadden et al., 2024), SciLitLLM (Li et al., 2025), Deep-Review (Zhu et al., 2025), representing models speciali…”
ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis
Wang, Q., Zeng, Q., Huang, L., Knight, K., Ji, H., & Rajani, N. F. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.44 · Cited by 38 · cited 1× · see citing works
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
Atla Selene Mini: A General Purpose Evaluation Model
Alexandru, A., Calvi, A., Broomfield, H., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2501.17195 · Cited by 33 · cited 1×
“1-8B (Alexandru et al., 2025).”
(2020) · doi:10.18653/v1/2020.findings-emnlp.112 · Cited by 32 · see citing works
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences
Stelmakh, I., Shah, N. B., Singh, A., & Daumé III, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i6.16610 · Cited by 24 · cited 1× · see citing works
“Several studies examined review quality by sur-veying authors, reviewers, and/or meta-reviewers on aspects such as negligence, fairness, and coverage (Khosla et al., 2012; Stelmakh et al., 2021; Goldberg et al., 2025).”
Peer reviews of peer reviews: A randomized controlled trial and other experiments
Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., & Shah, N. B. (2025) · PLOS ONE · doi:10.1371/journal.pone.0320444 · Cited by 18 · cited 3× · see citing works
“Several studies examined review quality by sur-veying authors, reviewers, and/or meta-reviewers on aspects such as negligence, fairness, and coverage (Khosla et al., 2012; Stelmakh et al., 2021; Goldberg et al., 2025).”
Automated Focused Feedback Generation for Scientific Writing Assistance
Chamoun, E., Schlichktrull, M., & Vlachos, A. (2024) · arXiv · doi:10.48550/arxiv.2405.20477 · Cited by 17 · cited 1× · see citing works
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
«I Do Not Have Time»—Is This the End of Peer Review in Public Health Sciences?
Künzli, N., Berger, A., Czabanowska, K., Lucas, R., Madarasova Geckova, A., Mantwill, S., & von dem Knesebeck, O. (2022) · Public Health Reviews · doi:10.3389/phrs.2022.1605407 · Cited by 17 · cited 1×
“Reviewers have less time to complete reviews; consequently, reviews suffer from poor quality and high variance (Rogers and Au-genstein, 2020; Künzli et al., 2022).”
Automated Focused Feedback Generation for Scientific Writing Assistance
Chamoun, E., Schlichtkrull, M., & Vlachos, A. (2024) · doi:10.18653/v1/2024.findings-acl.580 · Cited by 16 · see citing works
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
Kim, S., Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.08491 · Cited by 15 · cited 1× · see citing works
“We further evaluate fine-tuned LLM-Judges in the zero-shot setting, specifically Prometheus-2 (Kim et al., 2024), Flow-Judge-v0.”
PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews
Bharti, P. K., Navlakha, M., Agarwal, M., & Ekbal, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09662-3 · Cited by 12 · cited 2× · see citing works
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
PAICE, E. W. (2001) · Hospital Medicine · doi:10.12968/hosp.2001.62.3.2392 · Cited by 12 · see citing works
(2014) · doi:10.1080/00665983.2014.11771055 · Cited by 10
MARG: Multi-Agent Review Generation for Scientific Papers
D'Arcy, M., Hope, T., Birnbaum, L., & Downey, D. (2024) · arXiv · doi:10.48550/arxiv.2401.04259 · Cited by 9 · cited 1× · see citing works
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
HelpSteer2: Open-source dataset for training top-performing reward models
Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., & Kuchaiev, O. (2024) · arXiv · doi:10.48550/arxiv.2406.08673 · Cited by 7 · cited 1×
“While many human preference datasets have been collected in a pair-wise setting, where humans rank two (or more) responses, point-wise data has also shown promising results (Wang et al., 2024a,b).”
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
Self-citationAnnotation Sensitivity: Training Data Collection Methods Affect Model Performance
Kern, C. A., Eckman, S., Beck, J., Chew, R., Ma, B., & Kreuter, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.992 · Cited by 6 · see citing works
A Closer Look at the Limitations of Instruction Tuning
Ghosh, S., Evuru, C. K. R., Kumar, S., S, R., Aneja, D., Jin, Z., Duraiswami, R., & Manocha, D. (2024) · arXiv · doi:10.48550/arxiv.2402.05119 · Cited by 5 · cited 1×
“Previous research demonstrated that LoRA achieves comparable performance to full fine-tuning, while requiring significantly less time and compute (Dettmers et al., 2023; Ghosh et al., 2024).”
Automatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · arXiv · doi:10.48550/arxiv.2311.11967 · Cited by 5 · cited 1×
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
The lack of theory is painful: Modeling Harshness in Peer Review Comments
Verma, R., Roychoudhury, R., & Ghosal, T. (2022) · Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2022.aacl-main.67 · Cited by 5 · cited 1×
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
Automatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.684 · Cited by 5 · see citing works
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 4 · cited 2× · see citing works
“Recently, ICLR 2025 experimented with providing automatic feedback on reviews (Thakkar et al., 2025).”
Reviewer2: Optimizing Review Generation Through Prompt Generation
Gao, Z., Brantley, K., & Joachims, T. (2024) · arXiv · doi:10.48550/arxiv.2402.10886 · Cited by 3 · cited 1× · see citing works
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
Annotation Sensitivity: Training Data Collection Methods Affect Model Performance
Kern, C., Eckman, S., Beck, J., Chew, R., Ma, B., & Kreuter, F. (2023) · arXiv · doi:10.48550/arxiv.2311.14212 · Cited by 2 · cited 1×
“To maintain the quality of the annotations, we split our data into batches of 100-200 samples each, to ensure our annotators do not suffer from annotation fatigue (Kern et al., 2023).”
NLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2022) · arXiv · doi:10.48550/arxiv.2211.06651 · Cited by 1 · see citing works
Self-citationLazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews
Purkayastha, S., Li, Z., Lauscher, A., Qu, L., & Gurevych, I. (2025) · arXiv · doi:10.48550/arxiv.2504.11042 · Cited by 1 · cited 1×
“Purkayastha et al. (2025) also assess reviews, but with a different focus: They propose to detect specific “lazy” reviewing practices, i.e., comments that follow simple reviewing heuristics (as defined by the ACL reviewing guidelines (ARR, 2024b)).”
Self-citationBornmann, L., Haunschild, R., & Mutz, R. (2020) · arXiv · doi:10.48550/arxiv.2012.07675
Calderon, N., Reichart, R., & Dror, R. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.782
Flagged first, then by citations
1 of the 63 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- [8]8. Aoife Cahill, Leon Derczynski, and Kokil Jaidka. 2024. ACL Policy on Publication Ethics. https: //www.aclweb.org/adminwiki/index.php/ACL\_ Policy\_on\_Publication\_Ethics#Guidelines\_ for\_Generative\_Assistance\_in\_Peer\_Review . [Online; accessed 04-May-2025].
- [12]12. CVPR. 2025. CVPR 2025 Reviewer Guidelines . https://cvpr.thecvf.com/Conferences/ 2025/ReviewerGuidelines . [Online; accessed 02July-2025].
- [17]17. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, and 21 others. 2024. LLMs assist NLP researchers: Critique paper (meta-)reviewing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 5081-5099, Miami, Florida, USA. Association for Computational Linguistics.
- [20]20. study of peer review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 50495073, Toronto, Canada. Association for Computational Linguistics.
- [27]27. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of Large Language Models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net.
- [35]35. Aditya Khosla, Derek Hoiem, and Serge J. Belongie. 2012. Analysis of Reviews for CVPR 2012.
- [36]36. Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing finegrained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net.
Cited by
Sorted by citations
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y.-L., Hu, T., Ma, Y., Zhang, K., Patwardhan, M. S., & Cohan, A. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2604.27924 · Cited by 10
“As the field advances, future research must prioritize transparent and ethical deployment of automated reviews.”
Insights from the ICLR Peer Review and Rebuttal Process
Kargaran, A. H., Nikeghbal, N., Yang, J., & Ousidhoum, N. D. (2025) · arXiv.org · doi:10.48550/arxiv.2511.15462 · Cited by 10
“…et al., 2024), detected misinformed or deficient review points (Ryu et al., 2025; Zhang et al., 2025b), assessed the utility of reviews for authors (Sadallah et al., 2025), explored argumentative perspectives to identify disagreements between reviewers and scoring discrepancies across submission…”
RbtAct: Rebuttal as Supervision for Actionable Review Feedback GenerationPreprint
Wu, S., Ma, Y., Zhao, Y.-L., Hu, T., Jiang, O., Patwardhan, M. S., & Cohan, A. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2603.09723 · Cited by 6 · see citing works
“…rebuttal as a natural reward model to serve as a signal for action-able review generation, targeting the gap identified by prior evaluations that LLM-generated reviews are often generic and not sufficiently tied to revision (Sadallah et al., 2025; Hosseini and Horbach, 2023; Chamoun et al., 2024).”
Peerispect: Claim Verification in Scientific Peer Reviews
Ghorbanpour, A., Sadeghian, S., Daghighfarsoodeh, A., et al. (2026) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808368 · Cited by 5
AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
Zhou, K., & Tan, C. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2603.07019 · Cited by 4
“While there are efforts to evaluate review quality and usefulness (Ryu et al., 2025; Sadallah et al., 2025), no prior study has applied checklist-based evaluation to automatically assess rebuttal quality.”
Reward Modeling for Scientific Writing Evaluation
Şahinuç, F., Dutta, S., & Gurevych, I. (2026) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2601.11374 · Cited by 4
“Additionally, we incorporate the RevUtil dataset (Sadallah et al., 2025).”
InfluentialSelf-citationRubricReviewer: From Direct Critique to Objective and Comprehensive Rubric-Driven Peer ReviewPreprint
Guo, S., Hu, W., Zhao, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.00005
“Training-free agents tend to produce superficial critiques that gravitate toward neutral assessments (Zhu et al., 2025a; Li et al., 2025), while training-based re-viewers inherit the noise and uneven coverage of their human supervision (Sadallah et al., 2025; Li et al., 2026a).”
GoodPoint: Learning Constructive Scientific Paper Feedback from Author ResponsesPreprint
Mun, J., Jung, C., Zhou, X.-H., Kim, H., & Sap, M. (2026) · arXiv.org · doi:10.48550/arxiv.2604.11924 · Cited by 3 · see citing works
“In addition to our targeted quality dimensions, validity and action, specificity and helpfulness are included for a more comprehensive evaluation, following previous work (Sadallah et al., 2025; Liang et al., 2024).”
Sandström, U., & Thelwall, M. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0048 · Cited by 1 · see citing works
“Smaller open weights LLMs seem to be powerful enough for this task (Sadallah et al., 2025), allowing more security than when using cloud-based versions.”
Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers
Yang, Y., Thelwall, M., & He, G. (2026) · arXiv · doi:10.48550/arxiv.2609.01895
“Studies of review utility similarly emphasize actionability, grounding, specificity, verifiability, and helpfulness (P. Bharti et al., 2022; P. K. Bharti et al., 2024; Purkayastha et al., 2025; Sadallah et al., 2025).”
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality
Chen, Z., Yang, Z., Ma, Y., Backes, M., & Zhang, Y. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1170
“Annotators with prior peer-review experience are asked to identify the most constructive [50], most specific [53], and most faithful [7] review for each paper and to report their confidence level for each judgment.”
Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance
Purkayastha, S., Wan, Q., Lauscher, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.10118
“…(ARR) guidelines: lazy thinking , where oversimplified heuristics lead reviewers to dismiss submissions (Rogers and Augen-stein, 2024), and specificity issues , where vague phrasing such as “X is not clear” replaces precise feedback such as “X is not clear because of Y” (Sadallah et al., 2025).”
Self-citationHow Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices
Zhang, L.-L., Zhou, D.-J., Huang, C.-Y., et al. (2026) · doi:10.48550/arxiv.2610.11877
“RevUtil evaluates author-oriented actionability, grounding/specificity, verifiability, and helpfulness using 1,430 human-annotated review points (Sadallah et al., 2025).”
Peer review should constrain evaluative authority
Wang, H.-R., Spinde, T., Habereder, I., Lu, C.-S., & Echizen, I. (2026) · doi:10.48550/arxiv.2610.28646
“Synthetic data may broaden the training signal [67], but optimization and validation still rely on heterogeneous human judgments, evaluative standards, and heuristic metrics rather than verified criteria ground truth [22, 17, 68, 31, 32].”
Arawjo, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.35815
“Published papers at AI, NLP, and HCI venues now use LLM judges as metrics in controlled experiments, run statistical tests directly over their outputs (e.g., [7, 23, 41, 46, 88, 95, 98, 102, 110, 115, 122]), and produce 𝑝 -values that scientific claims rest upon.”
Artificial Intelligence in Research and Academic Publication
Zhang, Y., & Xie, Q. (2026) · IEEE Intelligent Systems · doi:10.1109/mis.2026.3726702
“9 Such data are used to train machine-learning models that give scores and revise suggestions to human reviews.”
Uncertainty-Aware Generation and Decision-Making Under Ambiguity
Daheim, N., & Gurevych, I. (2026) · arXiv.org · doi:10.48550/arxiv.2606.30578
“Recently, there have also been works to evaluate scientific reviews according to a set of pre-defined criteria (Sadallah et al., 2025; ¸Sahinuç et al., 2026).”
Self-citationJudgment-Grounded Expansion for Peer Review Generation
Lu, S., Qu, L.-Z., & Gurevych, I. (2026) · arXiv.org · doi:10.48550/arxiv.2606.23233
“Future work could design nonconfor-mity scores tailored to peer review by incorporating review-specific factors, such as actionability, verifiability, and helpfulness (Sadallah et al., 2025).”
Self-citationFeedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?
Yoshida, K., Kuroki, S., Imajuku, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.29857
“The same concern can be expressed in different words, and two comments can look similar on the surface or in embedding space while differing in concern, specificity, or actionability (Sadallah et al., 2025).”
Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time
Kuznetsov, I., Nayak, R., Rozovskaya, A., & Gurevych, I. (2026) · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2601.15172 · Cited by 1 · see citing works
“Each score is calculated based on review items and ranges be-tween 1 (low) and 5 (high), with labels defined in Sadallah et al. (2025).”
InfluentialSelf-citationReviewScore: Misinformed Peer Review Detection with Large Language Models
Ryu, H., Jang, D., Lee, H. S., Jeong, J., Kim, G., Cho, D., Chu, G., Hwang, M., Jang, H., Kim, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.21679
“Guo et al. (2023), Sadallah et al. (2025), Purkayastha et al. (2025), and Ou et al. (2025) introduce specific and objective criteria but those target narrow scope of reviews.”
- Authors
- Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T.
- ORCID
- Baumgärtner, Tim
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.04484
- OpenAlex
- W7080325606
- Cited by
- 0
- References
- 63
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- arXiv (Cornell University)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Keywords
- peer review, reviewer comments, author feedback, synthetic data generation, rationale generation, fine-tuned language models, actionability, helpfulness, verifiability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
