CoChat
Sign Up
PreprintCited by 23Academic Writing and PublishingTopic ModelingPublishing and Scholarly Communication

Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions

Open access
Zhang, Daoze, Bao, Zhijian, Du, Sihang, Zhao, Zhiyi, Zhang, Kuangling, Bao, Dezheng, Yang, Yang·2025·arXiv
DOI 10.48550/arxiv.2505.07920arXiv 2505.07920
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer review is a critical component of scientific progress in the fields like AI, but the rapid increase in submission volume has strained the reviewing system, which inevitably leads to reviewer shortages and declines review quality. Besides the growing research popularity, another key factor in this overload is the repeated resubmission of substandard manuscripts, largely due to the lack of effective tools for authors to self-evaluate their work before submission. Large Language Models (LLMs) show great promise in assisting both authors and reviewers, and their performance is fundamentally limited by the quality of the peer review data. However, existing peer review datasets face three major limitations: (1) limited data diversity, (2) inconsistent and low-quality data due to the use of revised rather than initial submissions, and (3) insufficient support for tasks involving rebuttal and reviewer-author interactions. To address these challenges, we introduce the largest consistency-ensured peer review and rebuttal dataset named Re^2, which comprises 19,926 initial submissions, 70,668 review comments, and 53,818 rebuttals from 24 conferences and 21 workshops on OpenReview. Moreover, the rebuttal and discussion stage is framed as a multi-turn conversation paradigm to support both traditional static review tasks and dynamic interactive LLM assistants, providing more practical guidance for authors to refine their manuscripts and helping alleviate the growing review burden. Our data and code are available in https://anonymous.4open.science/r/ReviewBench_anon/.

AI snapshot

Written from the paper’s full text.

TL;DR

Re² is a large consistency-ensured peer review dataset built from OpenReview, covering initial submissions, reviews, rebuttals, discussions, score changes, meta-reviews, and decisions. It frames rebuttal-discussion as multi-turn conversations and shows fine-tuning on the data improves LLM performance on review and rebuttal tasks.

Key findings

  • Re² contains 19,926 initial submissions, 70,668 review comments, and 53,818 rebuttals from 24 conferences and 21 workshops, spanning 45 venues from 2017 to 2025; the Re²-Rebuttal subset has 14,830 initial submissions paired with 53,818 rebuttal/discussion turns.
  • In acceptance prediction, SEA-E reaches 66.24 accuracy and DeepReviewer-14B reaches 67.89 accuracy, while zero-shot LLaMA-3.1-8B and Qwen2.5-7B accept all papers (100.00 recall, 60.19 accuracy).
  • Fine-tuning on Re² improves score prediction: LLaMA-3.1-8B MAE drops from 1.961 zero-shot to 1.141 SFT, and Qwen2.5-7B MAE drops from 2.043 to 1.182.
  • For review generation, fine-tuned LLaMA-3.1-8B ranks first in BLEU and ROUGE-L; in EmbedCos, fine-tuned LLaMA-3.1-8B and Qwen2.5-7B improve by 49.9% and 20.5% over third place, respectively. For rebuttal-discussion, fine-tuned LLaMA-3.1-8B achieves the best performance on four of five similarity metrics and best overall LLM-as-judge performance.

Methods

  • Data were crawled via the OpenReview API from 68 conferences (2013–2025), then filtered to 24 conferences and 21 workshops from 2017 onward. Initial submissions were recovered using conference deadlines and “Revision History” pages; PDFs were converted to LaTeX and Markdown with Doc2X.
  • Re²-Rebuttal was constructed by retaining papers with publicly accessible rebuttal stages, concatenating consecutive same-role responses into single turns, using DeepSeek-R1 to merge response titles, removing reminder-only responses, and inserting global responses as referential context in multi-turn dialogues.
  • Experiments cover acceptance prediction, score prediction, review generation, and rebuttal-discussion conversation. Re²-Review uses 1,000 test papers and Re²-Rebuttal uses 500 test papers; LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct are fine-tuned with LoRA for one epoch at learning rate 1×10⁻⁴ on four NVIDIA A100 80G GPUs, with BLEU, ROUGE-L, BERTScore, EmbedCos, MAE/MSE, and LLM-as-judge metrics.

Limitations

  • The paper states that experiments are limited to textual and tabular components of papers, excluding visual elements such as figures, even though figures are included in the released dataset; future work may use vision-language models.
  • The text makes evident that Re²-Rebuttal only includes papers whose reviewer-author rebuttal stage is publicly accessible, which may limit coverage and introduce selection bias.
  • The dataset is built from OpenReview and public review records, so it inherits the availability and formatting constraints of that platform.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.07920

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 16 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Rebuttal-included datasets still suffer from limitations such as original manuscript versions, limited data diversity and scale, or unreliability of simulated data.

    “However, although these datasets include rebuttal content, they still suffer from several limitations, such as the issue of original manuscript versions, limited data diversity and scale, or the unreliability of simulated data.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Kennard et al. [2021] source

      The Conclusion (p. unavailable) says publicly available peer-review data is concentrated in artificial intelligence, limiting its scope and generalizability; this supports the data-diversity concern. The dataset comparison in §3 gives dataset sizes, but does not establish limited scale as a shortcoming. The passages found do not identify original manuscript versions or unreliable simulated data as limitations. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Jin et al. [2024] source

      Section 2.3 (Data Selection; page unavailable) describes how AgentReview selected a diverse set of submissions, while the Limitation section discusses constraints on its own simulation. Neither passage assesses rebuttal-included datasets for original manuscript versions, limited diversity or scale, or unreliable simulated data. Searches for those dataset-specific limitations found no passage addressing them. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Wu et al. [2022]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisereasoning

    The majority of existing peer review datasets are based only on review data from the ICLR conference.

    “the majority of existing datasets [Kennard et al., 2021, Bharti et al., 2021, Wu et al., 2022, Zhang et al., 2022, Zhou et al., 2024, Weng et al., 2024, Jin et al., 2024, Zhu et al., 2025] are only based on the review data from the ICLR conference”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Kennard et al., 2021 source
    • cites Bharti et al., 2021 source
    • cites Wu et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    The median review scores of each conference are all between 5 and 7.

    “It can be demonstrated that the median scores of each conference are all between 5 and 7.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Fine-tuning on Re2 training data significantly improves models' review ability, as shown by notable decreases in MAE and MSE for score prediction.

    “After finetuning, both models show more reasonable outputs, with notable decreases in MAE and MSE for score prediction, demonstrating that the finetuning on our training data significantly improves the review ability.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    The Re2 dataset unifies all data into a consistent format to facilitate broad accessibility for peer researchers.

    “By overcoming the huge heterogeneity in data storage formats across different conferences and years, we unify all data into a consistent format to facilitate broad accessibility for peer researchers.”
    Quote in the paperSubstantial claim
  • May combine two claims. 7
  • Classified as probably not a substantial claim. 5

  • May combine two claims.

    Premisereasoning

    Most static review datasets could not guarantee consistency between their paper content and the papers under review, nor did they include rebuttals, making it difficult to support research about reviewer-author discussions.

    “However, although the above works have made progress in scholarly peer review, most of these datasets could not guarantee the consistency between their paper content and those under review, nor did they include rebuttals, making it difficult to support the research about the reviewers-authors discussions.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Kang et al. [2018] source

      The Introduction (p. 1647) says access to peer-review data had been limited, and §2.1 describes how PeerRead collected papers and reviews. Neither passage assesses whether most other datasets match the papers actually reviewed or include rebuttals. Searches for rebuttals, paper-version mismatches, and prior datasets did not find support for those claims or the asserted difficulty of studying reviewer–author discussions. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Dycke et al. [2022] source

      The Introduction (p. 1) says most existing datasets use task-specific formats, while A.1 (p. 13) says some datasets provide only submitted drafts. Neither passage says that most datasets fail to match paper content to the versions under review. Section 2.1 (p. 2) mentions author responses but does not say most datasets omit rebuttals or that their omission hinders research on reviewer–author discussions. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Zhang et al. [2022] source

  • May combine two claims.

    Premisecitation

    Existing peer review datasets have limited data diversity, being based on a few conferences and relatively small amounts of data.

    “existing peer review datasets face three major limitations: (1) limited data diversity”
    • Also stated in: § 1 Introduction · plain
      “From the perspective of data diversity, the paper sources of these existing datasets are limited to a few conferences, and the amount of data is also relatively small.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Kennard et al., 2021 source

      The Conclusion (page unavailable) says almost all publicly available peer review data comes from artificial intelligence, supporting the claim of limited diversity. The Data Source and Annotation section identifies DISAPERE’s own data as coming from one conference, but our copy does not establish that existing datasets generally draw from only a few conferences or contain relatively small amounts of data. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Bharti et al., 2021 source
    • cites Wu et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    The few existing datasets that involve rebuttals handle the rebuttal and discussion data in a coarse and insufficient manner and fail to guarantee consistency, falling far short of supporting research that treats rebuttal and discussion as a well-defined multi-turn conversation task.

    “For the few datasets [Kennard et al., 2021, Wu et al., 2022, Jin et al., 2024, Tan et al., 2024] involving rebuttals, they often handle the rebuttal and discussion data in a coarse and insufficient manner and also fail to guarantee the consistency of the data, falling far short of supporting research that treats rebuttal and discussion as a well-defined multi-turn conversation task.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Kennard et al., 2021 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites Wu et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Jin et al., 2024 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Most papers in Re2 range in length from 6,000 to 16,000 tokens, while the majority of reviews are distributed between 300 and 1,100 tokens.

    “Most papers range in length from 6,000 to 16,000 tokens, while the majority of reviews are distributed between 300 and 1,100 tokens.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ICLR and NeurIPS hold the highest proportion of the number of papers and reviews among conferences in Re2.

    “Among them, ICLR and NeurIPS hold the highest proportion of the number of papers and reviews.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    LLaMA-OpenReviewer-8B and CycleReviewer-8B have higher precision but lower F1, suggesting they are more strict and conservative in accepting papers.

    “LLaMA-OpenReviewer-8B and CycleReviewer-8B have higher precision but lower F1, suggesting they are more strict and conservative in accepting papers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ICLR has almost no peaks in its violin plot of review scores.

    “Additionally, by counting the peaks in each violin, we find that ICLR has almost no peaks”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The Re2 dataset is the largest real-world peer review dataset, featuring the widest range of conferences and the most complete review stages.

    “Second, our dataset is the largest real-world peer review dataset, which features the widest range of conferences and the most complete review stages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The rebuttal-discussion process constitutes a typical multi-turn conversation task.

    “This rebuttal-discussion process constitutes a typical multi-turn conversation task, which holds great potential and research value.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The experiments are limited to the textual and tabular components of papers, excluding visual elements such as figures.

    “our experiments are limited to the textual and tabular components of papers, excluding visual elements such as figures (even though they are included in the released dataset).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    LLMs can be used by authors as a pre-submission self-evaluation mechanism and can be integrated into the peer review process to support reviewers, as demonstrated by the ICLR 2025 conference.

    “LLM-based review tools can be used by authors as a pre-submission selfevaluation mechanism, allowing authors to identify and improve the weaknesses in the manuscripts, and thereby enhancing the work quality and reducing resubmission rates.Additionally, LLMs can be directly integrated into the peer review process to support reviewers in generating more specific and constructive feedback -as demonst”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites ICLR, 2025

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

11 of 11

  • The paper relies on this without support.

    The most crucial knowledge foundation for training language models with peer review capabilities is real-world high-quality paper review data.

    “For the training of language models with peer review capabilities, the most crucial knowledge foundation that determines the upper bound of the model performance is the real-world high-quality paper review data.”

  • The paper relies on this without support.

    Repeated resubmission of substandard manuscripts is a major contributor to submission overload, largely due to lack of effective tools for authors to self-evaluate before submission.

    “another key factor in this overload is the repeated resubmission of substandard manuscripts, largely due to the lack of effective tools for authors to self-evaluate their work before submission.”
    “another major contributor to the submission overload is the repeated resubmission of manuscripts that fall short of quality standards.This trend is often driven by the lack of effective tools for authors to objectively assess and improve their manuscripts prior to submission, resulting in multiple rounds of submission for the same study.”

  • The paper relies on this without support.

    The LLM-as-judge evaluation using LLaMA-3.1-8B-Inst with the five specified aspects provides a valid assessment of the quality of generated dialogue statements.

    “In the rebuttal-discussion conversation task, we use LLaMA-3.1-8B-Inst to evaluate the quality of generated dialogue statements.”
    “Here we instruct the judge to focus on five aspects to evaluate the results generated by the reviewer model, including accuracy, clarity, constructiveness, completeness, and quality.”

  • The paper relies on this without support.

    The lexical and semantic similarity metrics used (BLEU, ROUGE-L, BERTScore, EmbedCos) are valid measures of the quality of rebuttal-discussion conversation.

    “As shown in Tab.6, we show the comparison between the four metrics: BLEU, ROUGE-L, BERTScore, and EmbedCos.”
    “In our experiment, we report the ROUGE-L F1 score, which is the harmonic mean of ROUGE-L Precision and ROUGE-L Recall.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Re2 is not the largest dataset by the paper's own counts

    Major

    The paper calls Re2 the largest real-world peer review dataset, but Table 1 lists ReviewMT with more papers and review comments: 26,841 and 92,017, versus Re2's 19,926 and 70,668.

  • RR-MCQ is assigned conflicting source years

    Minor

    Table 1 identifies RR-MCQ's source as ICLR 2017, whereas the related-work section says RR-MCQ was drawn from the ICLR 2023 review-rebuttal forum.

  • Whiskers are both said to represent and not represent observed extremes

    Minor

    The appendix first says the whiskers represent the observed maximum and minimum, then says neither whisker represents the actual maximum or minimum.

  • ReviewMT is described as involving rebuttals but listed with none

    Minor

    The text identifies Tan et al. [2024] as a dataset involving rebuttals, while Table 1 lists zero rebuttals for ReviewMT.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. ICLR. Assisting iclr 2025 reviewers with feedback. https://blog.iclr.cc/2024/10/09/ iclr2025-assisting-reviewers/, 2025.
  2. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.

Cited by

Sorted by citations

Authors
Zhang, D., Bao, Z., Du, S., Zhao, Z., Zhang, K., Bao, D., & Yang, Y.
Affiliations
Zhejiang University (CN)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.07920
arXiv
2505.07920
OpenAlex
W6947845154
Semantic Scholar
6f1cf87439bfd84a0ec302d5362f12075267fb81
Cited by
0
References
21
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
Keywords
peer review, review datasets, multi-turn dialogue, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review