DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Open accessAbstract
At the foundation of scientific evaluation is the labor-intensive process of peer review. This critical task requires participants to consume vast amounts of highly technical text. Prior work has annotated different aspects of review argumentation, but discourse relations between reviews and rebuttals have yet to be examined. We present DISAPERE, a labeled dataset of 20k sentences contained in 506 review-rebuttal pairs in English, annotated by experts. DISAPERE synthesizes label sets from prior work and extends them to include fine-grained annotation of the rebuttal sentences, characterizing their context in the review and the authors' stance towards review arguments. Further, we annotate every review and rebuttal sentence. We show that discourse cues from rebuttals can shed light on the quality and interpretation of reviews. Further, an understanding of the argumentative strategies employed by the reviewers and authors provides useful signal for area chairs and other decision makers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The few existing datasets that involve rebuttals handle the rebuttal and discussion data in a coarse and insufficient manner and fail to guarantee consistency, falling far short of supporting research that treats rebuttal and discussion as a well-defined multi-turn conversation task.
“For the few datasets [Kennard et al., 2021, Wu et al., 2022, Jin et al., 2024, Tan et al., 2024] involving rebuttals, they often handle the rebuttal and discussion data in a coarse and insufficient manner and also fail to guarantee the consistency of the data, falling far short of supporting research that treats rebuttal and discussion as a well-defined multi-turn conversation task.”
From Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions (2025) · cites this paper as Kennard et al., 2021
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredRebuttal-included datasets still suffer from limitations such as original manuscript versions, limited data diversity and scale, or unreliability of simulated data.
“However, although these datasets include rebuttal content, they still suffer from several limitations, such as the issue of original manuscript versions, limited data diversity and scale, or the unreliability of simulated data.”
From Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions (2025) · cites this paper as Kennard et al. [2021]
The Conclusion (p. unavailable) says publicly available peer-review data is concentrated in artificial intelligence, limiting its scope and generalizability; this supports the data-diversity concern. The dataset comparison in §3 gives dataset sizes, but does not establish limited scale as a shortcoming. The passages found do not identify original manuscript versions or unreliable simulated data as limitations. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting peer review datasets have limited data diversity, being based on a few conferences and relatively small amounts of data.
“existing peer review datasets face three major limitations: (1) limited data diversity”
From Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions (2025) · cites this paper as Kennard et al., 2021
The Conclusion (page unavailable) says almost all publicly available peer review data comes from artificial intelligence, supporting the claim of limited diversity. The Data Source and Annotation section identifies DISAPERE’s own data as coming from one conference, but our copy does not establish that existing datasets generally draw from only a few conferences or contain relatively small amounts of data. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe majority of existing peer review datasets are based only on review data from the ICLR conference.
“the majority of existing datasets [Kennard et al., 2021, Bharti et al., 2021, Wu et al., 2022, Zhang et al., 2022, Zhou et al., 2024, Weng et al., 2024, Jin et al., 2024, Zhu et al., 2025] are only based on the review data from the ICLR conference”
From Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions (2025) · cites this paper as Kennard et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 4× · see citing works
“Hua et al. (2019) introduced the study dis- 127 course structure in peer review by annotating argu- 128 mentative propositions in the AMPERE dataset 129 with a set of labels tailored to the peer review 130 domain (EVALUATION, REQUEST, FACT, REFER- 131 ENCE, and QUOTE).”
Influential
Flagged first, then by citations
Research works printed but not linked: 17
- Ajjour, Wei-Fan Chen, Johannes Kiesel, Hen- ning Wachsmuth, and Benno Stein. 2017. Unit seg- mentation of argumentative texts. In Proceedings of the 4th Workshop on Argument Mining, pages 118- 128, Copenhagen, Denmark. Association for Compu- tational Linguistics. · DOI
- Jianzhu Bao, Bin Liang, Jingyi Sun, Yice Zhang, Min Yang, and Ruifeng Xu. 2021. Argument pair extrac- tion with mutual guidance and inter-sentence relation graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3923-3934, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
- Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analysis of scientific reviews. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020. · DOI · Paper page
- Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000-7011, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Liying Cheng, Tianyu Wu, Lidong Bing, and Luo Si. 2021. Argument pair extraction via attention-guided multi-layer multi-cross encoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6341-6353, Online. Association for Computational Linguistics. · DOI
- Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37-46.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
- Michael Fromm, Evgeniy Faerman, Max Berrendorf, Siddharth Bhargava, Ruoxia Qi, Yao Zhang, Lukas Dennert, Sophia Selle, Yang Mao, and Thomas Seidl. 2020. Argument mining driven analysis of peer- reviews. arXiv preprint arXiv:2012.07743. · arXiv
- Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my rebuttal matter? insights from a major NLP conference. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1274-1290, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
- Ivan Habernal and Iryna Gurevych. 2017. Argumenta- tion mining in user-generated web discourse. Com- putational Linguistics, 43(1):125-179. · DOI
- Tristan Miller, Maria Sukhareva, and Iryna Gurevych. 2019. A streamlined method for sourcing discourse- level argumentation annotations from the crowd. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1790-1796, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
- Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. As- sociation for Computational Linguistics. · DOI
- Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
- David Soergel, Adam Saunders, and Andrew McCallum. 2013. Open scholarship and peer review: a time for experimentation.
- Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. · DOI · Paper page
- Dietrich Trautmann, Johannes Daxenberger, Christian Stab, Hinrich Schütze, and Iryna Gurevych. 2020. Fine-grained argument unit recognition and classifi- cation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9048-9056.
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using · DOI · Paper page
Cited by
Sorted by citations
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions
Zhang, D., Bao, Z., Du, S., Zhao, Z., Zhang, K., Bao, D., & Yang, Y. (2025) · arXiv · doi:10.48550/arxiv.2505.07920
- Authors
- Kennard, N., O'Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 1234-1249
- arXiv
- 2110.08520
- OpenAlex
- W4287854541
- Cited by
- 17
- References
- 1
- Citations by year
- 2026: 6 · 2025: 2 · 2024: 1 · 2023: 8
- FWCI
- 56.22
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1955567, 1763618, 1922090)
- University of Southern California
- Chan Zuckerberg Initiative
- Defense Advanced Research Projects Agency (FA8750-17-C-0106, FA8750)
- Advanced Research Projects Agency
- Office of Naval Research
- Topics
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- peer review, discourse structure, academic writing, research evaluation, annotated corpus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
