DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Open accessAbstract
Neha Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, Andrew McCallum. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pragmatic labeling has been previously explored in Hua et al. 2019, Kuznetsov et al. 2022, and Kennard et al. 2022.
“Pragmatic labeling has been previously explored in (Hua et al., 2019;Kuznetsov et al., 2022;Kennard et al., 2022) and is usually cast as a discourse labeling task on free-form peer review text.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Kennard et al., 2022
The vast majority of NLP research in peer review so far focused on a few machine learning conferences that make their data available through OpenReview.net.
“yet the vast majority of NLP research in peer review so far focused on a few machine learning conferences that make their data available through the OpenReview.netplatform (e.g.Kennard et al., 2022;Shen et al., 2022).”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Kennard et al., 2022
Aspect-based analysis of peer reviews can help identify aspects of the manuscript that were insufficiently or vaguely covered by the reviewers.
“In particular, aspect-based analysis of peer reviews (Chakraborty et al., 2020; Kennard et al., 2022) can help identify aspects of the manuscript that were insufficiently or vaguely covered by the reviewers.”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Kennard et al., 2022
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., & Chang, K.-W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1323 · Cited by 829 · see citing works
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 4× · see citing works
“Hua et al. (2019) introduced the study dis- 127 course structure in peer review by annotating argu- 128 mentative propositions in the AMPERE dataset 129 with a set of labels tailored to the peer review 130 domain (EVALUATION, REQUEST, FACT, REFER- 131 ENCE, and QUOTE).”
InfluentialAPE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
Cheng, L., Bing, L., Yu, Q., Lu, W., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.569 · Cited by 58 · see citing works
Does My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1129 · Cited by 48 · see citing works
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.07082
Flagged first, then by citations
Cited by
Sorted by citations
Bauer, E., Greisel, M., Kuznetsov, I., Berndt, M., Kollar, I., Dresel, M., Fischer, M. R., & Fischer, F. (2023) · British Journal of Educational Technology · doi:10.1111/bjet.13336 · Cited by 130 · see citing works
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Narayanan Venkit, P., Zhang, N., Srinath, M., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.292 · Cited by 23 · see citing works
NLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.277 · Cited by 19 · see citing works
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
An Analysis of Tasks and Datasets in Peer Reviewing
Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A. (2024) · Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024) · doi:10.18653/v1/2024.sdp-1.24 · Cited by 6 · see citing works
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16253 · Cited by 2 · see citing works
“Other datasets focus on specific aspects of peer reviews, such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et…”
BackgroundCan AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions
Zhang, D., Bao, Z., Du, S., Zhao, Z., Zhang, K., Bao, D., & Yang, Y. (2025) · arXiv · doi:10.48550/arxiv.2505.07920
“…the huge differences in the availability and formatting of the review information across different conferences, the majority of existing datasets [Kennard et al., 2021, Bharti et al., 2021, Wu et al., 2022, Zhang et al., 2022, Zhou et al., 2024, Weng et al., 2024, Jin et al., 2024, Zhu et al.,…”
InfluentialPRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
Loc, N. P. P., La Viet, T. H., Khanh, T. T., Nguyen, D. A., Pham, T. A. N., Nguyen, T., Chawla, N. V., Buntine, W., Wong, K.-S., Doan, K. D., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26730
- Authors
- Kennard, N., O’Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., & McCallum, A.
- Affiliations
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Published
- 2022
- Pages
- 1234-1249
- OpenAlex
- W4287854541
- Cited by
- 17
- References
- 6
- Citations by year
- 2026: 6 · 2025: 2 · 2024: 1 · 2023: 8
- FWCI
- 56.22
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1955567, 1763618, 1922090)
- University of Southern California
- Chan Zuckerberg Initiative
- Defense Advanced Research Projects Agency (FA8750-17-C-0106, FA8750)
- Advanced Research Projects Agency
- Office of Naval Research
- Topics
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- peer review, discourse structure, academic writing, research evaluation, annotated corpus
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
