ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews
Open accessAbstract
Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, Doug Downey. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reviewer comments often initiate exchanges in which authors clarify evidence, defend methodological choices, or revise claims.
“Work on review-rebuttal interaction shows that reviewer comments often initiate exchanges in which authors clarify evidence, defend methodological choices, or revise claims (Cheng et al., 2020;D'Arcy et al., 2024).”
From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as D'Arcy et al., 2024
D'Arcy et al. (2024) operationalizes iterative refinement with an explicit final refinement stage where a separate multi-agent group assesses each comment for validity and clarity, deciding whether to revise or prune it.
“D' Arcy et al. (2024) operationalizes this with an explicit, final refinement stage.After aspect-specific agents generate initial comments, a separate multi-agent group is convened to assess each comment for validity and clarity, collaboratively deciding whether to revise or prune it.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as D'Arcy et al., 2024
Section 4 (Dataset Construction) describes manual identification of actionable review comments and alignment with paper edits, followed by synthetic labeling for a larger dataset. Section 3 (Task Definitions) defines comment–edit alignment and edit generation. Neither those passages nor searches for a final refinement stage and a separate multi-agent group describe agents assessing comments for validity and clarity or deciding whether to revise or prune them. (LLM, conf=0.76)
Experimental — accuracy not yet measuredMARG's leader-worker architecture, where specialized expert agents critique specific aspects such as experiments, clarity, and impact, helps produce more targeted, helpful feedback.
“D' Arcy et al. (2024) propose MARG, a framework utilizing a leader-worker architecture where specialized expert agents are tasked with critiquing specific aspects of a paper, such as its experiments, clarity, and impact.This division of labor helps produce more targeted, helpful feedback.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as D'Arcy et al., 2024
The Introduction identifies this paper’s contributions as the ARIES dataset and tasks for aligning review comments with edits and generating edits. Section 8.1 mentions a separately cited automated feedback system, but our copy does not describe MARG, a leader-worker architecture, specialized agents, or whether that division of labor improves feedback. Searches for MARG and those architectural details found no supporting passage. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 13
- Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers.
- Takumi Ito, Tatsuki Kuribayashi, Masatoshi Hidaka, Jun Suzuki, and Kentaro Inui. 2020. Langsmith: An Interactive Academic Text Revision System. arXiv:2010.04332 [cs]. ArXiv: 2010.04332. · arXiv
- Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural crf model for sentence alignment in text simplification. In Proceed- ings of the Association for Computational Linguistics (ACL).
- Zae Myung Kim, Wanyu Du, Vipul Raheja, Dhruv Ku- mar, and Dongyeop Kang. 2022. Improving Iterative Text Revision by Learning Where to Edit from Other Revision Tasks. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9986-9999, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
- Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, and Ahmed H. Awadal- lah. 2022. On Improving Summarization Fac- tual Consistency from Natural Language Feedback. ArXiv:2212.09968 [cs]. · arXiv
- Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. EdiT5: Semi- Autoregressive Text Editing with T5 Warm-Start. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2126-2138, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
- Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, To- moya Mizumoto, Jun Suzuki, and Kentaro Inui. 2022. Towards Automated Document Revision: Grammat- ical Error Correction, Fluency Edits, and Beyond. ArXiv:2205.11484 [cs]. OpenAI. 2023. Gpt-4 technical report. · arXiv
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learn- ing Library. In Advances in Neural Information Pro- cessing Systems, volume 32.
- Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEdIT: Text Editing by Task-Specific Instruction Tuning. ArXiv:2305.09857 [cs]. · arXiv
- Machel Reid and Graham Neubig. 2022. Learning to model editing processes. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2022, pages 3822-3832, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Be- yond. Foundations and Trends in Information Re- trieval, 3(4):333-389. Publisher: Now Publishers Inc. Hanover, MA, USA.
- Radim Řehůřek and Petr Sojka. 2010. Software Frame- work for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45-50, Val- letta, Malta. ELRA.
Cited by
Sorted by citations
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136 · see citing works
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34 · see citing works
Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks
Zhou, R., Chen, L., & Yu, K. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/48d359hjdvog · Cited by 28 · see citing works
Large language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15 · see citing works
SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers
Singh, S., Sarkar, N., & Cohan, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1163 · Cited by 7 · see citing works
Blodgett, S. L., Curry, A. C., Dev, S., Madaio, M., Nenkova, A., Yang, D., & Xiao, Z. (2024) · doi:10.18653/v1/2024.hcinlp-1 · Cited by 3 · see citing works
An Interactive Co-Pilot for Accelerated Research Ideation
Nigam, H., Patwardhan, M. S., Vig, L., & Shroff, G. M. (2024) · Workshop on Bridging Human-Computer Interaction and Natural Language Processing (HCI+NLP) · doi:10.18653/v1/2024.hcinlp-1.6 · Cited by 2 · see citing works
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
A Tale of Two Revisions: Summarizing Changes Across Document Versions
T.y.s.s, S., Modani, N., & Saxena, A. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.190 · Cited by 1 · see citing works
AutoRef: Generating Refinements of Reviews Given Guidelines
Chitnis, S., Patwardhan, M. S., goa.bits-pilani.ac.in, A. G. B.-P. A. I. A., goa.bits-pilani.ac.in, T. G. B.-P. A. I. T., Vig, L., & Shroff, G. M. (2024) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2024.sdp-1.17 · Cited by 1 · see citing works
Intelligent Support for Manuscript Authors Using Term Clouds in Scientific Reviews
Латыпова, В. А. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-77411-9_18 · Cited by 1 · see citing works
CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?
Ou, J., Walden, W. G., Sanders, K., Jiang, Z., Sun, K., Cheng, J., Jurayj, W., Wanner, M., Liang, S., Morgan, C., et al. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1185 · Cited by 1 · see citing works
Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers
Yang, Y., Thelwall, M., & He, G. (2026) · arXiv · doi:10.48550/arxiv.2609.01895
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Wu, S., Ma, Y., Zhao, Y., Hu, T., Jiang, O., Patwardhan, M., & Cohan, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1696
CASIMIR: A Corpus of Scientific Articles Enhanced with Multiple Author-Integrated Revisions
Jourdan, L., Boudin, F., Hernandez, N., & Dufour, R. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/4aa2twhyqaqa
GPT's Performance in Identifying Outcome Changes on ClinicalTrials.govPreprint
Ying, X., Vorland, C. J., Qureshı, R., Brown, A. W., Halil Kilicoglu, B., Saldanha, I. J., DeVito, N., & Mayo‐Wilson, E. (2024) · doi:10.31222/osf.io/npvwr
Madeira, C., Dopke, L., Losch, G., Magnaguagno, M. C., & Griebler, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39898-7_18
- Authors
- D’Arcy, M., Ross, A., Bransom, E., Kuehl, B., Bragg, J., Hope, T., & Downey, D.
- Affiliations
- Massachusetts Institute of Technology (US)
- Allen Institute for Artificial Intelligence (US)
- Hebrew University of Jerusalem (IL)
- Northwestern University (US)
- Venue
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2024
- Pages
- 6985-7001
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4402671845
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2
- FWCI
- 36.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (2033558)
- Topics
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Keywords
- peer review, corpus construction, academic writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
