MARG: Multi-Agent Review Generation for Scientific Papers
Open accessAbstract
We study the ability of LLMs to generate feedback for scientific papers and develop MARG, a feedback generation approach using multiple LLM instances that engage in internal discussion. By distributing paper text across agents, MARG can consume the full text of papers beyond the input length limitations of the base LLM, and by specializing agents and incorporating sub-tasks tailored to different comment types (experiments, clarity, impact) it improves the helpfulness and specificity of feedback. In a user study, baseline methods using GPT-4 were rated as producing generic or very generic comments more than half the time, and only 1.7 comments per paper were rated as good overall in the best baseline. Our system substantially improves the ability of GPT-4 to generate specific and helpful feedback, reducing the rate of generic comments from 60% to 29% and generating 3.7 good comments per paper (a 2.2x improvement).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing evaluations of AI-assisted peer review mainly focus on independent tasks or on final review quality and human alignment.
“existing evaluations of AI-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as D' Arcy et al. 2024
Section 6 (Automated evaluation; page unavailable) evaluates generated reviews by their overlap with human reviews, and Section 2.1 (Review generation; page unavailable) lists review-score prediction among prior tasks. The paper does not establish that existing evaluations mainly focus on overall review quality or the accuracy of final decisions. (LLM, conf=0.78)
Experimental — accuracy not yet measuredExisting multi-agent systems model review diversity only at a superficial level by assigning agents to predefined roles or text segments rather than explicitly modeling the evaluation criteria that experts apply, and thus fail to capture how human reviewers selectively assess different intellectual aspects of a paper based on their expertise.
“Many multi-agent systems simulate human roles at a surface level, missing the underlying criteria that lead experts to assess complementary intellectual aspects of a paper.”
From DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning (2026) · cites this paper as D'Arcy et al., 2024
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredAutomatic review generation methods neglect the fine-grained decision-making factors in humans and LLMs that shape complex assessment.
“While these methods offer valuable insights into downstream performance in peer review, they neglect the fine-grained decision-making factors in humans and LLMs that shape the complex assessment.”
From STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond (2025) · cites this paper as D'Arcy et al., 2024
Section 2.1 (page unavailable) says earlier review-generation methods are limited in scope or rely on templates rather than nuanced free-form comments; Section 1 describes the reasoning involved in assessing a paper. These passages support the concern that prior methods miss aspects of complex assessment, but they do not say those methods neglect fine-grained decision-making factors in both humans and LLMs. (LLM, conf=0.78)
Experimental — accuracy not yet measuredSubsequent work has produced a wide range of full review-generation systems, including multi-agent discussion, standardized review generation through fine-tuning, bias-aware pipelines, deep-thinking review generation, and RL-trained reviewers.
“Subsequent work has produced a wide range of full review-generation systems, including multi-agent discussion [D'Arcy et al., 2024], standardized review generation through fine-tuning [Yu et al., 2024], bias-aware pipelines [Tyser et al., 2024], deep-thinking review generation [Zhu et al., 2025], and RL-trained reviewers [Zeng et al., 2025].”
From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as D'Arcy et al., 2024
[40] proposed a multi-agent review generation system that improved the specificity and helpfulness of feedback provided compared to GPT-4, reducing the rate of generic comments from 60% to 29%.
“Building off of this work, [40] proposed a multi-agent review generation system that improved the specificity and helpfulness of feedback provided compared to GPT-4, reducing the rate of generic comments from 60% to 29%.”
From Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 (2025) · cites this paper as [40]
Few studies compare process-variable decision value, evaluate chain consistency, and examine interventional sensitivity of final decisions to intermediate process variables within a unified framework.
“Few studies compare process-variable decision value, evaluate chain consis-tency, and examine the interventional sensitivity of final decisions to intermediate process variables within a unified framework.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as D' Arcy et al. 2024
The Experiments and evaluation and 8.1 Scope sections discuss generating and assessing scientific-review comments, not a survey of studies using the claimed unified framework. Searches for “chain consistency,” “interventional sensitivity,” and process-variable decision value found no passage in our copy addressing whether few studies examine these together. (LLM, conf=0.74)
Experimental — accuracy not yet measuredPrior work largely treats critical text assessment as a black box, leaving unanswered how experts arrive at their judgement.
“Yet, existing work treats critical text assessment as a black box problem, limiting interpretability and human-AI collaboration.”
From STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond (2025) · cites this paper as D' Arcy et al., 2024
Section 2.1 (page unavailable) describes limitations of earlier review-generation methods, including template-based comments, limited input size, and use of a single prompt. It does not say that prior work treats critical text assessment as a black box, leaves expert judgment unexplained, or limits interpretability and human–AI collaboration. Searches for those concepts found no passage addressing that claim. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 21
- Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. LongBench: A Bilingual, Mul- titask Benchmark for Long Context Understanding. ArXiv:2308.14508 [cs]. · DOI · Paper page
- Karim Benharrak, Tim Zindulka, Florian Lehmann, Hendrik Heuer, and Daniel Buschek. 2023. Writer- Defined AI Personas for On-Demand Feedback Gen- eration. ArXiv:2309.10433 [cs. · arXiv
- Nell K Duke and P David Pearson. 2009. Effective prac- tices for developing reading comprehension. Journal of education, 189(1-2):107-122. Publisher: SAGE Publications Sage CA: Los Angeles, CA.
- Raymond Fok and Daniel S Weld. 2023. What can't large language models do? the future of ai-assisted academic writing. In In2Writing Workshop at CHI 2023.
- Frederic Gmeiner and Nur Yildirim. 2023. Dimen- sions for Designing LLM-based Writing Support. In In2Writing Workshop at CHI 2023.
- Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. 2023. MetaGPT: Meta Programming for Multi-Agent Collaborative Framework. ArXiv:2308.00352 [cs. · arXiv
- A. U. Kalnins, K. Halm, and M. Castillo. 2015. Screen- ing for Self-Plagiarism in a Subspecialty-versus- General Imaging Journal Using iThenticate. Amer- ican Journal of Neuroradiology, 36(6):1034-1038. Publisher: American Journal of Neuroradiology Sec- tion: Editorial Perspectives. · DOI
- Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. 2023. Can large language models provide useful feedback on research papers? A large- scale empirical analysis. ArXiv:2310.01783 [cs]. · arXiv
- Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. ArXiv:2307.03172 [cs. · arXiv
- Cerstin Mahlow. 2023. Writing Tools: Looking Back to Look Ahead. In In2Writing Workshop at CHI 2023. ArXiv:2303.17894 [cs. · arXiv
- Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Sim- ulacra of Human Behavior. ArXiv:2304.03442 [cs. · arXiv
- Zejiang Shen, Tal August, Pao Siangliulue, Kyle Lo, Jonathan Bragg, Jeff Hammerbacher, Doug Downey, Joseph Chee Chang, and David Sontag. 2023. Be- yond summarization: Designing ai support for real- world expository writing tasks. In In2Writing Work- shop at CHI 2023.
- Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. 2023a. Recursively Summarizing Enables Long- Term Dialogue Memory in Large Language Models. ArXiv:2308.15022 [cs]. · arXiv
- Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020a. Re- viewRobot: Explainable Paper Review Generation based on Knowledge Synthesis. arXiv:2010.06119 [cs]. ArXiv: 2010.06119. · arXiv
- Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b. Linformer: Self-Attention with Linear Complexity. ArXiv:2006.04768 [cs, stat]. · arXiv
- Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023b. Unleash- ing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self- Collaboration. ArXiv:2307.05300 [cs]. · arXiv
- Matthew Welsh, Daniel Navarro, and Steve Begg. 2011. Number Preference, Precision and Implicit Confi- dence. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33. Issue: 33.
- Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. ArXiv:2109.10862 [cs]. · arXiv
- Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: En- abling Next-Gen LLM Applications via Multi-Agent Conversation Framework. ArXiv:2308.08155 [cs. · arXiv
- Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catan- zaro. 2023. Retrieval meets Long Context Large Language Models. ArXiv:2310.03025 [cs]. · arXiv
- Weizhe Yuan and Pengfei Liu. 2022. KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-Training. In Proceedings of the First MiniCon Conference.
Cited by
Sorted by citations
Lee, J., Lee, J., & Yoo, J. (2025) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2025.22.4 · Cited by 44 · see citing works
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34 · see citing works
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 33 · see citing works
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 26 · see citing works
Huang, S., Wang, Q., Lu, W. P., Liu, L., Xu, Z., & Huang, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104225 · Cited by 22 · see citing works
Pu, K., Feng, K. J. K., Grossman, T., Hope, T., Mishra, B. D., Latzke, M., Bragg, J., Chang, J. C., & Siangliulue, P. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714057 · Cited by 18 · see citing works
Self-citationDetecting LLM-generated peer reviews
Rao, V. S., Kumar, A., Lakkaraju, H., & Shah, N. B. (2025) · PLOS One · doi:10.1371/journal.pone.0331871 · Cited by 14 · see citing works
“Finally, with the growing interest in using LLMs as reviewers [LS23, LZC + 23, DHBD24], it is important to acknowledge the risk that authors themselves could exploit indirect prompt injection to bias LLM-generated reviews in their favor.”
Using large language models to facilitate academic work in the psychological sciences
Sohail, A., & Zhang, L. (2025) · Current Psychology · doi:10.1007/s12144-025-07438-2 · Cited by 18 · see citing works
A large-scale randomized study of large language model feedback in peer review
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01188-x · Cited by 11 · see citing works
Zhong, Y., Leng, Y., Gu, Z., Guo, S., Li, P., Das, S., Liu, Y., Wan, J., & Liu, Y. (2025) · Journal of Materials Chemistry A · doi:10.1039/d5ta05224f · Cited by 11 · see citing works
LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions
Liao, Z., Antoniak, M., Cheong, I., Cheng, E. Y.-Y., Lee, A.-H., Lo, K., Chang, J. C., & Zhang, A. X. (2024) · arXiv · doi:10.48550/arxiv.2411.05025 · Cited by 7 · see citing works
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
Self-citationCan LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 4 · see citing works
PeerQA: A Scientific Question Answering Dataset from Peer Reviews
Baumgärtner, T., Briscoe, T., & Gurevych, I. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.22 · Cited by 3 · see citing works
Chang, Y., Li, Z., Zhang, H., Kong, Y., Wu, Y., So, H. K.-H., Guo, Z., Zhu, L., & Wong, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.790 · Cited by 2 · see citing works
Kim, S., Yoon, D., Gashteovski, K., Suk, J., Baek, J., Aggarwal, P., Wu, I., Zaverkin, V., Petkoski, S., Schrider, D. R., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.20668 · Cited by 1 · see citing works
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
Dycke, N., Zečević, M., Kuznetsov, I., Suess, B., Kersting, K., & Gurevych, I. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1107 · Cited by 1 · see citing works
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1476 · Cited by 1 · see citing works
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
“For example, Yu et al. (2024) proposed an LLM-based framework to automatically generate reviews, while D’Arcy et al. (2024) used multiple LLMs that engage in internal discussions to produce reviews.”
Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Yuan, S., Zhang, X., Ni, W., Xiao, T., Di, S., Zhu, J., Jiang, Z., Tan, R., Chen, L., & Ren, K. (2026) · arXiv · doi:10.48550/arxiv.2609.05947
“Existing evaluations mainly focus on performance on independent tasks (Kang et al. 2018; Zhou, Chen, and Yu 2024) or on the final review quality and human alignment of process-based reviewers (D’Arcy et al. 2024; Weng et al. 2025; Zhu et al. 2025; Garg et al. 2025).”
CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Deng, H., Ke, X., Li, Y., Hu, R., Huang, D., Wong, D. F., Wang, Y., Liu, X., & Zhang, M. (2026) · arXiv · doi:10.48550/arxiv.2605.07905
RbtAct: Rebuttal as Supervision for Actionable Review Feedback Generation
Wu, S., Ma, Y., Zhao, Y., Hu, T., Jiang, O., Patwardhan, M., & Cohan, A. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1696
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning
Zou, Z., Ansari, A., Zhang, D. C., Lee, D., & Yin, W. (2026) · arXiv · doi:10.48550/arxiv.2601.07611
“We observe that top-performing systems—GPT-4o and multi-agent approaches (Reviewer2, AgentReview, MARG, and DIAGPaper )—exhibit low overlap with these in-valid reviews for a desirable reason: they tend to generate valid, well-grounded weaknesses that do not align with invalid human reviews.”
InfluentialDo Methods Support the Claims? Intra-Paper Verification for Peer Review
Ballakuraya, R. S., Mahyari, A., & Srinivasan, A. (2026) · arXiv · doi:10.48550/arxiv.2607.26066
“To enhance feedback comprehensiveness and consistency, systems like AgentReview [5] and MARG [17] simulate multi-agent re-viewer deliberations to mirror real-world committee dynamics.”
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Seo, M., Baek, J., Lee, S., & Hwang, S. J. (2025) · arXiv · doi:10.48550/arxiv.2504.17192
Large language models for computational social science
Xu, F., Shao, C., Chen, L., Zeng, Q., Chen, Z., Sukiennik, N., Wang, J., Gao, C., Wang, H., Lian, J., et al. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02539-9
- Authors
- D'Arcy, M., Hope, T., Birnbaum, L., & Downey, D.
- Affiliations
- Hebrew University of Jerusalem (IL)
- Northwestern University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2401.04259
- OpenAlex
- W4390784029
- Cited by
- 9
- References
- 0
- Citations by year
- 2026: 2 · 2025: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2006851)
- Tencent
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- multi-agent LLM systems, LLM agents, agent specialization, GPT-4, user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
