Automated Focused Feedback Generation for Scientific Writing Assistance
Open accessAbstract
Scientific writing is a challenging task, particularly for novice researchers who often rely on feedback from experienced peers. Recent work has primarily focused on improving surface form and style rather than manuscript content. In this paper, we propose a novel task: automated focused feedback generation for scientific writing assistance. We present SWIF$^{2}$T: a Scientific WrIting Focused Feedback Tool. It is designed to generate specific, actionable and coherent comments, which identify weaknesses in a scientific paper and/or propose revisions to it. Our approach consists of four components - planner, investigator, reviewer and controller - leveraging multiple Large Language Models (LLMs) to implement them. We compile a dataset of 300 peer reviews citing weaknesses in scientific papers and conduct human evaluation. The results demonstrate the superiority in specificity, reading comprehension, and overall helpfulness of SWIF$^{2}$T's feedback compared to other approaches. In our analysis, we also identified cases where automatically generated reviews were judged better than human ones, suggesting opportunities for integration of AI-generated feedback in scientific writing.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior systems often assume that identified weaknesses are inherently correct.
“Second, prior systems often assume that identified weaknesses are inherently correct (Yu et al., 2024;Chamoun et al., 2024), overlooking the fact that even human reviews can be biased, or mistaken.”
From DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning (2026) · cites this paper as Chamoun et al., 2024
The Evaluation setup and Human vs LLM-generated reviews sections describe models hallucinating weaknesses, but neither says prior systems assume identified weaknesses are correct. The Human-written reviews section says human reviews are not treated as a gold standard; it does not establish the claimed assumption by prior systems. Searched for that assumption and for descriptions of how prior systems handle identified weaknesses. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 23
- Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical Error Correction: A Survey of the State of the Art. Computational Linguistics, 49(3):643-701. · DOI
- Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based Sentiment Anal- ysis of Scientific Reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL '20, page 207-216, New York, NY, USA. Association for Computing Machinery. · DOI · Paper page
- Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers.
- Wanyu Du, Zae Myung Kim, Vipul Raheja, Dhruv Ku- mar, and Dongyeop Kang. 2022. Read, Revise, Re- peat: A System Demonstration for Human-in-the- loop Iterative Text Revision. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 96-108, Dublin, Ireland. Association for Computational Linguistics. · DOI
- Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. NLPeer: A Unified Resource for the Computational Study of Peer Review. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049- 5073, Toronto, Canada. Association for Computa- tional Linguistics. · DOI · Paper page
- Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. 2021. Text Editing by Command. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5259-5274, Online. Association for Computational Linguistics. · DOI
- Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. PLOS ONE, 17(1):1-29. · DOI · Paper page
- Serge P.J.M. Horbach and Willem Halffman. 2018. The changing forms and expectations of peer review. Re- search Integrity and Peer Review, 3.
- Takumi Ito, Tatsuki Kuribayashi, Masatoshi Hidaka, Jun Suzuki, and Kentaro Inui. 2020. Langsmith: An Interactive Academic Text Revision System. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 216-226, Online. Association for Computational Linguistics. · DOI
- Takumi Ito, Tatsuki Kuribayashi, Hayato Kobayashi, Ana Brassard, Masato Hagiwara, Jun Suzuki, and Kentaro Inui. 2019. Diamonds in the Rough: Gener- ating Fluent Sentences from Early-Stage Drafts for Academic Writing Assistance. In Proceedings of the 12th International Conference on Natural Language Generation, pages 40-53, Tokyo, Japan. Association for Computational Linguistics. · DOI
- Léane Jourdan, Florian Boudin, Richard Dufour, and Nicolas Hernandez. 2023. Text revision in Scientific Writing Assistance: An Overview. 13th International Workshop on Bibliometric-enhanced Information Re- trieval (BIR 2023), pages 22-36.
- Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Papers), pages 1647-1661, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI · Paper page
- Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. 2014. Peer Review in Scientific Publications: Bene- fits, Critiques, & A Survival Guide. EJIFCC, 25:227- 43.
- Simon Knight, Sophie Abel, Antonette Shibani, Yoong Goh, Rianne Conijn, Andrew Gibson, Sowmya V., Elena Cotos, Agnes Sandor, and Simon Bucking- ham Shum. 2020. Are You Being Rhetorical? A Description of Rhetorical Move Annotation Tools and Open Corpus of Sample Machine-Annotated Rhetorical Moves. Journal of Learning Analytics, 7:138-154. · DOI
- Thomas S. Kuhn. 1962. The Structure of Scientific Revolutions. University of Chicago Press, Chicago.
- Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and Resubmit: An Inter- textual Model of Text-based Collaboration in Peer Review. Computational Linguistics, 48(4):949-986.
- Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFar- land, and James Zou. 2023. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. In arXiv preprint arXiv:2310.01783. · arXiv
- Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023. MOPRD: A multi- disciplinary open peer review dataset. Neural Com- puting and Applications, 35(34):24191-24206. · DOI · Paper page
- Rodrigo Frassetto Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. CoRR, abs/1910.14424. · arXiv
- Zachary Robertson. 2023. GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study.
- Antonette Shibani, Ratnavel Rajalakshmi, Faerie Mat- tins, Srivarshan Selvaraj, and Simon Knight. 2023. Visual representation of co-authorship with GPT-3: Studying human-machine interaction for effective writing. In Proceedings of the 16th International Conference on Educational Data Mining, pages 183- 193. International Educational Data Mining Society. · DOI
- Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020. Re- viewRobot: Explainable paper review generation based on knowledge synthesis. In Proceedings of the 13th International Conference on Natural Lan- guage Generation, Dublin, Ireland. Association for Computational Linguistics.
- Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? J. Artif. Int. Res., 75. · DOI · Paper page
Cited by
Sorted by citations
CoachGPT: A Scaffolding-based Academic Writing Assistant
Chen, F., Veng, S., Wilson, J., Li, X., & Fang, H. (2025) · Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3726302.3730143 · Cited by 6 · see citing works
AAAR-1.0: Assessing AI's Potential to Assist Research
Lou, R., Xu, H., Wang, S., Du, J., Kamoi, R., Lu, X., Xie, J., Sun, Y., Zhang, Y., Ahn, J. J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.22394 · Cited by 3 · see citing works
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
“Systems developed by Weng et al. (2025) train LLM-based reviewer agents to support automated research workflows, and Chamoun et al. (2024) propose feedback tools to help authors improve their scientific writing.”
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Various approaches have been put forward, testing LLM and agentic systems to generate reviews (Wang et al., 2020; Chamoun et al., 2024; D’Arcy et al., 2024; Gao et al., 2024; Liang et al., 2024b; Zhu et al., 2025).”
DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning
Zou, Z., Ansari, A., Zhang, D. C., Lee, D., & Yin, W. (2026) · arXiv · doi:10.48550/arxiv.2601.07611
“Second, prior systems often assume that identified weaknesses are inherently correct (Yu et al., 2024; Chamoun et al., 2024), overlooking the fact that even human reviews can be biased, or mistaken.”
- Authors
- Chamoun, E., Schlichktrull, M., & Vlachos, A.
- Affiliations
- University of Cambridge (GB)
- ORCID
- Vlachos, Andreas
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 9742-9763
- arXiv
- 2405.20477
- OpenAlex
- W4402683796
- Cited by
- 17
- References
- 0
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 2
- FWCI
- 70.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion, cc-by)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/Y009800/1)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- peer review, large language models, manuscript revision, automated feedback, novice researchers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
