Envisioning the Future of Peer Review: Investigating LLM-Assisted Reviewing Using ChatGPT as a Case Study
Open accessAbstract
Peer review ensures research quality but faces growing challenges. Large Language Models (LLMs) offer potential solutions, yet most research has focused on independent AI-generated reviews rather than human-AI collaboration. We conducted a within-subject experiment with 24 HCI reviewers to evaluate LLM-assisted peer review’s impact on review efficiency, quality, workload, and user perceptions. Analysing subjective ratings, objective completion data, qualitative interviews, and chat history, we found that ChatGPT significantly reduces workload but does not substantially shorten review time or improve review quality. Reviewers valued its support for summarisation, information retrieval, idea generation, and confidence-building but noted challenges in content verification, interaction inefficiencies, and the need for human oversight. Our findings contribute to the broader conversation on the role of technology in scholarly publishing, offering design insights for supportive review systems that complement rather than replace human expertise, and highlighting the importance of responsible integration in academic peer review.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing studies primarily evaluate whether LLMs can generate reviews, summarize manuscripts or imitate reviewer behaviour.
“Existing studies primarily evaluate whether LLMs can generate reviews, summarize manuscripts or imitate reviewer behaviour (Chen et al. 2025;Zhou et al. 2024;Farber 2025;Russo et al. 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Chen et al. 2025
The Introduction says prior peer-review research has focused largely on generating review-like feedback, and Related Work §2.2 describes a study comparing GPT-4 feedback with human reviews. The paper mentions summarisation as an LLM capability and as a use valued by participants, but does not say that existing peer-review studies primarily evaluated manuscript summarisation. (LLM, conf=0.84)
Experimental — accuracy not yet measuredChatGPT-assisted reviewing reduced perceived workload and improved support for summarization, information retrieval and idea generation, although improvements in review quality and review duration remained relatively limited.
“ChatGPT-assisted reviewing reduced perceived workload and improved support for summarization, information retrieval and idea generation, although improvements in review quality and review duration remained relatively limited”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Chen et al. 2025
Current studies mainly investigate either generic AI reviewers or AI-human collaboration.
“Current studies mainly investigate either generic AI reviewers or AI-human collaboration (Sarker et al. 2024;Chen et al. 2025;Ebadi et al. 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Chen et al. 2025
Much of the current literature focuses on isolated AI-generated reviews or single-reviewer assistance.
“Much of the current literature focuses on isolated AI-generated reviews or single-reviewer assistance (Chen et al. 2025;Sun et al. 2024;Farber 2025).”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Chen et al. 2025
Chen et al. (2025) found that LLM assistance was particularly useful for summarization, information retrieval and idea generation, although improvements in review quality and completion time remained limited.
“The authors of Chen et al. (2025) found that LLM assistance was particularly useful for summarization, information retrieval and idea generation, although improvements in review quality and completion time remained limited.”
From Who Reviews the Reviewer? A Multi‐Agent LLM Architecture With Meta‐Review Synthesis for Editorial Peer Review (2026) · cites this paper as Chen et al. (2025)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 59 · Cited by 5
Research works printed but not linked: 11
- [1][1] 2014. The NIPS experiment. Retrieved November 23, 2021 from http://blog.mrtz.org/2014/12/15/the-nips-experiment.html Go to Citation Google Scholar
- [2][2] 2022. Peer-review crisis creates problems for journals and scholars. Retrieved December 2, 2022 from https://www.insidehighered.com/news/2022/06/13/peer-review-crisis-creates-problems-journals-and-scholars Go to Citation Google Scholar
- [3][3] 2023. CHI 2024 — Papers track, post-round one outcomes report. Retrieved January 15, 2024 from https://chi2024.acm.org/2023/12/15/chi-2024-papers-track-post-round-one-outcomes-report-2/ Go to Citation Google Scholar
- [18][18] Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN (2023). Go to Citation Google Scholar
- [30][30] Juhani Iivari. 2016. How to improve the quality of peer reviews? Three suggestions for system-level changes. Communications of the Association for Information Systems 38, 1 (2016), 12. Go to Citation Google Scholar
- [32][32] Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. 2014. Peer review in scientific publications: benefits, critiques, & a survival guide. Ejifcc 25, 3 (2014), 227. Go to Citation Google Scholar
- [38][38] Pengyuan Li, Xiangying Jiang, and Hagit Shatkay. 2019. Figure and caption extraction from biomedical documents. Bioinformatics 35, 21 (2019), 4381–4388. Go to Citation Google Scholar
- [40][40] Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. 2024. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI 1, 8 (2024), AIoa2400196. Go to Citation Google Scholar
- [54][54] Paul Ralph. 2016. Practical suggestions for improving scholarly peer review quality and reducing cycle times. Communications of the association for information systems 38, 1 (2016), 13. Go to Citation Google Scholar
- [58][58] Nihar B Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. 2018. Design and analysis of the NIPS 2016 review process. Journal of machine learning research (2018). Go to Citation Google Scholar
- [71][71] Johannes Zagermann. 2024. CHI 2025 — Papers Track, post-review report (Round 1). Retrieved Feburary 24, 2025 from https://chi2025.acm.org/chi-2025-papers-track-post-review-report-round-1/ Go to Citation Google Scholar
- Authors
- Chen, S., Brumby, D., & Cox, A.
- Affiliations
- University College London (GB)
- Venue
- Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work
- Publisher
- ACM
- Published
- 2025-06-22
- Pages
- 1-18
- OpenAlex
- W4411505952
- Cited by
- 5
- References
- 59
- Citations by year
- 2026: 5
- FWCI
- 42.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Annual Symposium on Human-Computer Interaction for Work (CHIWORK) (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- peer review, large language models, ChatGPT, human-AI collaboration, scholarly publishing, review efficiency, review quality, information retrieval, summarization, AI-generated reviews, responsible AI adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
