PeerArg: Argumentative Peer Review with LLMs
Open accessAbstract
Peer review is an essential process to determine the quality of papers submitted to scientific conferences or journals. However, it is subjective and prone to biases. Several studies have been conducted to apply techniques from NLP to support peer review, but they are based on black-box techniques and their outputs are difficult to interpret and trust. In this paper, we propose a novel pipeline to support and understand the reviewing and decision-making processes of peer review: the PeerArg system combining LLMs with methods from knowledge representation. PeerArg takes in input a set of reviews for a paper and outputs the paper acceptance prediction. We evaluate the performance of the PeerArg pipeline on three different datasets, in comparison with a novel end-2-end LLM that uses few-shot learning to predict paper acceptance given reviews. The results indicate that the end-2-end LLM is capable of predicting paper acceptance from reviews, but a variant of the PeerArg pipeline outperforms this LLM.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies expose a persistent disconnect between aspect scores and final decisions.
“Recent studies expose a persistent disconnect between aspect scores and final decisions (Sukpanichnant et al., 2024).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Sukpanichnant et al., 2024
The A.2 Sentiment Analysis figure caption reports imperfect acceptance-decision prediction when the model uses aspect scores from reviews, which lends some support to a gap between scores and decisions. But accuracy varies across configurations, and the paper does not establish a persistent disconnect; Related Work instead reports a correlation between aspect-based sentiments and acceptance decisions. Our copy provides section labels but no page numbers for these passages. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 3× · see citing works
“Hua et al. (2019) applied argumentation to understand the content and structure of peer reviews.”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 23
- L. Amgoud, C. Cayrol, M.-C. Lagasquie-Schiex, and P. Livet. On bipolarity in argumentation frameworks. International Journal of Intelligent Systems, 23:1062 -1093, 10 2008. doi: 10.1002/ int.20307. · DOI
- K. Atkinson, P. Baroni, M. Giacomin, A. Hunter, H. Prakken, C. Reed, G. Simari, M. Thimm, and S. Villata. Towards artificial argumentation. AI Magazine, 38(3):25-36, Oct. 2017. doi: 10. 1609/aimag.v38i3.2704. URL https://ojs.aaai.org/aimagazine/ index.php/aimagazine/article/view/2704. · DOI · Paper page
- P. Baroni, A. Rago, and F. Toni. How many properties do we need for gradual argumentation? Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11544. URL https://ojs.aaai.org/index.php/ AAAI/article/view/11544. · DOI
- C. Bhatia, T. Pradhan, and S. Pal. Metagen: An academic meta- review generation system. In Proceedings of the 43rd Interna- tional ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 1653-1656, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164. doi: 10.1145/3397271.3401190. URL https://doi.org/10.1145/3397271.3401190. · DOI
- S. Bird, E. Klein, and E. Loper. Natural Language Processing with Python. O'Reilly Media, Inc., 2009.
- S. Chakraborty, P. Goyal, and A. Mukherjee. Aspect-based sen- timent analysis of scientific reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL '20, page 207-216, New York, NY, USA, 2020. As- sociation for Computing Machinery. ISBN 9781450375856. doi: 10.1145/3383583.3398541. URL https://doi.org/10.1145/ 3383583.3398541. · DOI · Paper page
- P. M. Dung. On the acceptability of arguments and its funda- mental role in nonmonotonic reasoning, logic programming and n-person games. Artificial Intelligence, 77(2):321-357, 1995. ISSN 0004-3702. doi: https://doi.org/10.1016/0004-3702(94) 00041-X. URL https://www.sciencedirect.com/science/article/ pii/000437029400041X. · DOI
- M. Fromm, E. Faerman, M. Berrendorf, S. Bhargava, R. Qi, Y. Zhang, L. Dennert, S. Selle, Y. Mao, and T. Seidl. Argument mining driven analysis of peer-reviews. In AAAI Conference on Artificial Intelligence, 2020. URL https://api.semanticscholar. org/CorpusID:229153112.
- T. Ghosal, S. Kumar, P. K. Bharti, and A. Ekbal. Peer re- view analyze: A novel benchmark resource for computational analysis of peer reviews. https://github.com/Tirthankar-Ghosal/ Peer-Review-Analyze-1.0, 2022a. Includes data from the B12Js yRb raw reviews of ICLR-2018, accessed on 2024-07- 23.
- T. Ghosal, S. Kumar, P. K. Bharti, and A. Ekbal. Peer review an- alyze: A novel benchmark resource for computational analysis of peer reviews. PloS one, 17(1):e0259238, 2022b.
- T. Ghosal, S. Kumar, P. K. Bharti, and A. Ekbal. Peer re- view analyze: A novel benchmark resource for computational analysis of peer reviews. https://github.com/Tirthankar-Ghosal/ Peer-Review-Analyze-1.0, 2022c. Includes data from the B13EC5u6W raw reviews of ICLR-2018, accessed on 2024-07- 23.
- D. Gorur, A. Rago, and F. Toni. Can large language models per- form relation-based argument mining? CoRR, abs/2402.11243, 2024. doi: 10.48550/ARXIV.2402.11243. URL https://doi.org/ 10.48550/arXiv.2402.11243. · DOI
- D. Kang, W. Ammar, B. Dalvi, M. van Zuylen, S. Kohlmeier, E. Hovy, and R. Schwartz. A dataset of peer reviews (Peer- Read): Collection, insights and NLP applications. In M. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pages 1647-1661, New Orleans, Louisiana, June 2018b. Association for Computational Linguis- tics. doi: 10.18653/v1/N18-1149. URL https://aclanthology. org/N18-1149. · DOI · Paper page
- S. Kumar, T. Ghosal, and A. Ekbal. APCS: Towards argu- ment based pros and cons summarization of peer reviews. In T. Ghosal, F. Grezes, T. Allen, K. Lockhart, A. Accomazzi, and S. Blanco-Cuaresma, editors, Proceedings of the Second Work- shop on Information Extraction from Scientific Publications, pages 117-129, Bali, Indonesia, Nov. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.wiesp-1.14. URL https://aclanthology.org/2023.wiesp-1. · DOI
- J. Li, A. Sato, K. Shimura, and F. Fukumoto. Multi-task peer- review score prediction. In M. K. Chandrasekaran, A. de Waard, G. Feigenblat, D. Freitag, T. Ghosal, E. Hovy, P. Knoth, D. Konopnicki, P. Mayr, R. M. Patton, and M. Shmueli-Scheuer, editors, Proceedings of the First Workshop on Scholarly Docu- ment Processing, pages 121-126, Online, Nov. 2020. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2020. sdp-1.14. URL https://aclanthology.org/2020.sdp-1. · DOI
- J. Lin, J. Song, Z. Zhou, Y. Chen, and X. Shi. Moprd: A multidisciplinary open peer review dataset. Neural Comput- ing and Applications, 35(34):24191-24206, Sept. 2023. ISSN 1433-3058. doi: 10.1007/s00521-023-08891-5. URL http: //dx.doi.org/10.1007/s00521-023-08891-5. · DOI · Paper page
- M. J. Mahoney. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1977. doi: https://doi.org/10.1007/BF01173636. · DOI · Paper page
- T. Mossakowski and F. Neuhaus. Modular semantics and char- acteristics for bipolar weighted argumentation graphs. CoRR, abs/1807.06685, 2018. URL http://arxiv.org/abs/1807.06685. · arXiv
- D. J.-L. Moys. Typographic layout and first impressions: test- ing how changes in text layout influence reader's judgments of documents. In Visible Language, 2017. URL https://api. semanticscholar.org/CorpusID:60988587.
- M. B. Nuijten and J. R. Polanin. "statcheck": Automatically detect statistical reporting inconsistencies to increase reproducibility of meta-analyses. Research synthesis methods, 11(5):574-579, 2020. doi: 10.1002/jrsm.1408. · DOI · Paper page
- N. Potyka. Interpreting neural networks as quantitative argu- mentation frameworks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7):6463-6470, May 2021. doi: 10.1609/aaai.v35i7.16801. URL https://ojs.aaai.org/index.php/ AAAI/article/view/16801. · DOI
- A. Rago, F. Toni, M. Aurisicchio, and P. Baroni. Discontinuity- free decision support with quantitative argumentation de- bates. In International Conference on Principles of Knowl- edge Representation and Reasoning, 2016. URL https://api. semanticscholar.org/CorpusID:32233959.
- R. Zhou, L. Chen, and K. Yu. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper review- ing tasks. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lan- guage Resources and Evaluation (LREC-COLING 2024), pages 9340-9351, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main.816.
Cited by
Sorted by citations
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
- Authors
- Sukpanichnant, P., Rapberger, A., & Toni, F.
- Affiliations
- Imperial College London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2409.16813
- OpenAlex
- W4403784477
- Cited by
- 0
- References
- 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Royal Academy of Engineering
- European Commission (101020934, HORIZON2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- peer review, large language models, knowledge representation, review bias, interpretability, scholarly publishing, few-shot learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
