An Analysis of Tasks and Datasets in Peer Reviewing
Open accessAbstract
Taking note of the current challenges of the peer review system, this paper inventories the research tasks for analysing and possibly automating parts of the reviewing process, like matching submissions with a reviewer's domain of expertise.For each of these tasks we list their associated datasets, analysing their quality in terms of available documentation of creation and use.Building up on this, we give a set of recommendations to take into account when collecting and releasing data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most studies in NLP for peer review focus exclusively on machine learning and NLP domains.
“Second, most studies in NLP for peer review focus exclusively on machine learning and NLP domains (Staudinger et al., 2024); we contribute to bridging the gap towards other lessrepresented scientific domains (Kuznetsov et al., 2024).”
From STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond (2025) · cites this paper as Staudinger et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Kwangsu Cho. 2008. Machine classification of peer comments in physics: 1st international conference on educational data mining, EDM 2008. Educational Data Mining 2008 -1st International Conference on Educational Data Mining, Proceedings, pages 192- 196.
- Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers.
- Steven Jecmen, Minji Yoon, Vincent Conitzer, Nihar B. Shah, and Fei Fang. 2023. A dataset on malicious paper bidding in peer review.
- Amir R. Kachooei and Mohammad H. Ebrahimzadeh. 2022. Editorial: What is peer review? Archives of Bone and Joint Surgery, 10(1):1-2. · DOI
- Ilia Kuznetsov, Osama Mohammed Afzal, Koen Der- cksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, and Iryna Gurevych. 2024. What can natural language processing do for peer review?
- Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023b. MOPRD: A multi- disciplinary open peer review dataset. Neural Com- puting and Applications, 35(34):24191-24206. · DOI · Paper page
- Dominik Pfütze, Eva Ritz, Julius Janda, and Roman Rietsche. 2022. A corpus for suggestion mining of german peer feedback. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 5539-5547. European Language Re- sources Association.
- Barbara Plank and Reinard van Dalen. 2019. Cite- Tracked: A Longitudinal Dataset of Peer Reviews and Citations. In BIRNDL@SIGIR.
- Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Is- abelle Guyon, and Ulrike von Luxburg. 2018. Design and analysis of the nips 2016 review process.
- Rajeev Verma, Rajarshi Roychoudhury, and Tirthankar Ghosal. 2022. The lack of theory is painful: Mod- eling harshness in peer review comments. In Pro- ceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 925-935. Association for Computa- tional Linguistics.
- Jiayao Zhang, Hongming Zhang, Zhun Deng, and Dan Roth. 2022. Investigating fairness disparities in peer review: A language model enhanced approach.
- Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Pro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340- 9351. ELRA and ICCL.
Cited by
Sorted by citations
Lin, T.-L., Chen, W.-C., Hsiao, T.-F., Liu, H.-I., Yeh, Y.-H., Chan, Y.-K., Lien, W.-S., Kuo, P.-Y., Yu, P. S., & Shuai, H.-H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.259 · Cited by 6
Zhou, L., Zhang, R., Dai, X., Hershcovich, D., & Li, H. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114988 · Cited by 3 · see citing works
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2 · see citing works
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
Dycke, N., Zečević, M., Kuznetsov, I., Suess, B., Kersting, K., & Gurevych, I. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.1107 · Cited by 1 · see citing works
Identifying New Promising Research Directions with Open Peer Reviews and Contextual Top2Vec
Devyatkin, D. A., Sochenkov, I. V., Popov, D., Zubarev, D., Ryzhova, A., Abanin, F., & Grigoriev, O. G. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9120319
Abbas, A., Ekowati, D., Suhariadi, F., & Fenitra, R. M. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-4377-8.ch007
Shibah, S. R. M. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.5887164
RECO-Rank: A Compensatory Multi-Evidence Ranking Model for Cross-Domain Reviewer RecommendationPreprint
Chen, J., Xu, J., & Fang, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9558981/v1
- Authors
- Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A.
- Affiliations
- TU Wien (AT)
- Venue
- Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)
- Published
- 2024
- Pages
- 257-268
- OpenAlex
- W4402669984
- Cited by
- 6
- References
- 0
- Citations by year
- 2026: 5 · 2025: 1
- FWCI
- 9.68
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Scholarly Document Processing (SDP) (publishedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Keywords
- peer review, research assignments, datasets, data quality, data documentation, domain knowledge, data release
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
