The lack of theory is painful: Modeling Harshness in Peer Review Comments
Open accessAbstract
Rajeev Verma, Rajarshi Roychoudhury, Tirthankar Ghosal. Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2022.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work analyses peer reviews on different aspects, including harshness, thoroughness and helpfulness, completeness, substantiation, and politeness.
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
From The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors (2025) · cites this paper as Verma et al., 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- Prabhat Kumar Bharti, Tirthankar Ghosal, Mayank Agarwal, and Asif Ekbal. 2022a. Betterpr: A dataset for estimating the constructiveness of peer review comments. In Linking Theory and Practice of Digital Libraries, pages 500-505, Cham. Springer Interna- tional Publishing.
- Prabhat Kumar Bharti, Tirthankar Ghosal, Mayank Agrawal, and Asif Ekbal. 2022b. How confident was your reviewer? estimating reviewer confidence from peer review texts. In Document Analysis Sys- tems, pages 126-139, Cham. Springer International Publishing.
- Daniela Calvetti and Lothar Reichel. 2003. Tikhonov regularization of large linear problems. BIT Numeri- cal Mathematics, 43(2):263-283.
- T.N. Flynn and A.A.J. Marley. 2014. Best-worst scal- ing: theory and methods. In Stephane Hess and An- drew Daly, editors, Handbook of Choice Modelling, Chapters, chapter 8, pages 178-201.
- Antigoni Maria Founta, Constantinos Djouvas, De- spoina Chatzakou, Ilias Leontiadis, Jeremy Black- burn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018a. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Con- ference on Web and Social Media.
- Tirthankar Ghosal. 2022. Studies in aspects of peer review: Novelty, scope, research lineage, review sig- nificance, and peer review outcome. SIGIR Forum, 55(2). · DOI
- Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Push- pak Bhattacharyya. 2019b. Deepsentipeer: Harness- ing sentiment in review texts to recommend peer review decisions. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 1120-1130.
- Ken Hyland and Feng (Kevin) Jiang. 2020. "this work is antithetical to the spirit of research": An anatomy of harsh peer reviews. Journal of English for Academic Purposes, 46:100867.
- Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (peerread): Collection, insights and nlp ap- plications. arXiv preprint arXiv:1804.09635. · arXiv
- Sandeep Kumar, Hardik Arora, Tirthankar Ghosal, and Asif Ekbal. 2022. Deepaspeer: Towards an aspect- level sentiment controllable framework for decision prediction from academic peer reviews. In 2022 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 1-11.
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. · arXiv
- Barbara Plank and Reinard van Dalen. 2019. Cite- tracked: A longitudinal dataset of peer reviews and citations. In BIRNDL@SIGIR.
- Anna Rogers. 2020a. Peer review in nlp: reject-if-not- sota.
- Anna Rogers. 2020b. Peer review in nlp: resource papers.
- Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power im- plications of language. In ACL.
- Christine -Tardy. 2018. We are all Reviewer 2: A Window into the secret world of peer review, pages 271-289. Springer International Publishing. · DOI
- Ke Wang and Xiaojun Wan. 2018. Sentiment analysis of peer review texts for scholarly papers. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 175- 184.
- Christie Wilcox. 2019. Rude reviews are perva- sive and sometimes harmful, study finds. Science, 366(6472):1433-1433. · DOI
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Cited by
Sorted by citations
PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews
Bharti, P. K., Navlakha, M., Agarwal, M., & Ekbal, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09662-3 · Cited by 12 · see citing works
“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text
Hada, R., Seth, A., Diddee, H., & Bali, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.115 · Cited by 12 · see citing works
Bharti, P. K., Agarwal, M., & Ekbal, A. (2024) · Scientometrics · doi:10.1007/s11192-024-04938-z · Cited by 6 · see citing works
Automatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.684 · Cited by 5 · see citing works
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1476 · Cited by 1 · see citing works
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Prior work analyses peer reviews on different aspects, including harshness (Verma et al., 2022), thoroughness and helpfulness (Severin et al., 2022), completeness (Yuan et al., 2022), substantiation (Guo et al., 2023), and politeness (Bharti et al., 2024).”
Traces of the Past: A Glimpse into the Journey of Alternative Learning System Students
Cataan, J., & Baluya, J. (2025) · Psychology and Education A Multidisciplinary Journal · doi:10.70838/pemj.480803
- Authors
- Verma, R., Roychoudhury, R., & Ghosal, T.
- Affiliations
- Charles University (CZ)
- Venue
- Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2022
- Pages
- 925-935
- OpenAlex
- W4404784079
- Cited by
- 5
- References
- 0
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 3
- FWCI
- 0.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Arts and Humanities › Literature and Literary Theory › Discourse Analysis in Language Studies
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Keywords
- peer review comments, harshness, academic writing, reviewer comments
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
