CoChat
Sign Up
Cited by 71Sentiment Analysis and Opinion MiningComputational and Text Analysis MethodsPublishing and Scholarly Communication

Peer review analyze: A novel benchmark resource for computational analysis of peer reviews

Open access
Ghosal, Tirthankar, Kumar, Sandeep, Bharti, Prabhat Kumar, Ekbal, Asif·2022·PLOS ONE
DOI 10.1371/journal.pone.0259238PMC PMC8794172
Open this paper in CoChat — free account

Abstract

Peer Review is at the heart of scholarly communications and the cornerstone of scientific publishing. However, academia often criticizes the peer review system as non-transparent, biased, arbitrary, a flawed process at the heart of science , leading to researchers arguing with its reliability and quality. These problems could also be due to the lack of studies with the peer-review texts for various proprietary and confidentiality clauses. Peer review texts could serve as a rich source of Natural Language Processing (NLP) research on understanding the scholarly communication landscape, and thereby build systems towards mitigating those pertinent problems. In this work, we present a first of its kind multi-layered dataset of 1199 open peer review texts manually annotated at the sentence level (∼ 17k sentences) across the four layers, viz. Paper Section Correspondence, Paper Aspect Category, Review Functionality, and Review Significance. Given a text written by the reviewer, we annotate: to which sections (e.g., Methodology, Experiments, etc.), what aspects (e.g., Originality/Novelty, Empirical/Theoretical Soundness, etc.) of the paper does the review text correspond to, what is the role played by the review text (e.g., appreciation, criticism, summary, etc.), and the importance of the review statement (major, minor, general) within the review. We also annotate the sentiment of the reviewer (positive, negative, neutral) for the first two layers to judge the reviewer’s perspective on the different sections and aspects of the paper. We further introduce four novel tasks with this dataset, which could serve as an indicator of the exhaustiveness of a peer review and can be a step towards the automatic judgment of review quality . We also present baseline experiments and results for the different tasks for further investigations. We believe our dataset would provide a benchmark experimental testbed for automated systems to leverage on current NLP state-of-the-art techniques to address different issues with peer review quality , thereby ushering increased transparency and trust on the holy grail of scientific research validation . Our dataset and associated codes are available at https://www.iitp.ac.in/~ai-nlp-ml/resources.html#Peer-Review-Analyze .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0259238

Cites

50 of 51 references verified1 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The guided skimming for peer review task is novel and builds upon recent work in cross-document modeling for peer review by Kuznetsov et al. (2022) and other related works.

    “The guided skimming for peer review task is novel and builds upon the recent work in cross-document modeling for peer review by Kuznetsov et al. (2022) and other related works (Qin and Zhang, 2022;Ghosal et al., 2022a).”

    From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Ghosal et al., 2022a

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 53 references couldn’t be matched to a known paper.

Research works printed but not linked: 13

  1. [18]Rogers A. Peer review in NLP: reject-if-not-SOTA; 2020. https://hackingsemantics.xyz/2020/reviewing-models/.
  2. [20]EMNLP. Advice on Reviewing for EMNLP; 2020. https://2020.emnlp.org/blog/2020-05-17-write-good-reviews.
  3. [21]ICLR 2021 Reviewer Instructions; 2020. https://iclr.cc/Conferences/2021/ReviewerGuide.
  4. [22]AAAI 2020 Reviewing Guidelines; 2019. https://aaai.org/Conferences/AAAI-20/wp-content/uploads/2019/09/AAAI-20-Reviewing-Guidelines.pdf.
  5. [23]ICML 2020 Reviewer Guidelines; 2020. https://icml.cc/Conferences/2020/ReviewerGuidelines.
  6. [24]NeurIPS 2020 Reviewer Guidelines; 2020. https://nips.cc/Conferences/2020/PaperInformation/ReviewerGuidelines.
  7. [26]How to Write a Peer Review—PLOS;. https://plos.org/resource/how-to-write-a-peer-review/.
  8. [30]Neumann M. ICLR Submissions 3 Years on; 2020. http://markneumann.xyz/visualessays/iclr/.
  9. [35]Cohen K, Fort K, Mieskes M, Névéol A. Reviewing Natural Language Processing Research. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts. Online: Association for Computational Linguistics; 2020. p. 16–18. Available from: https://www.aclweb.org/anthology/2020.acl-tutorials.4.
  10. [37]Meadows A. What Does Transparent Peer Review Mean and Why is it Important?; 2017. https://scholarlykitchen.sspnet.org/2017/08/03/transparent-peer-review-mean-important/.
  11. [50]Plank B, van Dalen R. CiteTracked: A Longitudinal Dataset of Peer Reviews and Citations. In: Chandrasekaran MK, Mayr P, editors. Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), Paris, France, July 25, 2019. vol. 2414 of CEUR Workshop Proceedings. CEUR-WS.org; 2019. p. 116–122. Available from: http://ceur-ws.org/Vol-2414/paper12.pdf.
  12. [62]Mihalcea R, Tarau P. TextRank: Bringing Order into Text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25-26 July 2004, Barcelona, Spain. ACL; 2004. p. 404–411. Available from: https://www.aclweb.org/anthology/W04-3252/.
  13. [64]Lin CY. ROUGE: A Package for Automatic Evaluation of Summaries. In: Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics; 2004. p. 74– 81. Available from: https://www.aclweb.org/anthology/W04-1013.

Cited by

Sorted by citations

Related

Authors
Ghosal, T., Kumar, S., Bharti, P. K., & Ekbal, A.
Affiliations
  • Charles University (CZ)
  • Indian Institute of Technology Patna (IN)
Corresponding author
Ghosal, Tirthankar
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2022-01-27
Volume
17
Issue
1
Pages
e0259238
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0259238
PMC
PMC8794172
OpenAlex
W4210467014
Cited by
66
References
53
Citations by year
2026: 13 · 2025: 14 · 2024: 12 · 2023: 16 · 2022: 11
FWCI
4.36
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Lincoln (University of Nebraska) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
meity (VISPHD-MEITY-2518)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
Keywords
peer review, open peer review, annotated dataset, peer review quality, scholarly communication
MeSH terms
Humans, Natural Language Processing, Reproducibility of Results, Databases, Factual, Peer Review, Research, Benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review