Cited by 190Sentiment Analysis and Opinion MiningNatural Language Processing TechniquesTopic Modeling

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Open access
Kang, Dongyeop, Ammar, Waleed, Dalvi, Bhavana, Zuylen, Madeleine, Kohlmeier, Sebastian, Hovy, Eduard, Schwartz, Roy·2018·Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)

Abstract

Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

AI snapshot

Written from the paper’s full text.

TL;DR

PeerRead is introduced as the first public dataset of scientific peer reviews, containing 14.7K paper drafts with accept/reject decisions and 10.7K textual reviews from ACL, NIPS, ICLR, CoNLL, and arXiv. The paper analyzes review patterns and defines two NLP tasks: predicting paper acceptance and predicting review aspect scores.

Key findings

  • PeerRead v1 contains 14,784 papers and 10,770 reviews across NIPS 2013–2017 (2,420 papers, 9,152 reviews), ICLR 2017 (427 papers, 1,304 reviews), ACL 2017 (137 papers, 275 reviews), CoNLL 2016 (22 papers, 39 reviews), and arXiv 2007–2017 (11,778 papers, no reviews).
  • In ACL 2017 reviews, overall recommendation correlates most with substance (ρ = 0.59) and clarity (ρ = 0.42), and least with soundness/correctness (ρ = 0.01) and originality (ρ = 0.08).
  • Reviews recommending oral presentation have a mean overall recommendation of 3.83 versus 2.92 for poster recommendations, a difference of 0.90 on a 1–5 scale.
  • Acceptance classification outperforms majority baselines in all tested sections: ICLR 65.3% vs. 57.6%, arXiv cs.cl 75.7% vs. 68.9%, cs.lg 70.7% vs. 67.9%, and cs.ai 92.6% vs. 92.1%; the abstract reports up to 21% error reduction, while the results section states up to 22%.
  • For aspect score prediction, the paper reports that the “Paper;Review” model outperforms the mean baseline in all four high-variance aspects examined, and the “Review” model outperforms the mean baseline in three out of four.

Methods

  • Dataset construction used three strategies: opt-in collection with Softconf for CoNLL 2016 and ACL 2017; crawling publicly available NIPS 2013–2017 and ICLR 2017 OpenReview reviews; and crawling arXiv 2007–2017 submissions, labeling accepted papers via DBLP matching and title Levenshtein distance < 0.1, and labeling “probably-rejected” papers by venue, arXiv category, cross-listing, and submission-date criteria.
  • Aspect scores were annotated for 1.3K ICLR 2017 reviews by two authors across seven aspects (appropriateness, clarity, originality, soundness/correctness, meaningful comparison, substance, impact) on a 1–5 scale, with a “not discussed” option; re-annotation of 30 reviews showed 77% consistent annotations, 2% inconsistent, and 21% “not discussed” mismatches.
  • NLP experiments included binary acceptance classification using 22 coarse features plus lexical features with classifiers such as logistic regression, SVM, Random Forest, and MLP, and aspect score regression using CNN, LSTM, and DAN architectures with GloVe 840B embeddings, evaluated by RMSE against a mean-score baseline.

Limitations

  • The paper states that ACL 2017 reviews were explicitly opted-in, likely biasing included ACL reviews toward higher quality, and that lower ICLR appropriateness scores are probably an annotation artifact because reviewers mention appropriateness mainly when a paper is inappropriate.
  • The paper notes that NIPS provides reviews only for accepted papers, that some “probably-rejected” arXiv papers may have been published at workshops or other venues, and that one false negative was found due to a significant title change.
  • The paper states its models are simple and intended to demonstrate dataset potential rather than be best-performing; evident from the text, arXiv rejection labels are heuristic “probably-rejected” rather than verified decisions, ICLR aspect scores are annotator-predicted rather than reviewer-provided, and aspect annotation covers only 1.3K reviews with many aspects often not discussed.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/n18-1149

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 1 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Reviews in the opted-in sections include both text and aspect scores on a scale of 1-5.

    “Reviews include both text and aspect scores (e.g., calrity) on a scale of 1-5.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Our best model outperforms the majority classifiers in all cases.

    “Our best model outperforms the majority classifiers in all cases.”
    Quote in the paperSubstantial claim
  • Findingown data

    The mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.

    “Another interesting observation is that the mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.”
    Quote in the paperNot a substantial claim
  • Findingown data

    In the feasibility study, annotators were able to infer about half of the aspect scores from the corresponding review text.

    “The annotators were able to infer about half of the aspect scores from the corresponding review text (the other half was not discussed in the review text).”
    Quote in the paperSubstantial claim
  • Findingown data

    The authors annotated 1.3K reviews with aspect scores based on the corresponding review text.

    “In order to increase the utility of the dataset, we annotated 1.3K reviews with aspect scores, based on the corresponding review text.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 16
  • May combine two claims. 14
  • Classified as probably not a substantial claim.

    Findingnone given

    PeerRead is collected and organized such that it is easy for other researchers to use it for research purposes, replicate experiments and make a fair comparison to previous results.

    “In contrast, we collected and organized PeerRead such that it is easy for other researchers to use it for research purposes, replicate experiments and make a fair comparison to previous results.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The opted-in review collection resulted in 39 reviews for 22 CoNLL 2016 submissions and 275 reviews for 137 ACL 2017 submissions.

    “This resulted in 39 reviews for 22 CoNLL 2016 submissions, and 275 reviews for 137 ACL 2017 submissions.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    In the data quality check, 364 arXiv papers matched the criteria, 352 of which were labeled as 'accepted'.

    “We found 364 papers which matched these criteria, 352 out of which were labeled as 'accepted'.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    PeerRead consists of 14.7K paper drafts with accept/reject decisions, including a subset of 3K papers with 10.7K expert textual reviews.

    “The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR.The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers.”
    • Also stated in: § 1 Introduction · plain
      “In total, the dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions, including a subset of 3K papers for which we have 10.7K textual reviews written by experts.”
    • Also stated in: § 1 Introduction · plain
      “total 14,784 10,770”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Most mean scores are similar in the ACL 2017 and ICLR 2017 sections, with a few notable exceptions.

    “Most of the mean scores are similar in both sections, with a few notable exceptions.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    PeerRead v1.0 is organized in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.

    “We organize v1.0 of the PeerRead dataset in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    PeerRead contains 14.7K papers and 10.7K reviews.

    “We introduced PeerRead, the first publicly available peer review dataset for research purposes, containing 14.7K papers and 10.7K reviews.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Papers in each section are split into training, development and test sets with 0.9:0.05:0.05 ratios.

    “The papers in each section are further split into standard training, development and test sets with 0.9:0.05:0.05ratios.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Findingnone given

    The paper defines a novel task of predicting the score of each aspect in a review based on the paper and review contents.

    “We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    The arXiv section contains 11,778 papers, with 2,891 accepted and 8,887 probably-rejected labels.

    “arXiv 2007-2017 11,778 --2,891 / 8,887”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    On average, 77% of annotations were consistent (exact or off by 1 point) and 2% were inconsistent (off by 2 points or more).

    “On average, 77% of the annotations were consistent (i.e., the re-annotation was exactly the same as the original annotation, or was off by 1 point) and 2% were inconsistent (i.e., the re-annotation was off by 2 points or more).”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Findingnone given

    The paper defines a novel task of predicting the acceptance of a paper based on textual features.

    “We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    About 49% of the reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.

    “For example, about 49% of the reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    The best model outperforms the majority classifier in all cases, with up to 22% error reduction.

    “Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingnone given

    Aspect scores assigned by reviewers are included in the opted-in sections but missing from the remaining reviews.

    “While aspect scores assigned by reviewers are included in the opted-in sections in PeerRead, they are missing from the remaining reviews.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The CoNLL 2016 section includes aspect scores.

    “The CoNLL 2016 section also includes aspect scores but is too small for training.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    NIPS provides reviews only for accepted papers.

    “Note that NIPS provide reviews only for accepted papers.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    In ACL 2017, reviews recommending oral presentation have higher mean scores than poster for substance, clarity, meaningful comparison, and impact, but lower for soundness/correctness, and similar for originality.

    “Presentation format Oral Poster ∆ stdev Recommendation 3.83 2.92 0.90 0.89 Substance 3.91 3.29 0.62 0.84 Clarity 4.19 3.72 0.47 0.90 Meaningful comparison 3.60 3.36 0.24 0.82 Impact 3.27 3.09 0.18 0.54 Originality 3.91 3.88 0.02 0.87 Soundness/Correctness 3.93 4.18 -0.25 0.91”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The CoNLL 2016 section is too small for training.

    “The CoNLL 2016 section also includes aspect scores but is too small for training.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Only official reviews of reviewers assigned to review that paper are used.

    “The platform also allows any person to review the paper by adding a comment, but we only use the official reviews of reviewers assigned to review that paper.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    This baseline is guaranteed to obtain mean square errors less than or equal to the majority baseline.

    “This baseline is guaranteed to obtain mean square errors less than or equal to the majority baseline.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim.

    Study design and conductown data

    The same annotators re-annotated a random sample of 30 reviews to assess annotation consistency.

    “In order to assess annotation consistency, the same annotators re-annotated a random sample consisting of 30 reviews.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    For consistency, only the first arXiv version of each paper (accepted or rejected) is included in the dataset.

    “For consistency, we only include the first arXiv version of each paper (accepted or rejected) in the dataset.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    If a paper has multiple versions, the submission date of the first version is considered.

    “If a paper has multiple versions, we consider the submission date of the first version.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    ACL 2017 reviews were explicitly opted-in while ICLR 2017 reviews include all official reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.

    “We note, however, that ACL 2017 reviews were explicitly opted-in while the ICLR 2017 reviews include all official reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Some 'probably-rejected' papers may be published at workshops or other venues.

    “Note that some of the 'probably-rejected' papers may be published at workshops or other venues.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Annotators only considered the review text, and did not have access to the papers.

    “Importantly, our annotators only considered the review text, and did not have access to the papers.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    NIPS 2014 organizers assigned 10% of conference submissions to two different sets of reviewers and observed that the two committees disagreed on the accept/reject decision for more than a quarter of the papers.

    “For example, the organizers of the NIPS 2014 conference assigned 10% of conference submissions to two different sets of reviewers to measure the consistency of the peer reviewing process, and observed that the two committees disagreed on the accept/reject decision for more than a quarter of the papers (Langford and Guzdial, 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Langford and Guzdial, 2015 source
28 of 28
  • The paper relies on this without support.

    Quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue.

    “Despite these efforts, quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue (e.g., journal editors and program chairs).”
  • The paper relies on this without support.

    Most peer reviews are not made publicly available.

    “Surprisingly, however, most of the reviews are not made publicly available.”
  • The paper relies on this without support.

    Review length is often thought of as a measure of review quality.

    “Since review length is often thought of as a measure of its quality, this raises interesting questions about the quality of reviews in ICLR vs. ACL conferences.”
  • The paper relies on this without support.

    In many publication venues, reviewers assign numeric aspect scores as part of the peer review.

    “In many publication venues, reviewers assign numeric aspect scores (e.g., clarity, originality, substance) as part of the peer review.”
  • The paper relies on this without support.

    The presentation format decision of accepted papers is based on recommendation by the reviewers.

    “The presentation format decision of accepted papers is based on recommendation by the reviewers.”
  • The paper relies on this without support.

    The baseline models lack the sophistication to assess the quality of the work discussed in the given paper.

    “Since our models lack the sophistication to assess the quality of the work discussed in the given paper, this might indicate that some of the features we define are correlated with strong papers, or bias reviewers' judgments.”
  • The paper relies on this without support.

    Aspects with higher RMSE scores for the 'Mean' baseline indicate higher variance among the review scores for this aspect.

    “We note that aspects with higher RMSE scores for the 'Mean' baseline indicate higher variance among the review scores for this aspect, so we focus our discussion on these aspects.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • PeerRead's last update in its GitHub repository dates back to May 2018, and its content is limited to a few conferences before 2018.

    “Despite the fact that this dataset was supposed to be released periodically, the last update available in its GitHub repositoryfoot_4 dates back to May 2018.The content is currently limited, based on a few conferences before 2018.”

    From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Kang et al., 2018]

    The Introduction says the authors planned periodic releases, and §2.3 lists five sections in version 1.0, covering conferences through 2017 and arXiv submissions through 2017. The paper does not establish that May 2018 was the last GitHub update or that the dataset remained limited to those sections at the time of the citing sentence. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • PeerRead, MOPRD, and NLPeer are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.

    “PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”

    From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Kang et al., 2018

    The Abstract reports 14.7K paper drafts and 10.7K corresponding reviews written by experts, supporting the description of PeerRead as a large paper-and-review dataset. But §2.4, “Aspect Score Annotations,” says the authors annotated 1.3K reviews with aspect scores, so describing PeerRead as without annotations is not supported. This source does not establish the properties of MOPRD or NLPeer. (LLM, conf=0.88)

    Experimental — accuracy not yet measured
  • Science Parse provides fast metadata and rough text at web scale.

    “Science Parse (Sci, 2017-2025) Kang et al. (2018) PDF → JSON Fast metadata + rough text; webscale.”

    From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al. (2018)

    Section 2.3, Organization and Preprocessing (page not available), says the authors used Science Parse to extract text from paper PDFs and stored the resulting PeerRead records in JSON-encoded files. The paper does not establish that Science Parse itself outputs JSON, is fast, produces only rough text, or operates at web scale. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • Existing evaluations of AI-assisted peer review mainly focus on independent tasks or on final review quality and human alignment.

    “existing evaluations of AI-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions.”

    From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018

    Section 4 (“NLP Tasks”) describes predicting paper acceptance, and Table 5 reports test accuracy for that task, supporting the final-decision accuracy part. The paper does not establish that existing AI-assisted peer-review evaluations mainly focus on this outcome or on the quality of generated reviews. Page numbers were not available in our copy. (LLM, conf=0.78)

    Experimental — accuracy not yet measured
  • Existing peer-review datasets usually contain heterogeneous paper-review-decision records with differing field structures, text granularity, and label sources.

    “First, existing peer-review datasets usually contain heterogeneous paper-review-decision records.Their field structures, text granularity, and label sources differ”

    From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018

    Section 2.3 describes PeerRead’s paper-review-decision records, and Sections 2.1 and 2.4 show that its sections differ in review availability and how labels or scores were obtained. But the paper describes PeerRead as the first public peer-review dataset for research; it does not establish that existing datasets usually have differing field structures or text granularity. Section 2.3 instead describes a common JSON representation for PeerRead’s records. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Maximum error reduction is reported as both 21% and 22%

    Minor

    The abstract gives a maximum error reduction of 21%, whereas the results section gives 22% for the same paper-acceptance task.

1 of 1

Cites

Flagged first, then by citations

1 of the 22 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. Sarah Greaves, Joanna Scott, Maxine Clarke, Linda Miller, Timo Hannay, Annette Thomas, and Philip Campbell. 2006. Nature's trial of open peer review. Nature 444(971):10-1038.
  2. Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. 2011. A quantitative analy- sis of peer review. In Proc. of ISSI.
  3. Drummond Rennie. 2016. Make peer review scien- tific: thirty years on from the first congress on peer review, drummond rennie reflects on the improve- ments brought about by research into the process- and calls for more. Nature 535(7610):31-34.
  4. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running aver- age of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26-31.

Cited by

Sorted by citations

Authors
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R.
Affiliations
  • Carnegie Mellon University (US)
  • Allen Institute for Artificial Intelligence (US)
  • University of Washington (US)
Venue
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Published
2018
Pages
1647-1661
OpenAlex
W2964211782
Cited by
171
References
22
Citations by year
2026: 14 · 2025: 15 · 2024: 15 · 2023: 33 · 2022: 26 · 2021: 28 · 2020: 19 · 2019: 20 · 2018: 1
FWCI
6.77
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
peer review, scholarly publishing, natural language processing applications, research evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review