CoChat
Sign Up
PreprintCited by 32Computational and Text Analysis MethodsAcademic Writing and PublishingSentiment Analysis and Opinion Mining

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Open access
Kang, Dongyeop, Ammar, Waleed, Dalvi, Bhavana, van Zuylen, Madeleine, Kohlmeier, Sebastian, Hovy, Eduard, Schwartz, Roy·2018·arXiv
DOI 10.48550/arxiv.1804.09635arXiv 1804.09635
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer reviewing is a central component in the scientific publishing process. We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1) providing an opportunity to study this important artifact. The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers. We describe the data collection process and report interesting observed phenomena in the peer reviews. We also propose two novel NLP tasks based on this dataset and provide simple baseline models. In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline. In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.

AI snapshot

Written from the paper’s full text.

TL;DR

PeerRead is presented as the first public dataset of scientific peer reviews, containing 14.7K paper drafts with accept/reject decisions and 10.7K expert reviews across ACL, NIPS, ICLR, CoNLL, and arXiv. The paper analyzes review phenomena and defines two NLP tasks: paper acceptance classification and review aspect score prediction.

Key findings

  • PeerRead v1 contains 14,784 paper drafts and 10,770 reviews across NIPS 2013–2017, ICLR 2017, ACL 2017, CoNLL 2016, and arXiv 2007–2017, with accept/reject or probably-rejected labels.
  • In ACL 2017, overall recommendation correlates most with substance (Pearson r = 0.59) and clarity (0.42), and least with soundness/correctness (0.01) and originality (0.08). Reviews recommending oral vs. poster presentations differ by 0.90 in mean recommendation (3.83 vs. 2.92).
  • Acceptance classification beats the majority baseline in all reported settings: ICLR 65.3 vs. 57.6, cs.cl 75.7 vs. 68.9, cs.lg 70.7 vs. 67.9, cs.ai 92.6 vs. 92.1. The abstract reports up to 21% error reduction; §4.1 text says up to 22%.
  • For aspect score prediction, the Paper;Review model outperforms the Mean baseline in all four high-variance aspects, and the Review model outperforms Mean in three out of four; on average all models slightly improve over Mean.

Methods

  • Data collection used three strategies: opt-in reviews via Softconf for CoNLL 2016 and ACL 2017; crawling NIPS 2013–2017 accepted papers/reviews and ICLR 2017 OpenReview official reviews; and crawling arXiv 2007–2017 submissions, labeling accepted papers by matching to DBLP/target venues and probably-rejected papers by submission-date and category criteria.
  • Preprocessing organized data into five sections, split 0.9:0.05:0.05 into train/dev/test, extracted paper text with Science Parse, and stored JSON paper objects with decisions and reviews.
  • Acceptance classification used 22 coarse features plus lexical features with classifiers including logistic regression, SVM, Random Forest, kNN, Decision Tree, MLP, AdaBoost, and Naive Bayes, tuned by 5-fold cross-validation. Aspect prediction used CNN, LSTM, and DAN regression models conditioned on paper text, review text, or both, with GloVe embeddings and RMSProp, compared against a mean-score baseline.

Limitations

  • The paper states that NIPS provides reviews only for accepted papers; ACL 2017 reviews were explicitly opted-in while ICLR 2017 includes all official reviews, likely biasing ACL review quality positively.
  • The paper notes that lower ICLR appropriateness scores are probably an annotation artifact because reviewers may only address appropriateness when a paper is inappropriate; annotators also only considered review text, not papers, and could infer about half of aspect scores in a feasibility study, with average absolute difference 0.51 from gold scores.
  • The paper states the models are simple and intended to demonstrate dataset uses rather than be best-performing. The text also makes evident that arXiv “probably-rejected” labels are heuristic, not true rejections, and some may be published at workshops or other venues; a data-quality check found one false negative among 12 manually inspected papers.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1804.09635

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 1 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § 2.4 Aspect Score Annotations · p. 3

    The annotators were able to infer about half of the aspect scores from the corresponding review text.

    “The annotators were able to infer about half of the aspect scores from the corresponding review text (the other half was not discussed in the review text).”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    The PeerRead dataset consists of 14.7K paper drafts and their corresponding accept/reject decisions from top-tier venues including ACL, NIPS and ICLR.

    “The dataset consists of 14.7K paper drafts and the corresponding accept / reject decisions in top-tier venues including ACL, NIPS and ICLR.”
    • Also stated in: § 1 Introduction · p. 1 · plain
      “In total, the dataset consists of 14.7K paper drafts and the corresponding accept / reject decisions, including a subset of 3K papers for which we have 10.7K textual reviews written by experts.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 3 Data-Driven Analysis of Peer Reviews · p. 5

    The mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.

    “Another interesting observation is that the mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 4.2 Review Aspect Score Prediction · p. 7

    On average, all models slightly improve over the 'Mean' baseline.

    “On average, all models slightly improve over the 'Mean' baseline.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 2.4 Aspect Score Annotations · p. 3

    The average absolute difference between annotated aspect scores and gold reviewer scores is 0.51 on a 1-5 scale, considering only cases where the aspect was discussed.

    “On average, the absolute di erence between our annotated scores and the gold scores originally provided by reviewers is 0.51 (on a 1-5 scale, considering only those cases where the aspect was discussed in the review text).”
    Quote in the paperSubstantial claim
  • May combine two claims. 15
  • Classified as probably not a substantial claim. 15

  • May combine two claims.

    Findingown data
    § 4.1 Paper Acceptance Classification · p. 6

    Some features have a large contribution to the acceptance classification decision: adding an appendix, a large number of theorems or equations, the average length of the text preceding a citation, the number of papers cited by this paper that were published in the five years before submission, whether the abstract contains 'state of the art' for ICLR or 'neural' for arXiv, and length of title.

    “The table shows that some features have a large contribution on the classification decision: adding an appendix, a large number of theorems or equations, the average length of the text preceding a citation, the number of papers cited by this paper that were published in the five years before the submission of this paper, whether the abstract contains a phrase 'state of the art' for ICLR or 'neural”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.2 Review Aspect Score Prediction · p. 6

    The two sections of PeerRead that include aspect scores are ACL 2017 and ICLR 2017.

    “For this task, we use the two sections of PeerRead which include aspect scores: ACL 2017 and ICLR 2017.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 2.4 Aspect Score Annotations · p. 4

    On average, 77% of annotations were consistent (exactly same or off by 1 point) and 2% were inconsistent (off by 2 points or more).

    “On average, 77% of the annotations were consistent (i.e., the re-annotation was exactly the same as the original annotation, or was o by 1 point) and 2% were inconsistent (i.e., the re-annotation was o by 2 points or more).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 2.1 Review Collection · p. 2

    The dataset contains 39 reviews for 22 CoNLL 2016 submissions and 275 reviews for 137 ACL 2017 submissions.

    “This resulted in 39 reviews for 22 CoNLL 2016 submissions, and 275 reviews for 137 ACL 2017 submissions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 2.4 Aspect Score Annotations · p. 4

    The remaining 1,304 official reviews are all written by anonymous reviewers assigned by the program committee to review a particular submission.

    “The remaining 1,304 o cial reviews are all written by anonymous reviewers assigned by the program committee to review a particular submission.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 2.4 Aspect Score Annotations · p. 4

    About 49% of reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.

    “For example, about 49% of the reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6 Conclusion · p. 8

    PeerRead contains 14.7K papers and 10.7K reviews.

    “We introduced PeerRead, the first publicly available peer review dataset for research purposes, containing 14.7K papers and 10.7K reviews.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given
    § 4.2 Review Aspect Score Prediction · p. 6

    The CoNLL 2016 section of PeerRead includes aspect scores but is too small for training.

    “The CoNLL 2016 section also includes aspect scores but is too small for training.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.3 Organization and Preprocessing · p. 3

    The dataset is organized in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.

    “We organize v1.0 of the PeerRead dataset in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3 Data-Driven Analysis of Peer Reviews · p. 5

    Most of the mean scores are similar in both ACL 2017 and ICLR 2017 sections, with a few notable exceptions.

    “Most of the mean scores are similar in both sections, with a few notable exceptions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Paper Acceptance Classification · p. 6

    Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.

    “Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.”
    • Also stated in: § Abstract · p. 1 · plain
      “In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 2.4 Aspect Score Annotations · p. 3

    Aspect scores assigned by reviewers are included in the opted-in sections in PeerRead, but they are missing from the remaining reviews.

    “While aspect scores assigned by reviewers are included in the opted-in sections in PeerRead, they are missing from the remaining reviews.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3 Data-Driven Analysis of Peer Reviews · p. 5

    The lower mean appropriateness score in ICLR 2017 compared to ACL 2017 is probably an artifact of the annotation process, because reviewers probably only address appropriateness explicitly if the paper is inappropriate.

    “While this might indicate that ICLR 2017 attracted more irrelevant submissions, this is probably an artifact of our annotation process: reviewers probably only address appropriateness explicitly in their review if the paper is inappropriate, which leads to a strong negative bias against this category in our ICLR dataset.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Reviewers probably only address appropriateness explicitly in their review if the paper is inappropriate.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 4.2 Review Aspect Score Prediction · p. 6

    The baseline that always predicts the mean score of an aspect is guaranteed to obtain mean square errors less than or equal to the majority baseline.

    “This baseline is guaranteed to obtain mean square errors less than or equal to the majority baseline.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 2.4 Aspect Score Annotations · p. 4

    The same annotators re-annotated a random sample of 30 reviews to assess annotation consistency.

    “In order to assess annotation consistency, the same annotators re-annotated a random sample consisting of 30 reviews.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3 Data-Driven Analysis of Peer Reviews · p. 4

    The annotators only considered the review text, and did not have access to the papers.

    “Importantly, our annotators only considered the review text, and did not have access to the papers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 2.4 Aspect Score Annotations · p. 4

    We randomly reordered the reviews before annotation so that annotator judgments based on one review are less affected by other reviews of the same paper.

    “We randomly reordered the reviews before annotation so that the annotator judgments based on one review are less a ected by other reviews of the same paper.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 4.1 Paper Acceptance Classification · p. 6

    We tuned hyperparameters using 5-fold cross validation on the training set.

    “We use the standard test split and tune our hyperparameters using 5-fold cross validation on the training set.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given
    § 6 Conclusion · p. 8

    The paper defines two novel tasks based on PeerRead: predicting the acceptance of a paper based on textual features, and predicting the score of each aspect in a review based on the paper and review contents.

    “We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 6 Conclusion · p. 8

    PeerRead is the first publicly available peer review dataset for research purposes.

    “We introduced PeerRead, the first publicly available peer review dataset for research purposes, containing 14.7K papers and 10.7K reviews.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Most peer reviews are not made publicly available.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction · p. 1

    The NIPS 2014 organizers assigned 10% of conference submissions to two different sets of reviewers and observed that the two committees disagreed on the accept/reject decision for more than a quarter of the papers.

    “For example, the organizers of the NIPS 2014 conference assigned 10% of conference submissions to two di erent sets of reviewers to measure the consistency of the peer reviewing process, and observed that the two committees disagreed on the accept / reject decision for more than a quarter of the papers (Langford and Guzdial, 2015).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Langford and Guzdial, 2015 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationown data
    § 4.1 Paper Acceptance Classification · p. 6

    Since our models lack the sophistication to assess the quality of the work discussed in the given paper, this might indicate that some of the features we define are correlated with strong papers, or bias reviewers' judgments.

    “Since our models lack the sophistication to assess the quality of the work discussed in the given paper, this might indicate that some of the features we define are correlated with strong papers, or bias reviewers' judgments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 3 Data-Driven Analysis of Peer Reviews · p. 5

    ACL 2017 reviews were explicitly opted-in while ICLR 2017 reviews include all official reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.

    “We note, however, that ACL 2017 reviews were explicitly opted-in while the ICLR 2017 reviews include all o cial reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data
    § 5 Related Work · p. 7

    PeerRead is a large-scale dataset that enables the wider scientific community to further study the properties of peer review.

    “In this work, we provide a large scale dataset that would enable the wider scientific community to further study the properties of peer review, and potentially come up with enhancements to current peer review model.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Most peer reviews are not made publicly available.
24 of 24

  • The paper relies on this without support.

    Quantitative studies of peer reviews had been limited to the few individuals who had access to peer reviews of a given venue.

    “Despite these e orts, quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue (e.g., journal editors and program chairs).”

  • The paper relies on this without support.

    Reviewer comments often focus on the key strengths or weaknesses of the paper and are not meant to be a comprehensive assessment of each aspect.

    “This is expected since reviewer comments often focus on the key strengths or weaknesses of the paper and are not meant to be a comprehensive assessment of each aspect.”

  • The paper relies on this without support.

    Reviewers probably only address appropriateness explicitly in their review if the paper is inappropriate.

    “reviewers probably only address appropriateness explicitly in their review if the paper is inappropriate, which leads to a strong negative bias against this category in our ICLR dataset.”

  • The paper relies on this without support.

    Review length is often thought of as a measure of its quality.

    “Since review length is often thought of as a measure of its quality, this raises interesting questions about the quality of reviews in ICLR vs. ACL conferences.”

  • The paper relies on this without support.

    Most peer reviews are not made publicly available.

    “Surprisingly, however, most of the reviews are not made publicly available.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Maximum error reduction is reported as both 21% and 22%

    Minor

    The paper gives two different upper bounds for the acceptance-classification error reduction.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Sarah Greaves, Joanna Scott, Maxine Clarke, Linda Miller, Timo Hannay, Annette Thomas, and Philip Campbell. 2006. Nature's trial of open peer review. Nature 444(971):10-1038.
  2. Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. 2011. A quantitative analysis of peer review. In Proc. of ISSI .
  3. Drummond Rennie. 2016. Make peer review scientific: thirty years on from the first congress on peer review, drummond rennie reflects on the improvements brought about by research into the processand calls for more. Nature 535(7610):31-34.
  4. Seán G Roberts and Tessa Verhoef. 2016. Doubleblind reviewing at evolang 11 reveals gender bias. Journal of Language Evolution 1(2):163-167.
  5. Tijmen Tieleman and Geo rey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26-31.

Cited by

Sorted by citations

Related

Authors
Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R.
Affiliations
  • Carnegie Mellon University (US)
  • Allen Institute for Artificial Intelligence (US)
  • University of Washington (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1804.09635
arXiv
1804.09635
OpenAlex
W2798441907
Cited by
28
References
11
Citations by year
2025: 1 · 2024: 1 · 2023: 5 · 2022: 2 · 2021: 3 · 2020: 11 · 2019: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
peer review, scholarly publishing, natural language processing tasks, ACL, NIPS, ICLR

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review