The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; none of the 1 sources its claims cite is checked for support yet.
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Open accessAbstract
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
PeerRead is introduced as the first public dataset of scientific peer reviews, containing 14.7K paper drafts with accept/reject decisions and 10.7K textual reviews from ACL, NIPS, ICLR, CoNLL, and arXiv. The paper analyzes review patterns and defines two NLP tasks: predicting paper acceptance and predicting review aspect scores.
Key findings
- PeerRead v1 contains 14,784 papers and 10,770 reviews across NIPS 2013–2017 (2,420 papers, 9,152 reviews), ICLR 2017 (427 papers, 1,304 reviews), ACL 2017 (137 papers, 275 reviews), CoNLL 2016 (22 papers, 39 reviews), and arXiv 2007–2017 (11,778 papers, no reviews).
- In ACL 2017 reviews, overall recommendation correlates most with substance (ρ = 0.59) and clarity (ρ = 0.42), and least with soundness/correctness (ρ = 0.01) and originality (ρ = 0.08).
- Reviews recommending oral presentation have a mean overall recommendation of 3.83 versus 2.92 for poster recommendations, a difference of 0.90 on a 1–5 scale.
- Acceptance classification outperforms majority baselines in all tested sections: ICLR 65.3% vs. 57.6%, arXiv cs.cl 75.7% vs. 68.9%, cs.lg 70.7% vs. 67.9%, and cs.ai 92.6% vs. 92.1%; the abstract reports up to 21% error reduction, while the results section states up to 22%.
- For aspect score prediction, the paper reports that the “Paper;Review” model outperforms the mean baseline in all four high-variance aspects examined, and the “Review” model outperforms the mean baseline in three out of four.
Methods
- Dataset construction used three strategies: opt-in collection with Softconf for CoNLL 2016 and ACL 2017; crawling publicly available NIPS 2013–2017 and ICLR 2017 OpenReview reviews; and crawling arXiv 2007–2017 submissions, labeling accepted papers via DBLP matching and title Levenshtein distance < 0.1, and labeling “probably-rejected” papers by venue, arXiv category, cross-listing, and submission-date criteria.
- Aspect scores were annotated for 1.3K ICLR 2017 reviews by two authors across seven aspects (appropriateness, clarity, originality, soundness/correctness, meaningful comparison, substance, impact) on a 1–5 scale, with a “not discussed” option; re-annotation of 30 reviews showed 77% consistent annotations, 2% inconsistent, and 21% “not discussed” mismatches.
- NLP experiments included binary acceptance classification using 22 coarse features plus lexical features with classifiers such as logistic regression, SVM, Random Forest, and MLP, and aspect score regression using CNN, LSTM, and DAN architectures with GloVe 840B embeddings, evaluated by RMSE against a mean-score baseline.
Limitations
- The paper states that ACL 2017 reviews were explicitly opted-in, likely biasing included ACL reviews toward higher quality, and that lower ICLR appropriateness scores are probably an annotation artifact because reviewers mention appropriateness mainly when a paper is inappropriate.
- The paper notes that NIPS provides reviews only for accepted papers, that some “probably-rejected” arXiv papers may have been published at workshops or other venues, and that one false negative was found due to a significant title change.
- The paper states its models are simple and intended to demonstrate dataset potential rather than be best-performing; evident from the text, arXiv rejection labels are heuristic “probably-rejected” rather than verified decisions, ICLR aspect scores are annotator-predicted rather than reviewer-provided, and aspect annotation covers only 1.3K reviews with many aspects often not discussed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
28 Own result · 13 No basis given · 6 Own method · 6 Own interpretation · 1 No full text of the sources
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reviews include both text and aspect scores (e.g., calrity) on a scale of 1-5.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.1 Review Collection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our best model outperforms the majority classifiers in all cases.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Another interesting observation is that the mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotators were able to infer about half of the aspect scores from the corresponding review text (the other half was not discussed in the review text).”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to increase the utility of the dataset, we annotated 1.3K reviews with aspect scores, based on the corresponding review text.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This process results in 8,887 'probably-rejected' papers.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.1 Review Collection
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The aspects which correlate most strongly with the final recommendation are substance (which concerns the amount of work rather than its quality) and clarity.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a total of 1,304 official, anonymous reviews for 427 papers (177 accepted and 255 rejected).”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.1 Review Collection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We analyzed the dataset, showing interesting trends such as a high correlation between overall recommendation and recommending an oral presentation.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 6 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline.”
- Also stated in: § 1 Introduction · plain
“Our results show that we can predict the accept/reject decisions with 6-21% error reduction compared to the majority reject-all baseline, in four different sections of PeerRead.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
- Probably not a substantial claim. 16
- It may state two separate claims as one. 14
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast, we collected and organized PeerRead such that it is easy for other researchers to use it for research purposes, replicate experiments and make a fair comparison to previous results.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 4.2 Review Aspect Score Prediction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This resulted in 39 reviews for 22 CoNLL 2016 submissions, and 275 reviews for 137 ACL 2017 submissions.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.1 Review Collection
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found 364 papers which matched these criteria, 352 out of which were labeled as 'accepted'.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.1 Review Collection
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR.The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers.”
- Also stated in: § 1 Introduction · plain
“In total, the dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions, including a subset of 3K papers for which we have 10.7K textual reviews written by experts.”
- Also stated in: § 1 Introduction · plain
“total 14,784 10,770”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most of the mean scores are similar in both sections, with a few notable exceptions.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We organize v1.0 of the PeerRead dataset in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.3 Organization and Preprocessing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We introduced PeerRead, the first publicly available peer review dataset for research purposes, containing 14.7K papers and 10.7K reviews.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 6 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The papers in each section are further split into standard training, development and test sets with 0.9:0.05:0.05ratios.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.3 Organization and Preprocessing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 6 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“arXiv 2007-2017 11,778 --2,891 / 8,887”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On average, 77% of the annotations were consistent (i.e., the re-annotation was exactly the same as the original annotation, or was off by 1 point) and 2% were inconsistent (i.e., the re-annotation was off by 2 points or more).”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 6 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, about 49% of the reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > Experimental setup.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While aspect scores assigned by reviewers are included in the opted-in sections in PeerRead, they are missing from the remaining reviews.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The CoNLL 2016 section also includes aspect scores but is too small for training.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Note that NIPS provide reviews only for accepted papers.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Presentation format Oral Poster ∆ stdev Recommendation 3.83 2.92 0.90 0.89 Substance 3.91 3.29 0.62 0.84 Clarity 4.19 3.72 0.47 0.90 Meaningful comparison 3.60 3.36 0.24 0.82 Impact 3.27 3.09 0.18 0.54 Originality 3.91 3.88 0.02 0.87 Soundness/Correctness 3.93 4.18 -0.25 0.91”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The CoNLL 2016 section also includes aspect scores but is too small for training.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The platform also allows any person to review the paper by adding a comment, but we only use the official reviews of reviewers assigned to review that paper.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This baseline is guaranteed to obtain mean square errors less than or equal to the majority baseline.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to assess annotation consistency, the same annotators re-annotated a random sample consisting of 30 reviews.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For consistency, we only include the first arXiv version of each paper (accepted or rejected) in the dataset.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If a paper has multiple versions, we consider the submission date of the first version.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We note, however, that ACL 2017 reviews were explicitly opted-in while the ICLR 2017 reviews include all official reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 2.4 Aspect Score Annotations
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that some of the 'probably-rejected' papers may be published at workshops or other venues.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, our annotators only considered the review text, and did not have access to the papers.”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > B Reviewer Instructions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“For example, the organizers of the NIPS 2014 conference assigned 10% of conference submissions to two different sets of reviewers to measure the consistency of the peer reviewing process, and observed that the two committees disagreed on the accept/reject decision for more than a quarter of the papers (Langford and Guzdial, 2015).”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Langford and Guzdial, 2015 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
Quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue.
“Despite these efforts, quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue (e.g., journal editors and program chairs).”
The paper relies on this without support.
Most peer reviews are not made publicly available.
“Surprisingly, however, most of the reviews are not made publicly available.”
The paper relies on this without support.
Review length is often thought of as a measure of review quality.
“Since review length is often thought of as a measure of its quality, this raises interesting questions about the quality of reviews in ICLR vs. ACL conferences.”
The paper relies on this without support.
In many publication venues, reviewers assign numeric aspect scores as part of the peer review.
“In many publication venues, reviewers assign numeric aspect scores (e.g., clarity, originality, substance) as part of the peer review.”
The paper relies on this without support.
The presentation format decision of accepted papers is based on recommendation by the reviewers.
“The presentation format decision of accepted papers is based on recommendation by the reviewers.”
The paper relies on this without support.
The baseline models lack the sophistication to assess the quality of the work discussed in the given paper.
“Since our models lack the sophistication to assess the quality of the work discussed in the given paper, this might indicate that some of the features we define are correlated with strong papers, or bias reviewers' judgments.”
The paper relies on this without support.
Aspects with higher RMSE scores for the 'Mean' baseline indicate higher variance among the review scores for this aspect.
“We note that aspects with higher RMSE scores for the 'Mean' baseline indicate higher variance among the review scores for this aspect, so we focus our discussion on these aspects.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite the fact that this dataset was supposed to be released periodically, the last update available in its GitHub repositoryfoot_4 dates back to May 2018.The content is currently limited, based on a few conferences before 2018.”
From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Kang et al., 2018]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”
From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Kang et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Science Parse (Sci, 2017-2025) Kang et al. (2018) PDF → JSON Fast metadata + rough text; webscale.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al. (2018)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“existing evaluations of AI-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, existing peer-review datasets usually contain heterogeneous paper-review-decision records.Their field structures, text granularity, and label sources differ”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Few studies compare process-variable decision value, evaluate chain consis-tency, and examine the interventional sensitivity of final decisions to intermediate process variables within a unified framework.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical analyses show that reviewer comments are distributed across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation rather than concentrated in a single type of problem (Dycke et al., 2023;Kang et al., 2018;Stephen, 2022).”
From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as Kang et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Review score prediction has been first introduced in (Kang et al., 2018) and further explored in (Ghosal et al., 2019).”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Kang et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Parts of the PeerRead data (Kang et al., 2018) have been created with explicit consent for publication and processing by both reviewers and authors, and we include them into our resource.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Kang et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“Early approaches in this domain were confined to sub-tasks, such as predicting paper acceptance from metadata and abstract features or regressing review scores from paper content (Kang et al., 2018;Checco et al., 2021).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“These foundational efforts deconstructed the peer review process into analytical (Teufel et al., 1999), predictive (Kang et al., 2018), and narrowly generative components (Dycke et al., 2023b).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al., 2018
The cited source says what the claim attributes to it.
In the citing paper
“These datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, although the above works have made progress in scholarly peer review, most of these datasets could not guarantee the consistency between their paper content and those under review, nor did they include rebuttals, making it difficult to support the research about the reviewers-authors discussions.”
From Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions (2025) · cites this paper as Kang et al. [2018]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > Abstract
“up to 21% error reduction compared to the majority baseline”
§ A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications > Experimental setup.
“Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“We use three neural architectures: convolutional neural networks (CNN, Zhang et al., 2015), recurrent neural networks (LSTM, Hochreiter and Schmidhuber, 1997), and deep averaging networks (DAN, Iyyer et al., 2015).”
MethodScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1×
“We use python’s sklearn’s implementation of all models (Pedregosa et al., 2011).”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2×
“We use GloVe 840B embeddings (Pennington et al., 2014) as input word representations, without tuning, and keep the 35K most frequent words and replace the rest with an UNK vector.”
MethodBinary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 4,481 · cited 1×
“We use three neural architectures: convolutional neural networks (CNN, Zhang et al., 2015), recurrent neural networks (LSTM, Hochreiter and Schmidhuber, 1997), and deep averaging networks (DAN, Iyyer et al., 2015).”
MethodDeep Unordered Composition Rivals Syntactic Methods for Text Classification
Iyyer, M., Manjunatha, V., Boyd-Graber, J. L., & Daumé, H. (2015) · doi:10.3115/v1/p15-1162 · Cited by 729 · cited 1×
“We use three neural architectures: convolutional neural networks (CNN, Zhang et al., 2015), recurrent neural networks (LSTM, Hochreiter and Schmidhuber, 1997), and deep averaging networks (DAN, Iyyer et al., 2015).”
MethodOpen peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261 · cited 2×
“Editors of the British Journal of Psychiatry found di ff erences in courtesy between signed and unsigned reviews (Walsh et al., 2000).”
BackgroundContent-Based Citation Recommendation
Bhagavatula, C., Feldman, S., Power, R., & Ammar, W. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1022 · Cited by 188 · cited 2×
“Tools like Citeomatic 29 (Bhagavatula et al., 2018) are especially useful in finding relevant articles not cited in the manuscript.”
BackgroundSelf-citationBalachandran, V. (2013) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2013.6606642 · Cited by 179
Balachandran, V. (2013) · doi:10.5555/2486788.2486915 · Cited by 145
Research works printed but not linked: 4
- Sarah Greaves, Joanna Scott, Maxine Clarke, Linda Miller, Timo Hannay, Annette Thomas, and Philip Campbell. 2006. Nature's trial of open peer review. Nature 444(971):10-1038.
- Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. 2011. A quantitative analy- sis of peer review. In Proc. of ISSI.
- Drummond Rennie. 2016. Make peer review scien- tific: thirty years on from the first congress on peer review, drummond rennie reflects on the improve- ments brought about by research into the process- and calls for more. Nature 535(7610):31-34.
- Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running aver- age of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26-31.
Cited by
Sorted by citations
Items 101–191 of 191
Interpretable Aspect-Aware Capsule Network for Peer Review Based Citation Count Prediction
Li, S., Li, Y., Zhao, W. X., Ding, B., & Wen, J.-R. (2021) · ACM Transactions on Information Systems · doi:10.1145/3466640 · Cited by 4
“Recently, a public dataset of peer reviews has been released to lower the barrier to studying peer reviews for the scientific community [33].”
BackgroundFFCD: A Fast-and-Frugal Coherence Detection Method
Duari, S., & Bhatnagar, V. (2021) · IEEE Access · doi:10.1109/access.2021.3135048 · Cited by 4
Between acceptance and rejection
Fernandes, G. L., & de Melo, P. O. S. V. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530935 · Cited by 5
“[17] made available the first public database containing 10,770 paper reviews and suggested some tasks that could be explored from it, such as predicting the score and the final decision of a paper.”
BackgroundMethodInfluentialDid AI get more negative recently?
Beese, D., Altunbaş, B., Güzeler, G., & Eger, S. (2023) · Royal Society Open Science · doi:10.1098/rsos.221159 · Cited by 4
“We use 5,453 accepted and 14,974 rejected papers from the ICLR conferences in 2013 and 2017– 2021, collected from OpenReview 21 , and from ACL, EMNLP, NAACL, NeurIPS, AAAI, ICML, and ICLR over the years 2007–2017 as collected by Kang et al. (2018).”
MethodCSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue SummarizationPreprint
Lin, H., Ma, L., Zhu, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13139 · Cited by 4
“Recently, there exist various types of document summarization datasets such as news reports (Nallapati et al., 2016; Narayan et al., 2018), wiki passages (Liu et al., 2018), and scientific papers (Kang et al., 2018).”
BackgroundHuang, J., Huang, W.-B., Bu, Y., Cao, Q., Shen, H., & Cheng, X. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101427 · Cited by 4
Automatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.684 · Cited by 15
Automatic Academic Paper Rating Based on Modularized Hierarchical Attention Network
Kang, K., Zhang, H., Li, Y., Luo, X., & Silamu, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_52 · Cited by 3
Haq, T. U., Anand, D., & Kapoor, N. (2022) · IEEE Global Conference for Advancement in Technology (GCAT) · doi:10.1109/gcat55367.2022.9971874 · Cited by 3
“[4] For a subset of the publications, the collection also includes 10.7K textual peer evaluations provided by specialists.”
BackgroundEnhancing the examination of obstacles in an automated peer review system
Fernandes, G. L., & de Melo, P. O. S. V. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00382-1 · Cited by 3
“The data provided by Kang et al. [22] were used by these two works, but other reviews needed to be collected to run their experiments.”
Combining Sentiment Lexica with a Multi-View Variational Autoencoder
Hoyle, A., Wolf-Sonkin, L., Wallach, H., Cotterell, R., & Augenstein, I. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1065 · Cited by 7
“, 2018) with splits ACL 2017 and ICLR 2017 (Kang et al., 2018).”
MethodInfluentialOverview of Argumentative Text Understanding for AI Debater Challenge
Yuan, J., Cheng, L., He, R., Li, Y., Bing, L., Wei, Z., Liu, Q., Shen, C., Zhang, S., Sun, C., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_45 · Cited by 3
Kumar, A., Ghosal, T., Bhattacharjee, S., & Ekbal, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16802-4_17 · Cited by 3
SPAMID-PAIR: A Novel Indonesian Post–Comment Pairs Dataset Containing Emoji
Chrismanto, A. R., Sari, A. K., & Suyanto, Y. (2022) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2022.0131110 · Cited by 3
“Even though learning datasets in NLP are quite widely available in English, such as IMDB Dataset [19]–[21], SMS Spam UCI [22], FLAIR [23], [24], Twitter Spam [25], [26], YouTube Comments [15], PeerRead [27], and Huggingface Community Datasets [28].”
What Have We Learned from OpenReview?Preprint
Wang, G., Peng, Q., Zhang, Y., & Zhang, M. (2022) · Research Square · doi:10.21203/rs.3.rs-1558902/v1 · Cited by 3
Reviewer2: Optimizing Review Generation Through Prompt Generation
Gao, Z., Brantley, K., & Joachims, T. (2024) · arXiv · doi:10.48550/arxiv.2402.10886 · Cited by 3
Uncertainty Aware Review Hallucination for Science Article Classification
Friedl, K., Rizos, G., Stappen, L., Hasan, M., Specia, L., Hain, T., & Schuller, B. W. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.443 · Cited by 2
“Whereas previous studies have used either the former (Kang et al., 2018; Wang and Wan, 2018; Ghosal et al., 2019) or a soft label average of the latter (Stappen et al.”
BackgroundMethodInfluentialSemi-Supervised Learning for Aspect-Based Sentiment Analysis
Zheng, H., Zhang, J., Suzuki, Y., Fukumoto, F., & Nishizaki, H. (2021) · International Conference on Cyberworlds (CW) · doi:10.1109/cw52790.2021.00042 · Cited by 3
Intermediate-Task Transfer Learning for Peer Review Score Prediction
Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2023) · doi:10.18653/v1/2023.ijcnlp-srw.6 · Cited by 2
“We also compared our method to the PeerRead (Kang et al., 2018) which is shown in Table 4.”
BackgroundMethodResultInfluentialA Content-Based Novelty Measure for Scholarly Publications: A Proof of Concept
Wang, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-57867-0_31 · Cited by 5
Reviewer 2 Must Be Stopped: Transformer-Based Approaches for Predicting Paper Acceptance
Rimkus, L., Verbickas, J., & Batista-Navarro, R. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63616-5_19 · Cited by 2
ARGUABLY @ AI Debater-NLPCC 2021 Task 3: Argument Pair Extraction from Peer Review and Rebuttals
Kohli, G. S., Kaur, P., Singh, M., Ghosal, T., & Rana, P. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_48 · Cited by 2
Dershowitz, N., & Verma, R. M. (2023) · Communications of the ACM · doi:10.1145/3584664 · Cited by 3
Determining the Credibility of Science Communication
Augenstein, I. (2021) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2021.sdp-1.1 · Cited by 2
“Research has thus focused on automatically generating peer reviews from paper content (Wang et al., 2020), as well as on studying how well review scores can be predicted from review texts (Kang et al., 2018; Plank and van Dalen, 2019).”
BackgroundOn the Identification and Forecasting of Hate Speech in Inceldom
Gajo, P., Muti, A., Korre, K., Bernardini, S., & Barrón‐Cedeño, A. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_042 · Cited by 2
“Following the approach of Kang et al. (2018), our baselines are the means of the scores contained in the development and test partitions of the produced hate score datasets.”
MethodA Joint Model for Multimodal Document Quality AssessmentPreprint
Shen, A., Salehi, B., Baldwin, T. J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.01010 · Cited by 2
“…FA indicator, which is a bronze star icon in the top right corner of the webpage) are removed from the screenshot. arXiv dataset The arXiv dataset (Kang et al. 2018) consists of three subsets of academic articles under the arXiv repository of Computer Science (cs), from the three subject areas…”
InfluentialEarlier Isn't Always Better: Sub-aspect Analysis on Corpus and System Biases in SummarizationPreprint
Jung, T., Kang, D., Mentch, L., & Hovy, E. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11723 · Cited by 10
Self-citationLLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16253 · Cited by 2
“PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”
BackgroundChang, Y., Li, Z., Zhang, H., Kong, Y., Wu, Y., So, H. K.-H., Guo, Z., Zhu, L., & Wong, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.790 · Cited by 2
Szumega, J., Bougueroua, L., Gkotse, B., Jouvelot, P., & Ravotti, F. (2023) · arXiv · doi:10.48550/arxiv.2312.04576 · Cited by 8
“They compare existing peer-review datasets and their applications; in the vast majority, the existing items are used to predict acceptance and scores ([Gao et al., 2019, Kang et al., 2018]) or citations ([Plank and van Dalen, 2019]).”
BackgroundMethodNeurReview: A Neural Architecture Based Conformity Prediction of Peer Reviews
Meng, J. (2022) · IEEE Access · doi:10.1109/access.2022.3224019 · Cited by 1
“More recently, Kang et al. [14] have collected and analyzed openly available peer review data PeerRead for the first time.”
BackgroundIntelligent Support for Manuscript Authors Using Term Clouds in Scientific Reviews
Латыпова, В. А. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-77411-9_18 · Cited by 1
Wu, W.-Q., Zhang, C.-Z., Zhao, Y., & Bao, T. (2026) · Scientometrics · doi:10.1007/s11192-026-05645-7 · Cited by 2
“As pioneers in this field, Kang et al. (2018) introduced PeerRead, a publicly available dataset for research purposes, consisting of 14.7K paper drafts and corresponding peer review reports from top venues, including ACL, NeurIPS, and ICLR.”
Arora, H., Shinde, K., & Ghosal, T. (2023) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl57899.2023.00015 · Cited by 2
Recommendations in Visual Analytics - An Analytical Approach for Elaboration in Science
Blazevic, M., Sina, L. B., Secco, C. A., et al. (2023) · International Conference Information Visualisation (IV) · doi:10.1109/iv60283.2023.00052 · Cited by 1
“Kang et al. created an extensive dataset based on 14.700 paper drafts and reviews [24].”
BackgroundInfluentialA GAN-BERT based decision making approach in peer review
Boukhris, I., & Zaâbi, C. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01269-y · Cited by 1
“In our work, we used the ACL2017 and ICLR2017 datasets from the PeerRead dataset (Kang et al. 2018).”
Does opening up peer review benefit science in terms of citations?
Ni, J., Zhao, Z.-Y., Shao, Y., Liu, S., Li, W., & Li, J. (2020) · ASIS&T Annual Meeting · doi:10.1002/pra2.258 · Cited by 1
“For example, Kang et al. (2018) provide the first public data set of scientific peer reviews that can be used for research.”
BackgroundA Lightweight and Explainable Citation Recommendation System
Yin, J. (2021) · Scholarship@Western (Western University) · Cited by 1
NUS-IDS at PragTag-2023: Improving Pragmatic Tagging of Peer Reviews through Unlabeled Data
Gollapalli, S. D., Huang, Y., & Ng, S.-K. (2023) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/2023.argmining-1.25 · Cited by 1
“Recently, efforts are underway for collecting relevant datasets for designing assistive automation aids for peer review (Yuan et al., 2022; Checco et al., 2021; Kang et al., 2018; Ghosal et al., 2022; Dycke et al., 2023a).”
BackgroundEvaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence RepresentationsPreprint
Chen, M., Chu, Z., & Gimpel, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00142 · Cited by 5
“We construct the dataset from PeerRead (Kang et al., 2018), which consists of scientific papers from a variety of fields.”
MethodWeightScale: Interpreting Weight Change in Neural NetworksPreprint
Agrawal, A., Tendle, A., Sikka, H., et al. (2023) · doi:10.31219/osf.io/2cnb9 · Cited by 1
Ranking Scientific Papers Using Preference Learning.Preprint
Dycke, N., Simpson, E., Kuznetsov, I., & Gurevych, I. (2021) · arXiv (Cornell University) · Cited by 1
Hasan, M. T., Shamael, M. N., Billah, H. M. M., Akter, A., Hossain, M. A. E., Islam, S., Islam, S., & Shatabda, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5142870 · Cited by 4
OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
Idahl, M., & Ahmadi, Z. (2024) · arXiv · doi:10.48550/arxiv.2412.11948 · Cited by 1
Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions
Zhang, D., Bao, Z., Du, S., Zhao, Z., Zhang, K., Bao, D., & Yang, Y. (2025) · arXiv · doi:10.48550/arxiv.2505.07920 · Cited by 26
“A few other works [Kang et al., 2018, Shen et al., 2021, Yuan et al., 2022, Dycke et al., 2022, D’Arcy et al., 2023, Gao et al., 2024] incorporate reviews from additional conferences like NeurIPS and ACL.”
Linguistic Versus Latent Relations for Modeling Coherent Flow in ParagraphsPreprint
Kang, D., Hayashi, H., Black, A. W., & Hovy, E. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11790
Self-citationWikiAsp: A Dataset for Multi-domain Aspect-based SummarizationPreprint
Hayashi, H., Budania, P., Wang, P., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07832
“In contrast, generic summarization models are tested on a much wider variety of gen-res, from newswire (Nallapati et al., 2016; Grusky et al., 2018), to academic papers (Kang et al., 2018; Kedzie et al., 2018), to movie scripts (Gorinski and Lapata, 2015).”
BackgroundDetermining the Credibility of Science CommunicationPreprint
Augenstein, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14473
“Research has thus focused on automatically generating peer reviews from paper content (Wang et al., 2020), as well as on studying how well review scores can be predicted from review texts (Kang et al., 2018; Plank and van Dalen, 2019).”
BackgroundOn Positivity Bias in Negative ReviewsPreprint
Aithal, M., & Tan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12056
“• PeerRead (Kang et al., 2018).”
Towards controllable neural generation of arguments
Hua, X. (2021) · Digital Repository Service (Northeastern University) · doi:10.17760/d20412319
What Have We Learned from OpenReview?
Wang, G., Peng, Q., Zhang, Y., & Zhang, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85896-4_6 · Cited by 3
MReD: A Meta-Review Dataset for Controllable Text Generation.Preprint
Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., & Si, L. (2021) · arXiv (Cornell University)
Unsupervised Learning for the Automatic Management of Scientific Aspect in Academic Conferences
Medakene, A. N., Eddoud, A. M. A., & Bouanane, K. (2022) · Smart innovation, systems and technologies · doi:10.1007/978-981-19-0011-2_1
Impact factor analysis report of paper acceptance at ICLR
Hu, S. (2023) · Atlantis Highlights in Computer Sciences/Atlantis highlights in computer sciences · doi:10.2991/978-94-6463-102-9_68
A Modular Hierarchical Model for Paper Quality Evaluation
Deng, X., Li, S., Yu, J., Ma, J., Ji, B., Lin, W., Song, S., & Yi, Z. (2023) · Artificial Intelligence Advances · doi:10.5121/csit.2023.130702
Aspect-based sentiment analysis of online peer reviews and prediction of paper acceptance results
(2023) · Data Science and Informetrics · doi:10.59494/dsi.2023.1.4 · Cited by 1
Hybrid Deep Learning Based Model on Sentiment Analysis of Peer Reviews on Scientific Papers
Sarkar, R., Singh, P., Jaber, M. M., Nandan, S., Mishra, S., Satapathy, S. K., & Pattnaik, C. R. (2023) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-35081-8_9
A Dataset of Argumentative Dialogues on Scientific Papers
Ruggeri, F., Mesgar, M., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.425 · Cited by 10
“While the problem of distinguishing between opinionated and factual content is not new (Yu and Hatzivassiloglou, 2003; Kang et al., 2018; Hua et al., 2019; Cheng et al., 2020), there is no dataset”
BackgroundKuznetsov, I., & Gurevych, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.633
ReviVal: Towards Automatically Evaluating the Informativeness of Peer Reviews
Verma, R., Ghosal, T., Bhattacharjee, S., Ekbal, A., & Bhattacharyya, P. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625341
“• Review-Paper Aspect Category We follow the ACL 2016 reviewing guidelines for our aspect labels (Section 3.2) [20].”
BackgroundInfluentialExploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals
Purkayastha, S., Lauscher, A., & Gurevych, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.894 · Cited by 13
Improving Peer-Review Score Prediction via Pretrained Model with Intermediate-Task Training
Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2024) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom60618.2024.10418356
“PeerRead stands out as the inaugural openly accessible peer-review dataset [1], providing peer review details, e.g., final decisions, aspect scores, and review contents.”
BackgroundMethodInfluentialAPCS: Towards Argument Based Pros and Cons Summarization of Peer Reviews
Kumar, S., Ghosal, T., & Ekbal, A. (2023) · Workshop on Information Extraction from Scientific Publications (WIESP) · doi:10.18653/v1/2023.wiesp-1.14
“A dataset of scientific peer reviews was made available to facilitate research in this domain(Kang et al., 2018).”
BackgroundGPT as a Reviewer: Automatic Evaluation of Academic Papers
Tas, B., & Aksoy, M. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00080
“Reference [18] introduced PeerRead, a dataset that has become a valuable resource for training classifiers to predict paper acceptance.”
Bharti, P. K., Agarwal, M., & Ekbal, A. (2025) · Journal of Information Science · doi:10.1177/01655515251353189
Analyzing we-sentences in Scientific Writing to Predict Peer Review Outcomes
López-Reyes, G. A., Ortega-Mendoza, R. M., Pérez-Cortes, O., Calderón-Suárez, R., & Castro-Espinoza, F. A. (2025) · IEEE Access · doi:10.1109/access.2025.3606415
“For evaluating the ideas of this paper, we used two English benchmark datasets related to scientific reviewing: PeerRead [12] and ASpect-enhAnced Peer Review dataset (ASAP-Review) [11].”
InfluentialDynamic optimization of peer review length using information density analysis
Bharti, P. K., & Bhathawala, T. (2025) · Scientometrics · doi:10.1007/s11192-025-05416-w
“The PeerRead dataset (Kang et al., 2018) has provided a valuable baseline for analyzing the textual and structural attributes of peer reviews.”
InfluentialAn Interpretable Soft-Sensor Framework for Dissertation Peer Review Using BERT
Wang, M., Su, J., Chen, Z.-D., Yang, W.-C., & Yang, X. (2025) · Italian National Conference on Sensors · doi:10.3390/s25206411
“To further evaluate the generalizability of our proposed method, we identified several top-performing models from the table above and evaluated them on the PeerRead dataset [29].”
Identifying New Promising Research Directions with Open Peer Reviews and Contextual Top2Vec
Devyatkin, D. A., Sochenkov, I. V., Popov, D., Zubarev, D., Ryzhova, A., Abanin, F., & Grigoriev, O. G. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9120319
“Study [17] describes the review score prediction and the paper decision prediction on two datasets with ICLR submissions; a similar dataset is also proposed in the paper [18].”
Kumar, S., Arora, H., Ghosal, T., & Ekbal, A. (2026) · Scientometrics · doi:10.1007/s11192-025-05529-2
“For instance, the PeerRead dataset (Kang et al., 2018) includes drafts of papers and their corresponding acceptance or rejection decisions from top-tier conferences.”
Bharti, P. K. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01059-z · Cited by 1
“Kang et al. [29] introduced the PeerRead dataset, and Ghosal et al. [30] proposedDeepSentiPeerforacceptanceandscoreprediction.”
Assisted authoring for avoiding inadequate claims in scientific reporting
Koroleva, A. (2020) · theses.fr (ABES) · doi:10.70675/f035bb8azddf1z4a7dzbf39zdf86c6947bb9
“…important for automated data analysis techniques (e.g. Natural Language Processing - NLP) which are being developed to automate some parts of the systematic review process (cf. Ananiadou et al. (2009); O’Mara-Eves et al. (2015)) and peer review (cf. Gehringer et al. (2018); Kang et al. (2018)).”
MethodDe la complexité de l'annotation manuelle : méthodologie, biais et recommandations
Baledent, A. (2022) · theses.fr (ABES) · doi:10.70675/0cd58283zd4d9z471bza157z20e9f7ba2a9f
All Accept, No Reject: Evaluating LLMs as “Peer” Reviewers
Verma, N., & Landrum, A. R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791300 · Cited by 1
“For the first study, we used a subset of data from an openly available dataset of manuscripts and human-written peer reviews called PeerRead [15].”
InfluentialChen, X., Meng, X., Lv, S., Zhang, Y., & Chen, Y. J. (2026) · Scientometrics · doi:10.1007/s11192-026-05594-1
“We conducted a comparative experiment using the PeerRead dataset (Kang et al. 2018), a standard benchmark containing papers and their acceptance decisions.”
Tan, F., Liu, G., & Zhao, R. (2026) · IEEE Access · doi:10.1109/access.2026.3718020
“PeerRead is built on the full PeerRead dataset released in [28], which collects submissions spanning 2007 to 2017 from top AI venues including ACL, EMNLP, NAACL, NIPS, ICLR and AAAI.”
InfluentialWho's Adam? Benchmarking Hallucinations in Scientific Dialogue
Zhang, Z., Lei, T., Yang, K., & Li, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817577
RECO-Rank: A Compensatory Multi-Evidence Ranking Model for Cross-Domain Reviewer RecommendationPreprint
Chen, J., Xu, J., & Fang, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9558981/v1
Predicting Replication from Domain-Stripped Fingerprints, Where Citations FailPreprint
Kulikowski, E. (2026) · Research Square · doi:10.21203/rs.3.rs-10335634/v1 · Cited by 1
Design and Development of a Robust Assamese Language Dataset for Author Attribution
Medhi, S. P., & Sarma, S. K. (2026) · Springer proceedings in mathematics & statistics · doi:10.1007/978-981-95-4813-2_27
Pugazhanthi, N., Jettipalle, D. S., G., R. M., et al. (2026) · Proceedings of the 2026 15th International Conference on Software and Computer Applications · doi:10.1145/3801619.3801643
Yang, C., & Zhang, C. (2026) · Research Square · doi:10.21203/rs.3.rs-10959205/v1
Does reviewers’ sentiment disagreement during peer review predict novelty and disruption?
Sun, Z., Zhang, X., Li, M., & Wang, Y. (2026) · Scientometrics · doi:10.1007/s11192-026-05816-6
Peerify: Benchmarking Peer-Review Claim Verification
Daghighfarsoodeh, A., Ebrahimi, S., Ghorbanpour, A., Sadeghian, S., Cheraghi, R., Arabzadeh, N., & Bagheri, E. (2026) · arXiv · doi:10.48550/arxiv.2609.25046
“A related line analyzes peer-review content and dynamics, including review quality, argumentation, sentiment, and reviewer behavior (Hua et al., 2019; Rogers and Augenstein, 2020; Kang et al., 2018; Ebrahimi et al., 2025a).”
Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers
Yang, Y., Thelwall, M., & He, G. (2026) · arXiv · doi:10.48550/arxiv.2609.01895
“Empirical analyses show that reviewer comments are distributed across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation rather than concentrated in a single type of problem (Dycke et al., 2023; Kang et al., 2018; Stephen, 2022).”
Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Yuan, S., Zhang, X., Ni, W., Xiao, T., Di, S., Zhu, J., Jiang, Z., Tan, R., Chen, L., & Ren, K. (2026) · arXiv · doi:10.48550/arxiv.2609.05947
“We construct process-aligned data from PeerRead, NLPeer ARR-22, and OpenReview-ICLR.”
InfluentialCoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
Deng, H., Ke, X., Li, Y., Hu, R., Huang, D., Wong, D. F., Wang, Y., Liu, X., & Zhang, M. (2026) · arXiv · doi:10.48550/arxiv.2605.07905
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Li, M., Wang, C., Li, X., Zeng, X., Li, D., Shi, P., Zhou, D., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2608.08975
Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attribution
Saxena, Y., Bommireddy, R., Padia, A., & Gaur, M. (2025) · arXiv · doi:10.48550/arxiv.2509.21557
CASIMIR: A Corpus of Scientific Articles Enhanced with Multiple Author-Integrated Revisions
Jourdan, L., Boudin, F., Hernandez, N., & Dufour, R. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/4aa2twhyqaqa
Metadata
- Authors
- Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R.
- Affiliations
- Carnegie Mellon University (US)
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
- Published
- 2018
- Pages
- 1647-1661
- OpenAlex
- W2964211782
- Semantic Scholar
- 53489041a08ef1a6cd19b4c95ce092a148283b6b
- Cited by
- 270
- References
- 22
- Citations by year
- 2026: 52 · 2025: 40 · 2024: 27 · 2023: 43 · 2022: 28 · 2021: 32 · 2020: 27 · 2019: 20 · 2018: 1
- FWCI
- 7.10
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- peer review, scholarly publishing, natural language processing applications, research evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
