A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Open accessAbstract
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.
AI snapshot
Written from the paper’s full text.
TL;DR
PeerRead is introduced as the first public dataset of scientific peer reviews, containing 14.7K paper drafts with accept/reject decisions and 10.7K textual reviews from ACL, NIPS, ICLR, CoNLL, and arXiv. The paper analyzes review patterns and defines two NLP tasks: predicting paper acceptance and predicting review aspect scores.
Key findings
- PeerRead v1 contains 14,784 papers and 10,770 reviews across NIPS 2013–2017 (2,420 papers, 9,152 reviews), ICLR 2017 (427 papers, 1,304 reviews), ACL 2017 (137 papers, 275 reviews), CoNLL 2016 (22 papers, 39 reviews), and arXiv 2007–2017 (11,778 papers, no reviews).
- In ACL 2017 reviews, overall recommendation correlates most with substance (ρ = 0.59) and clarity (ρ = 0.42), and least with soundness/correctness (ρ = 0.01) and originality (ρ = 0.08).
- Reviews recommending oral presentation have a mean overall recommendation of 3.83 versus 2.92 for poster recommendations, a difference of 0.90 on a 1–5 scale.
- Acceptance classification outperforms majority baselines in all tested sections: ICLR 65.3% vs. 57.6%, arXiv cs.cl 75.7% vs. 68.9%, cs.lg 70.7% vs. 67.9%, and cs.ai 92.6% vs. 92.1%; the abstract reports up to 21% error reduction, while the results section states up to 22%.
- For aspect score prediction, the paper reports that the “Paper;Review” model outperforms the mean baseline in all four high-variance aspects examined, and the “Review” model outperforms the mean baseline in three out of four.
Methods
- Dataset construction used three strategies: opt-in collection with Softconf for CoNLL 2016 and ACL 2017; crawling publicly available NIPS 2013–2017 and ICLR 2017 OpenReview reviews; and crawling arXiv 2007–2017 submissions, labeling accepted papers via DBLP matching and title Levenshtein distance < 0.1, and labeling “probably-rejected” papers by venue, arXiv category, cross-listing, and submission-date criteria.
- Aspect scores were annotated for 1.3K ICLR 2017 reviews by two authors across seven aspects (appropriateness, clarity, originality, soundness/correctness, meaningful comparison, substance, impact) on a 1–5 scale, with a “not discussed” option; re-annotation of 30 reviews showed 77% consistent annotations, 2% inconsistent, and 21% “not discussed” mismatches.
- NLP experiments included binary acceptance classification using 22 coarse features plus lexical features with classifiers such as logistic regression, SVM, Random Forest, and MLP, and aspect score regression using CNN, LSTM, and DAN architectures with GloVe 840B embeddings, evaluated by RMSE against a mean-score baseline.
Limitations
- The paper states that ACL 2017 reviews were explicitly opted-in, likely biasing included ACL reviews toward higher quality, and that lower ICLR appropriateness scores are probably an annotation artifact because reviewers mention appropriateness mainly when a paper is inappropriate.
- The paper notes that NIPS provides reviews only for accepted papers, that some “probably-rejected” arXiv papers may have been published at workshops or other venues, and that one false negative was found due to a significant title change.
- The paper states its models are simple and intended to demonstrate dataset potential rather than be best-performing; evident from the text, arXiv rejection labels are heuristic “probably-rejected” rather than verified decisions, ICLR aspect scores are annotator-predicted rather than reviewer-provided, and aspect annotation covers only 1.3K reviews with many aspects often not discussed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 1 cited sources checked · 0 supported · 0 flagged
- Findingown data
Reviews in the opted-in sections include both text and aspect scores on a scale of 1-5.
“Reviews include both text and aspect scores (e.g., calrity) on a scale of 1-5.”
Quote in the paperNot a substantial claim - Findingown data
Our best model outperforms the majority classifiers in all cases.
“Our best model outperforms the majority classifiers in all cases.”
Quote in the paperSubstantial claim - Findingown data
The mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.
“Another interesting observation is that the mean appropriateness score is lower in ICLR 2017 compared to ACL 2017.”
Quote in the paperNot a substantial claim - Findingown data
In the feasibility study, annotators were able to infer about half of the aspect scores from the corresponding review text.
“The annotators were able to infer about half of the aspect scores from the corresponding review text (the other half was not discussed in the review text).”
Quote in the paperSubstantial claim - Findingown data
The authors annotated 1.3K reviews with aspect scores based on the corresponding review text.
“In order to increase the utility of the dataset, we annotated 1.3K reviews with aspect scores, based on the corresponding review text.”
Quote in the paperSubstantial claim
- Findingown data
The arXiv labeling process resulted in 8,887 'probably-rejected' papers.
“This process results in 8,887 'probably-rejected' papers.”
Quote in the paperNot a substantial claim - Findingown data
In ACL 2017, the aspects most strongly correlated with the final recommendation are substance and clarity.
“The aspects which correlate most strongly with the final recommendation are substance (which concerns the amount of work rather than its quality) and clarity.”
Quote in the paperSubstantial claim - Findingown data
The ICLR 2017 collection consists of 1,304 official anonymous reviews for 427 papers (177 accepted and 255 rejected).
“a total of 1,304 official, anonymous reviews for 427 papers (177 accepted and 255 rejected).”
Quote in the paperNot a substantial claim - Findingown data
There is a high correlation between overall recommendation and recommending an oral presentation in PeerRead.
“We analyzed the dataset, showing interesting trends such as a high correlation between overall recommendation and recommending an oral presentation.”
Quote in the paperNot a substantial claim - Findingown data
Simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline.
“In the first task, we show that simple models can predict whether a paper is accepted with up to 21% error reduction compared to the majority baseline.”
- Also stated in: § 1 Introduction · plain
“Our results show that we can predict the accept/reject decisions with 6-21% error reduction compared to the majority reject-all baseline, in four different sections of PeerRead.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · plain
- Findingown data
In ACL 2017, soundness/correctness and originality are least correlated with the final recommendation.
“In contrast, soundness/correctness and originality are least correlated with the final recommendation.”
Quote in the paperSubstantial claim - Findingreasoning
In ACL 2017, the two aspects with the highest variance are 'originality' and 'clarity'.
“In the ACL 2017 section, the two aspects with the highest variance are 'originality' and 'clarity'.”
Quote in the paperNot a substantial claim - Findingown data
The NIPS 2013-2017 collection consists of 9,152 reviews for 2,420 papers.
“We collected all accepted papers and their reviews for NIPS 2013-2017, a total of 9,152 reviews for 2,420 papers.”
Quote in the paperNot a substantial claim - Findingown data
Comments in ACL 2017 reviews tend to be about 50% longer than those in ICLR 2017 reviews.
“The comments in ACL 2017 reviews tend to be about 50% longer than those in the ICLR 2017 reviews.”
Quote in the paperNot a substantial claim - Findingown data
Simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.
“In the second task, we predict the numerical scores of review aspects and show that simple models can outperform the mean baseline for aspects with high variance such as 'originality' and 'impact'.”
Quote in the paperSubstantial claim - Findingnone given
The 'Paper;Review' model outperforms the 'Mean' baseline in all four aspects.
“Surprisingly, the 'Paper;Review' model outperforms the 'Mean' baseline in all four aspects, and the 'Review' model outperforms the 'Mean'”
Quote in the paperNot a substantial claim - Findingreasoning
In ICLR 2017, the two aspects with the highest variance are 'appropriateness' and 'meaningful comparison'.
“In the ICLR 2017 section, the two aspects with the highest variance are 'appropriateness' and 'meaningful comparison'.”
Quote in the paperNot a substantial claim - Findingnone given
On average, all models slightly improve over the 'Mean' baseline.
“On average, all models slightly improve over the 'Mean' baseline.”
Quote in the paperNot a substantial claim - Findingown data
In ACL 2017, the average 'overall recommendation' score in reviews recommending an oral presentation is 0.9 higher than in reviews recommending a poster presentation.
“Notably, the average 'overall recommendation' score in reviews recommending an oral presentation is 0.9 higher than in reviews recommending a poster presentation, suggesting that reviewers tend to recommend oral presentation for submissions which are holistically stronger.”
Quote in the paperSubstantial claim - Findingnone given
PeerRead is the first publicly available peer review dataset for research purposes.
“We introduced PeerRead, the first publicly available peer review dataset for research purposes, containing 14.7K papers and 10.7K reviews.”
Quote in the paperNot a substantial claim - Findingreasoning
The arXiv labeling process resulted in 2,891 accepted papers.
“This results in a total of 2,891 accepted papers.”
Quote in the paperNot a substantial claim - Findingown data
On average, the absolute difference between annotated scores and gold scores is 0.51 on a 1-5 scale, considering only cases where the aspect was discussed.
“On average, the absolute difference between our annotated scores and the gold scores originally provided by reviewers is 0.51 (on a 1-5 scale, considering only those cases where the aspect was discussed in the review text).”
Quote in the paperSubstantial claim - Findingown data
Manual inspection of the remaining 12 papers showed one false negative due to a significant change in the paper title.
“Manual inspection of the remaining 12 papers showed that one of the papers was indeed a false negative (i.e., labeled as 'probably-rejected' but accepted to one of the target venues) due to a significant change in the paper title.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The lower mean appropriateness score in ICLR 2017 is probably an artifact of the annotation process: reviewers probably only address appropriateness explicitly if the paper is inappropriate, leading to a strong negative bias against this category in the ICLR dataset.
“While this might indicate that ICLR 2017 attracted more irrelevant submissions, this is probably an artifact of our annotation process: reviewers probably only address appropriateness explicitly in their review if the paper is inappropriate, which leads to a strong negative bias against this category in our ICLR dataset.”
Quote in the paperNot a substantial claim - Interpretationown data
Reviewers tend to recommend oral presentation for submissions which are holistically stronger.
“Notably, the average 'overall recommendation' score in reviews recommending an oral presentation is 0.9 higher than in reviews recommending a poster presentation, suggesting that reviewers tend to recommend oral presentation for submissions which are holistically stronger.”
Quote in the paperSubstantial claim - Interpretationnone given
PeerRead v1 is the first public dataset of scientific peer reviews available for research purposes.
“We present the first public dataset of scientific peer reviews available for research purposes (PeerRead v1), 1 providing an opportunity to study this important artifact.”
- Also stated in: § 1 Introduction · plain
“The goal of this paper is to lower the barrier to studying peer reviews for the scientific community by introducing the first public dataset of peer reviews for research purposes: PeerRead.”
Quote in the paperSubstantial claim- Rests on an assumption: Quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue.
- Rests on an assumption: Most peer reviews are not made publicly available.
- Also stated in: § 1 Introduction · plain
- Classified as probably not a substantial claim. 16
- May combine two claims. 14
Classified as probably not a substantial claim.
Findingnone givenPeerRead is collected and organized such that it is easy for other researchers to use it for research purposes, replicate experiments and make a fair comparison to previous results.
“In contrast, we collected and organized PeerRead such that it is easy for other researchers to use it for research purposes, replicate experiments and make a fair comparison to previous results.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe opted-in review collection resulted in 39 reviews for 22 CoNLL 2016 submissions and 275 reviews for 137 ACL 2017 submissions.
“This resulted in 39 reviews for 22 CoNLL 2016 submissions, and 275 reviews for 137 ACL 2017 submissions.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the data quality check, 364 arXiv papers matched the criteria, 352 of which were labeled as 'accepted'.
“We found 364 papers which matched these criteria, 352 out of which were labeled as 'accepted'.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataPeerRead consists of 14.7K paper drafts with accept/reject decisions, including a subset of 3K papers with 10.7K expert textual reviews.
“The dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues including ACL, NIPS and ICLR.The dataset also includes 10.7K textual peer reviews written by experts for a subset of the papers.”
- Also stated in: § 1 Introduction · plain
“In total, the dataset consists of 14.7K paper drafts and the corresponding accept/reject decisions, including a subset of 3K papers for which we have 10.7K textual reviews written by experts.”
- Also stated in: § 1 Introduction · plain
“total 14,784 10,770”
Quote in the paperSubstantial claim- Also stated in: § 1 Introduction · plain
May combine two claims.
Findingown dataMost mean scores are similar in the ACL 2017 and ICLR 2017 sections, with a few notable exceptions.
“Most of the mean scores are similar in both sections, with a few notable exceptions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenPeerRead v1.0 is organized in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.
“We organize v1.0 of the PeerRead dataset in five sections: CoNLL 2016, ACL 2017, ICLR 2017, NIPS 2013-2017 and arXiv 2007-2017.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataPeerRead contains 14.7K papers and 10.7K reviews.
“We introduced PeerRead, the first publicly available peer review dataset for research purposes, containing 14.7K papers and 10.7K reviews.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenPapers in each section are split into training, development and test sets with 0.9:0.05:0.05 ratios.
“The papers in each section are further split into standard training, development and test sets with 0.9:0.05:0.05ratios.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingnone givenThe paper defines a novel task of predicting the score of each aspect in a review based on the paper and review contents.
“We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe arXiv section contains 11,778 papers, with 2,891 accepted and 8,887 probably-rejected labels.
“arXiv 2007-2017 11,778 --2,891 / 8,887”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataOn average, 77% of annotations were consistent (exact or off by 1 point) and 2% were inconsistent (off by 2 points or more).
“On average, 77% of the annotations were consistent (i.e., the re-annotation was exactly the same as the original annotation, or was off by 1 point) and 2% were inconsistent (i.e., the re-annotation was off by 2 points or more).”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingnone givenThe paper defines a novel task of predicting the acceptance of a paper based on textual features.
“We defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAbout 49% of the reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.
“For example, about 49% of the reviews discussed the 'originality' aspect, while only 5% discussed 'appropriateness'.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe best model outperforms the majority classifier in all cases, with up to 22% error reduction.
“Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenAspect scores assigned by reviewers are included in the opted-in sections but missing from the remaining reviews.
“While aspect scores assigned by reviewers are included in the opted-in sections in PeerRead, they are missing from the remaining reviews.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe CoNLL 2016 section includes aspect scores.
“The CoNLL 2016 section also includes aspect scores but is too small for training.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenNIPS provides reviews only for accepted papers.
“Note that NIPS provide reviews only for accepted papers.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn ACL 2017, reviews recommending oral presentation have higher mean scores than poster for substance, clarity, meaningful comparison, and impact, but lower for soundness/correctness, and similar for originality.
“Presentation format Oral Poster ∆ stdev Recommendation 3.83 2.92 0.90 0.89 Substance 3.91 3.29 0.62 0.84 Clarity 4.19 3.72 0.47 0.90 Meaningful comparison 3.60 3.36 0.24 0.82 Impact 3.27 3.09 0.18 0.54 Originality 3.91 3.88 0.02 0.87 Soundness/Correctness 3.93 4.18 -0.25 0.91”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe CoNLL 2016 section is too small for training.
“The CoNLL 2016 section also includes aspect scores but is too small for training.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenOnly official reviews of reviewers assigned to review that paper are used.
“The platform also allows any person to review the paper by adding a comment, but we only use the official reviews of reviewers assigned to review that paper.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThis baseline is guaranteed to obtain mean square errors less than or equal to the majority baseline.
“This baseline is guaranteed to obtain mean square errors less than or equal to the majority baseline.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataThe same annotators re-annotated a random sample of 30 reviews to assess annotation consistency.
“In order to assess annotation consistency, the same annotators re-annotated a random sample consisting of 30 reviews.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningFor consistency, only the first arXiv version of each paper (accepted or rejected) is included in the dataset.
“For consistency, we only include the first arXiv version of each paper (accepted or rejected) in the dataset.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenIf a paper has multiple versions, the submission date of the first version is considered.
“If a paper has multiple versions, we consider the submission date of the first version.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningACL 2017 reviews were explicitly opted-in while ICLR 2017 reviews include all official reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.
“We note, however, that ACL 2017 reviews were explicitly opted-in while the ICLR 2017 reviews include all official reviews, which is likely to result in a positive bias in review quality of the ACL reviews included in this study.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Study design and conductnone givenSome 'probably-rejected' papers may be published at workshops or other venues.
“Note that some of the 'probably-rejected' papers may be published at workshops or other venues.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenAnnotators only considered the review text, and did not have access to the papers.
“Importantly, our annotators only considered the review text, and did not have access to the papers.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationNIPS 2014 organizers assigned 10% of conference submissions to two different sets of reviewers and observed that the two committees disagreed on the accept/reject decision for more than a quarter of the papers.
“For example, the organizers of the NIPS 2014 conference assigned 10% of conference submissions to two different sets of reviewers to measure the consistency of the peer reviewing process, and observed that the two committees disagreed on the accept/reject decision for more than a quarter of the papers (Langford and Guzdial, 2015).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Langford and Guzdial, 2015 source
The paper relies on this without support.
Quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue.
“Despite these efforts, quantitative studies of peer reviews had been limited, for the most part, to the few individuals who had access to peer reviews of a given venue (e.g., journal editors and program chairs).”
The paper relies on this without support.
Most peer reviews are not made publicly available.
“Surprisingly, however, most of the reviews are not made publicly available.”
The paper relies on this without support.
Review length is often thought of as a measure of review quality.
“Since review length is often thought of as a measure of its quality, this raises interesting questions about the quality of reviews in ICLR vs. ACL conferences.”
The paper relies on this without support.
In many publication venues, reviewers assign numeric aspect scores as part of the peer review.
“In many publication venues, reviewers assign numeric aspect scores (e.g., clarity, originality, substance) as part of the peer review.”
The paper relies on this without support.
The presentation format decision of accepted papers is based on recommendation by the reviewers.
“The presentation format decision of accepted papers is based on recommendation by the reviewers.”
The paper relies on this without support.
The baseline models lack the sophistication to assess the quality of the work discussed in the given paper.
“Since our models lack the sophistication to assess the quality of the work discussed in the given paper, this might indicate that some of the features we define are correlated with strong papers, or bias reviewers' judgments.”
The paper relies on this without support.
Aspects with higher RMSE scores for the 'Mean' baseline indicate higher variance among the review scores for this aspect.
“We note that aspects with higher RMSE scores for the 'Mean' baseline indicate higher variance among the review scores for this aspect, so we focus our discussion on these aspects.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PeerRead's last update in its GitHub repository dates back to May 2018, and its content is limited to a few conferences before 2018.
“Despite the fact that this dataset was supposed to be released periodically, the last update available in its GitHub repositoryfoot_4 dates back to May 2018.The content is currently limited, based on a few conferences before 2018.”
From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Kang et al., 2018]
The Introduction says the authors planned periodic releases, and §2.3 lists five sections in version 1.0, covering conferences through 2017 and arXiv submissions through 2017. The paper does not establish that May 2018 was the last GitHub update or that the dataset remained limited to those sections at the time of the citing sentence. (LLM, conf=0.86)
Experimental — accuracy not yet measuredPeerRead, MOPRD, and NLPeer are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.
“PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”
From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Kang et al., 2018
The Abstract reports 14.7K paper drafts and 10.7K corresponding reviews written by experts, supporting the description of PeerRead as a large paper-and-review dataset. But §2.4, “Aspect Score Annotations,” says the authors annotated 1.3K reviews with aspect scores, so describing PeerRead as without annotations is not supported. This source does not establish the properties of MOPRD or NLPeer. (LLM, conf=0.88)
Experimental — accuracy not yet measuredScience Parse provides fast metadata and rough text at web scale.
“Science Parse (Sci, 2017-2025) Kang et al. (2018) PDF → JSON Fast metadata + rough text; webscale.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al. (2018)
Section 2.3, Organization and Preprocessing (page not available), says the authors used Science Parse to extract text from paper PDFs and stored the resulting PeerRead records in JSON-encoded files. The paper does not establish that Science Parse itself outputs JSON, is fast, produces only rough text, or operates at web scale. (LLM, conf=0.85)
Experimental — accuracy not yet measuredExisting evaluations of AI-assisted peer review mainly focus on independent tasks or on final review quality and human alignment.
“existing evaluations of AI-assisted peer review mainly focus on the overall quality of generated reviews or the accuracy of final decisions.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
Section 4 (“NLP Tasks”) describes predicting paper acceptance, and Table 5 reports test accuracy for that task, supporting the final-decision accuracy part. The paper does not establish that existing AI-assisted peer-review evaluations mainly focus on this outcome or on the quality of generated reviews. Page numbers were not available in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measuredExisting peer-review datasets usually contain heterogeneous paper-review-decision records with differing field structures, text granularity, and label sources.
“First, existing peer-review datasets usually contain heterogeneous paper-review-decision records.Their field structures, text granularity, and label sources differ”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
Section 2.3 describes PeerRead’s paper-review-decision records, and Sections 2.1 and 2.4 show that its sections differ in review availability and how labels or scores were obtained. But the paper describes PeerRead as the first public peer-review dataset for research; it does not establish that existing datasets usually have differing field structures or text granularity. Section 2.3 instead describes a common JSON representation for PeerRead’s records. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Few studies compare process-variable decision value, evaluate chain consistency, and examine interventional sensitivity of final decisions to intermediate process variables within a unified framework.
“Few studies compare process-variable decision value, evaluate chain consis-tency, and examine the interventional sensitivity of final decisions to intermediate process variables within a unified framework.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
The Introduction says quantitative studies of peer reviews had been limited, and §2.4 compares how review aspect scores correlate with overall recommendations. The cited paper does not establish that few studies examine all three topics in a unified framework; the passages found do not evaluate chain consistency or interventional sensitivity to intermediate variables. No page number was provided for these passages. (LLM, conf=0.82)
Experimental — accuracy not yet measuredReviewer comments are distributed across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation rather than concentrated in a single type of problem.
“Empirical analyses show that reviewer comments are distributed across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation rather than concentrated in a single type of problem (Dycke et al., 2023;Kang et al., 2018;Stephen, 2022).”
From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as Kang et al., 2018
Section 2.4 (page unavailable) reports that textual reviews discuss multiple aspects, including originality, clarity, soundness and impact, at different rates. It does not establish the claimed distribution across all eight listed types: methodology, theory and writing are not categories in its seven-aspect annotation, and recommendation is treated as an overall judgment rather than a comment type. (LLM, conf=0.85)
Experimental — accuracy not yet measuredReview score prediction has been first introduced in Kang et al. 2018 and further explored in Ghosal et al. 2019.
“Review score prediction has been first introduced in (Kang et al., 2018) and further explored in (Ghosal et al., 2019).”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Kang et al., 2018
Parts of the PeerRead data have been created with explicit consent for publication and processing by both reviewers and authors, and are included in NLPEER.
“Parts of the PeerRead data (Kang et al., 2018) have been created with explicit consent for publication and processing by both reviewers and authors, and we include them into our resource.”
From NLPeer: A Unified Resource for the Computational Study of Peer Review (2023) · cites this paper as Kang et al., 2018
Early approaches in this domain were confined to sub-tasks, such as predicting paper acceptance from metadata and abstract features or regressing review scores from paper content.
“Early approaches in this domain were confined to sub-tasks, such as predicting paper acceptance from metadata and abstract features or regressing review scores from paper content (Kang et al., 2018;Checco et al., 2021).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al., 2018
Foundational efforts deconstructed the peer review process into analytical, predictive, and narrowly generative components.
“These foundational efforts deconstructed the peer review process into analytical (Teufel et al., 1999), predictive (Kang et al., 2018), and narrowly generative components (Dycke et al., 2023b).”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Kang et al., 2018
The datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.
“These datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Kang et al. 2018
Most static review datasets could not guarantee consistency between their paper content and the papers under review, nor did they include rebuttals, making it difficult to support research about reviewer-author discussions.
“However, although the above works have made progress in scholarly peer review, most of these datasets could not guarantee the consistency between their paper content and those under review, nor did they include rebuttals, making it difficult to support the research about the reviewers-authors discussions.”
From Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions (2025) · cites this paper as Kang et al. [2018]
The Introduction (p. 1647) says access to peer-review data had been limited, and §2.1 describes how PeerRead collected papers and reviews. Neither passage assesses whether most other datasets match the papers actually reviewed or include rebuttals. Searches for rebuttals, paper-version mismatches, and prior datasets did not find support for those claims or the asserted difficulty of studying reviewer–author discussions. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Maximum error reduction is reported as both 21% and 22%
MinorThe abstract gives a maximum error reduction of 21%, whereas the results section gives 22% for the same paper-acceptance task.
“up to 21% error reduction compared to the majority baseline”
“Our best model outperforms the majority classifier in all cases, with up to 22% error reduction.”
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1× · see citing works
“We use three neural architectures: convolutional neural networks (CNN, Zhang et al., 2015), recurrent neural networks (LSTM, Hochreiter and Schmidhuber, 1997), and deep averaging networks (DAN, Iyyer et al., 2015).”
MethodScikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · cited 1× · see citing works
“We use python’s sklearn’s implementation of all models (Pedregosa et al., 2011).”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 2× · see citing works
“We use GloVe 840B embeddings (Pennington et al., 2014) as input word representations, without tuning, and keep the 35K most frequent words and replace the rest with an UNK vector.”
MethodBinary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · cited 1× · see citing works
“We use three neural architectures: convolutional neural networks (CNN, Zhang et al., 2015), recurrent neural networks (LSTM, Hochreiter and Schmidhuber, 1997), and deep averaging networks (DAN, Iyyer et al., 2015).”
MethodDeep Unordered Composition Rivals Syntactic Methods for Text Classification
Iyyer, M., Manjunatha, V., Boyd-Graber, J. L., & Daumé, H. (2015) · doi:10.3115/v1/p15-1162 · Cited by 729 · cited 1× · see citing works
“We use three neural architectures: convolutional neural networks (CNN, Zhang et al., 2015), recurrent neural networks (LSTM, Hochreiter and Schmidhuber, 1997), and deep averaging networks (DAN, Iyyer et al., 2015).”
MethodOpen peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261 · cited 2× · see citing works
“Editors of the British Journal of Psychiatry found di ff erences in courtesy between signed and unsigned reviews (Walsh et al., 2000).”
BackgroundBalachandran, V. (2013) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2013.6606642 · Cited by 179 · see citing works
Content-Based Citation Recommendation
Bhagavatula, C., Feldman, S., Power, R., & Ammar, W. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1022 · Cited by 174 · cited 2× · see citing works
“Tools like Citeomatic 29 (Bhagavatula et al., 2018) are especially useful in finding relevant articles not cited in the manuscript.”
BackgroundSelf-citationBalachandran, V. (2013) · doi:10.5555/2486788.2486915 · Cited by 145 · see citing works
RevRank: A Fully Unsupervised Algorithm for Selecting the Most Helpful Book Reviews
Tsur, O., & Rappoport, A. (2009) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v3i1.13945 · Cited by 136 · cited 1× · see citing works
“Also related to our acceptance predicting task are (Tsur and Rappoport, 2009) and Ashok et al.”
NIH peer review percentile scores are poorly predictive of grant productivity
Fang, F. C., Bowen, A. D., & Casadevall, A. (2016) · eLife · doi:10.7554/elife.13323 · Cited by 109 · cited 1× · see citing works
“Fang et al. (2016) presented similar observations for NIH grant application reviews and their productivity.”
BackgroundSuccess with Style: Using Writing Style to Predict the Success of Novels
Ashok, V. G., Feng, S., & Choi, Y. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1181 · Cited by 84 · cited 1× · see citing works
“Also related to our acceptance predicting task are (Tsur and Rappoport, 2009) and Ashok et al. (2013), both of which focuses on predicting book reviews.”
BackgroundAlternatives to Peer Review: Novel Approaches for Research Evaluation
Birukou, A., Wakeling, J. R., Bartolini, C., Casati, F., Marchese, M., Мирыленка, К., Osman, N., Ragone, A., Carles Sierra, A., & Вассеф, А. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2011.00056 · Cited by 70 · cited 1× · see citing works
“Ragone et al. (2011) and Birukou et al. (2011) analyzed ten CS conferences and found low correlation between review scores and the impact of papers in terms of future number of citations.”
BackgroundDouble-blind reviewing at EvoLang 11 reveals gender bias
Roberts, S. G., & Verhoef, T. (2016) · Journal of Language Evolution · doi:10.1093/jole/lzw009 · Cited by 60 · cited 1× · see citing works
“double blind review experiments, which pointed to single-blind reviews leading to increased biases towards male authors (Roberts and Verhoef, 2016) and famous institutions (Tomkins et al.”
BackgroundThe arbitrariness of reviews, and advice for school administrators
Langford, J., & Guzdial, M. (2015) · Communications of the ACM · doi:10.1145/2732417 · Cited by 49 · cited 2× · see citing works
“…2014 conference assigned 10% of conference submissions to two di ff erent sets of reviewers to measure the consistency of the peer reviewing process, and observed that the two committees disagreed on the accept / reject decision for more than a quarter of the papers (Langford and Guzdial, 2015).”
BackgroundPopularity of arXiv.org within Computer SciencePreprint
Sutton, C. A., & Gong, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.05225 · Cited by 22 · cited 3× · see citing works
“With the evergrowing rates of articles being submitted to toptier conferences in Computer Science and pre-print repositories (Sutton and Gong, 2017), there is a need to expedite the peer review process.”
BackgroundMethodInfluentialSingle versus Double Blind Reviewing at WSDM 2017Preprint
Tomkins, A., Zhang, M., & Heavlin, W. (2017) · arXiv.org · doi:10.48550/arxiv.1702.00502 · Cited by 18 · cited 2× · see citing works
“double blind review experiments, which pointed to single-blind reviews leading to increased biases towards male authors (Roberts and Verhoef, 2016) and famous institutions (Tomkins et al., 2017).”
BackgroundPreserving the Quality of Scientific Research: Peer Review of Research Articles
Silva, P. U. K. D., & Vance, C. K. (2017) · Fascinating life sciences · doi:10.1007/978-3-319-50627-2_6 · Cited by 15 · see citing works
Bonaccorsi, A. J., Ferrara, A., & Malgarini, M. (2018) · Springer eBooks · doi:10.1007/978-3-319-68554-0_11 · Cited by 4 · see citing works
Effectiveness of Anonymization in Double-Blind Review
Goues, C. L., Brun, Y., Apel, S., et al. (2017) · arXiv · doi:10.48550/arxiv.1709.01609
Flagged first, then by citations
1 of the 22 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Sarah Greaves, Joanna Scott, Maxine Clarke, Linda Miller, Timo Hannay, Annette Thomas, and Philip Campbell. 2006. Nature's trial of open peer review. Nature 444(971):10-1038.
- Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. 2011. A quantitative analy- sis of peer review. In Proc. of ISSI.
- Drummond Rennie. 2016. Make peer review scien- tific: thirty years on from the first congress on peer review, drummond rennie reflects on the improve- ments brought about by research into the process- and calls for more. Nature 535(7610):31-34.
- Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running aver- age of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26-31.
Cited by
Sorted by citations
Rogers, A., Boyd-Graber, J., & Okazaki, N. (2023) · doi:10.18653/v1/2023.acl-short · Cited by 208 · see citing works
A context-aware citation recommendation model with BERT and graph convolutional networks
Jeong, C., Jang, S., Park, E., & Choi, S. (2020) · Scientometrics · doi:10.1007/s11192-020-03561-y · Cited by 186 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212 · see citing works
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · see citing works
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96 · see citing works
Peer review analyze: A novel benchmark resource for computational analysis of peer reviews
Ghosal, T., Kumar, S., Bharti, P. K., & Ekbal, A. (2022) · PLOS ONE · doi:10.1371/journal.pone.0259238 · Cited by 66 · see citing works
“For definitions of each aspect-label, please follow the appendix section in [49].”
InfluentialDeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1106 · Cited by 62 · see citing works
“Training is done with ICLR 2017+ICLR 2018 papers/reviews, † → Cross-Domain: Training on ICLR and testing upon the entire data of ACL/CoNLL, ‡Test Set is kept the same as (Kang et al., 2018), RMSE→Root Mean Squared Error, ∗ →65.”
BackgroundMethodResultInfluentialAPE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
Cheng, L., Bing, L., Yu, Q., Lu, W., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.569 · Cited by 58 · see citing works
Intrinsic Evaluation of Summarization Datasets
Bommasani, R., & Cardie, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.649 · Cited by 54 · see citing works
“PeerRead (Kang et al., 2018) is a dataset of paper drafts from top-tier computer science venues as well as arXiv.”
MethodDoes My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1129 · Cited by 48 · see citing works
“In line with Kang et al. (2018), we first assess the impact of individual aspect scores on the overall score by measuring their Pearson correlation, illustrated in Fig.”
BackgroundMethodResultInfluentialA Neural Citation Count Prediction Model based on Peer Review Text
Li, S., Zhao, W. X., Yin, E. J., & Wen, J.-R. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1497 · Cited by 45 · see citing works
“The data of NIPS 2013-2016 and ICLR 2017 have been shared by Kang et al. (2018).”
BackgroundMethodThelwall, M. A. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0011 · Cited by 54 · see citing works
The main trends for multi-tier supply chain in Industry 4.0 based on Natural Language Processing
Zhou, R., Awasthi, A., & Cardinal, J. S.-L. (2020) · Computers in Industry · doi:10.1016/j.compind.2020.103369 · Cited by 41 · see citing works
Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
Chen, M., Chu, Z., & Gimpel, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1060 · Cited by 27 · see citing works
“We construct the dataset from PeerRead (Kang et al., 2018), which consists of scientific papers from a variety of fields.”
MethodReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis
Wang, Q., Zeng, Q., Huang, L., Knight, K., Ji, H., & Rajani, N. F. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.44 · Cited by 38 · see citing works
“We follow the data split of PeerRead (Kang et al., 2018) 6 .”
InfluentialXue, Z., He, G., Liu, J., Jiang, Z., Zhao, S. X., & Lu, W. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103216 · Cited by 33 · see citing works
(2020) · doi:10.18653/v1/2020.findings-emnlp.112 · Cited by 32 · see citing works
“…revealed only four conference papers on peer review from a meta-research perspective: a paper-reviewer matching tool (Anjum et al., 2019), a corpus of reviews (Kang et al., 2018), and two experimental studies using NLP to explain the observed reviews (Caragea et al., 2019; Gao et al., 2019).”
MethodThelwall, M. A., & Yaghi, A. (2025) · Scientometrics · doi:10.1007/s11192-025-05287-1 · Cited by 36 · see citing works
“For this study, metadata and reviewer scores for ICLR submissions were extracted from a repository curated for research purposes (Kang et al., 2018).”
Predicting article quality scores with machine learning: The U.K. Research Excellence Framework
Thelwall, M. A., Kousha, K., Wilson, P., Makita, M., Abdoli, M., Stuart, E., Levitt, J. M., Knoth, P., & Cancellieri, M. (2023) · Quantitative Science Studies · doi:10.1162/qss_a_00258 · Cited by 29 · see citing works
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization
Jung, T., Kang, D., Mentch, L., & Hovy, E. H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1327 · Cited by 27 · see citing works
Self-citationMReD: A Meta-Review Dataset for Structure-Controllable Text Generation
Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., & Si, L. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.198 · Cited by 23 · see citing works
“Researchers recently explore the peer review domain data for a few tasks, such as PeerRead (Kang et al., 2018) for paper decision predictions, AM-PERE (Hua et al., 2019) for proposition classification in reviews, and RR (Cheng et al., 2020) for paired-argument extraction from review-rebuttal pairs.”
BackgroundUncertainty Estimation and Reduction of Pre-trained Models for Text Regression
Wang, Y., Beck, D., Baldwin, T., & Verspoor, K. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00483 · Cited by 23 · see citing works
“For SA, we use: (1) a large-scale product review dataset, Yelp (Sabnis, 2018); and (2) a small paper review rating dataset, PeerRead (Kang et al., 2018), augmented with 399 Spanish paper reviews (Keith et al., 2017) machine-translated into English.”
MethodInfluentialLLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Narayanan Venkit, P., Zhang, N., Srinath, M., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.292 · Cited by 23 · see citing works
Defining quality in peer review reports: a scoping review
Sizo, A., Lino, A. D. P., Rocha, Á., & Reis, L. P. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02435-0 · Cited by 26 · see citing works
Wu, W., Zhang, C., & Zhao, Y. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70005 · Cited by 26 · see citing works
“Kang et al. (2018) were pioneers in introducing the first large-scale peer-review dataset, PeerRead, comprising full-text papers from ACL, ICLR, and NIPS, along with the corresponding editorial decisions of acceptance or rejection.”
Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction
Wenniger, G. M. D. B., van Dongen, T., Aedmaa, E., Kruitbosch, H., Valentijn, E. A., & Schomaker, L. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.18 · Cited by 30 · see citing works
“arXiv sub-domain dataset Majority class prediction Benchmark (Kang et al., 2018) BiLSTM (Shen et al.”
BackgroundMethodResultInfluentialCSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization
Lin, H., Ma, L., Zhu, J., Xiang, L., Zhou, Y., Zhang, J., & Zong, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.365 · Cited by 32 · see citing works
“Recently, there exist various types of document summarization datasets such as news reports (Nallapati et al., 2016; Narayan et al., 2018), wiki passages (Liu et al., 2018), and scientific papers (Kang et al., 2018).”
BackgroundA Joint Model for Multimodal Document Quality Assessment
Shen, A., Salehi, B., Baldwin, T., & Qi, J. (2019) · 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl.2019.00024 · Cited by 24 · see citing works
“…FA indicator, which is a bronze star icon in the top right corner of the webpage) are removed from the screenshot. arXiv dataset The arXiv dataset (Kang et al. 2018) consists of three subsets of academic articles under the arXiv repository of Computer Science (cs), from the three subject areas…”
InfluentialA Sentiment Augmented Deep Architecture to Predict Peer Review Outcomes
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2019.00096 · Cited by 19 · see citing works
“Table 2: Results on Aspect Score Prediction and Accept/Reject Classification Tasks, Task-1 (score prediction) training is done with only ICLR 2017 papers/reviews while Task 2 (classification) training is done with ICLR 2017+ICLR 2018 papers/reviews, † → Cross-Domain: Training on ICLR and Testing upon entire data of AC…”
MethodResultInfluentialTransferring Knowledge from Discourse to Arguments: A Case Study with Scientific Abstracts
Accuosto, P., & Saggion, H. (2019) · Workshop on Argument Mining (ArgMining) · doi:10.18653/v1/w19-4505 · Cited by 21 · see citing works
“We also intend to expand our acceptance prediction experiments using the PeerRead dataset (Kang et al., 2018), 21 which has a greater coverage than the NIPS and ICLR subsets that we used in our experiments.”
MethodBreaking Down Walls of Text: How Can NLP Benefit Consumer Privacy?
Ravichander, A., Black, A. W., Norton, T. B., Wilson, S., & Sadeh, N. (2021) · doi:10.18653/v1/2021.acl-long.319 · Cited by 22 · see citing works
“NLP has made inroads in digesting large amounts of text in domains such as scientific publications and news (Jain et al., 2020; Cachola et al., 2020; Kang et al., 2018; Rush et al., 2015; See et al., 2017), with several”
BackgroundMethodNLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.277 · Cited by 19 · see citing works
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
Deng, Z., Peng, H., Xia, C., Li, J., He, L., & Yu, P. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.555 · Cited by 19 · see citing works
“Kang et al. (2018) collected a dataset of peer reviews from several conferences and predicted paper acceptance decision by using paper draft.”
BackgroundThe influence of opening up peer review on the citations of journal articles
Ni, J., Zhao, Z.-Y., Shao, Y., Liu, S., Li, W., Zhuang, Y., Qu, J.-M., Cao, Y., Lian, N., & Li, J. (2021) · Scientometrics · doi:10.1007/s11192-021-04182-9 · Cited by 19 · see citing works
“The linguistic characteristics of the comments and responses are sometimes linked to ratings, citations, etc. Kang et al. (2018) built Natural Language Processing (NLP) models based on their database to predict whether the paper was going to be accepted.”
BackgroundInfluentialM4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection
Wang, Y., Mansurov, J., Ivanov, P., Su, J., Shelmanov, A., Tsvigun, A., Whitehouse, C., Afzal, O. M., Mahmoud, T., Sasaki, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.14902 · Cited by 19 · see citing works
“PeerRead Reviews We sampled 586 academic papers in fields of NLP and machine learning top-tier conferences released by ReerRead corpus (Kang et al., 2018), with metadata including title, abstract, and multiple human reviews for each paper.”
BackgroundMethodInfluentialKumar, S., Ghosal, T., Bharti, P. K., & Ekbal, A. (2021) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl52503.2021.00081 · Cited by 19 · see citing works
Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review
Kuznetsov, I., Buchmann, J., Eichler, M., & Gurevych, I. (2022) · Computational Linguistics · doi:10.1162/coli_a_00455 · Cited by 17 · see citing works
Bharti, P. K., Ghosal, T., Agarwal, M., & Ekbal, A. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00375-0 · Cited by 19 · see citing works
“Mention could be made of the PeerRead [10] dataset.”
InfluentialWang, Z.-Y., Zhang, H., Chen, H.-H., Feng, Y.-H., & Ding, J.-H. (2024) · Journal of King Saud University: Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102119 · Cited by 53 · see citing works
“These features can not only be obtained in advance, but also play a crucial role in identifying the quality of the paper (Kang et al., 2018; Checco et al., 2021; Kousha and Thelwall, 2023).”
InfluentialDoes My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.1903.11367 · Cited by 16 · see citing works
“In line with Kang et al. (2018), we first assess the impact of individual aspect scores on the overall score by measuring their Pearson correlation, illustrated in Fig.”
BackgroundMethodResultInfluentialUncertainty-Aware Machine Support for Paper Reviewing on the Interspeech 2019 Submission Corpus
Stappen, L., Rizos, G., Hasan, M., Hain, T., & Schuller, B. W. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2862 · Cited by 15 · see citing works
“The scores are predictable to a degree, in accordance to [6], as the RMSE and MAE values are between 0.”
BackgroundMethodInfluentialMachine learning approach to predicting the acceptance of academic papers
Skorikov, M., & Momen, S. (2020) · IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) · doi:10.1109/iaict50021.2020.9172011 · Cited by 15 · see citing works
“The data is taken from the PeerRead [11] dataset, which consists of 14.7K machine learning paper drafts and corresponding accept and reject decisions from top-tier venues.”
BackgroundMethodLarge language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15 · see citing works
“For example, PeerRead contains reviews from machine learning conferences (Kang et al., 2018); NLPeer focuses on computational linguistics venues (Dycke et al., 2023); and Openreviewer offers a larger and more recent collection of human reviews from leading machine learning conferences (Szumega et…”
Detecting LLM-generated peer reviews
Rao, V. S., Kumar, A., Lakkaraju, H., & Shah, N. B. (2025) · PLOS One · doi:10.1371/journal.pone.0331871 · Cited by 14 · see citing works
Hug, S. E. (2024) · Scientometrics · doi:10.1007/s11192-023-04915-y · Cited by 13 · see citing works
“According to the estimates in Table 2, all evaluation criteria are positively related to the overall judgment, suggesting that referees use a broad range of criteria to evaluate an application and integrate the criteria in a linear, additive way.”
PEERAssist: Leveraging on Paper-Review Interactions to Predict Peer Review Decisions
Bharti, P. K., Ranjan, S., Ghosal, T., Agrawal, M., & Ekbal, A. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-91669-5_33 · Cited by 16 · see citing works
ER-AE: Differentially Private Text Generation for Authorship Anonymization
Bo, H., Ding, S. H. H., Fung, B. C. M., & Iqbal, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.314 · Cited by 27 · see citing works
“• Academic Review Dataset: All the public reviews from NeurIPS (2013-2018) and ICLR (2017) based on the original data and the web crawler provided by [21].”
BackgroundCREPE: Open-Domain Question Answering with False Presuppositions
Yu, X., Min, S., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.583 · Cited by 18 · see citing works
Wu, W., Xi, H., & Zhang, C. (2024) · Scientometrics · doi:10.1007/s11192-024-05070-8 · Cited by 12 · see citing works
“Among them, Kang et al. [15] introduced the first publicly available dataset, PeerRead, designed for scientific peer review research.”
A General Approach to Multimodal Document Quality Assessment
Shen, A., Salehi, B., Qi, J., et al. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11647 · Cited by 12 · see citing works
Qiao, F., Xu, L., & Han, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02934-0_7 · Cited by 13 · see citing works
Multi-task Peer-Review Score Prediction
Li, J., Sato, A., Shimura, K., & Fukumoto, F. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.14 · Cited by 15 · see citing works
“We use the ICLR and ACL datasets in the Peer-Read Dataset (Kang et al., 2018) because they provide the scores of the peer-review aspects.”
BackgroundMethodInfluentialMOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · see citing works
“MOPRD is composed of paper metadata, manuscripts of the initial submission and following revisions, review comments, meta-re-views, author’s rebuttal letters, and editorial decisions of papers across various disciplines.”
BackgroundInfluentialA Hybrid Citation Recommendation Model With SciBERT and GraphSAGE
Dinh, T. N., Pham, P., Giang, N. L., Nguyên, N. T., & Vo, B. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3490774 · Cited by 13 · see citing works
Pendyala, V. S., Kamdar, K., & Mulchandani, K. (2025) · Electronics · doi:10.3390/electronics14020256 · Cited by 14 · see citing works
Back to the Future – Temporal Adaptation of Text Representations
Bjerva, J., Kouw, W. M., & Augenstein, I. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6240 · Cited by 16 · see citing works
A Deep Multimodal Investigation To Determine the Appropriateness of Scholarly Submissions
Ghosal, T., Raj, A., Ekbal, A., Saha, S., & Bhattacharyya, P. (2019) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2019.00039 · Cited by 11 · see citing works
PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews
Bharti, P. K., Navlakha, M., Agarwal, M., & Ekbal, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09662-3 · Cited by 12 · see citing works
“Kang et al. (2018) introduced PeerRead, which is the first large-scale peer review dataset.”
Li, Y.-X., Kou, G., Peng, Y., & Morente-Molinera, J. A. (2024) · Applied intelligence (Boston) · doi:10.1007/s10489-024-05765-8 · Cited by 8 · see citing works
KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-training
Yuan, W., & Liu, P. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21418 · Cited by 10 · see citing works
“…analysis (Tomkins, Zhang, and Heavlin 2017; Stelmakh, Shah, and Singh 2019), aspect-based sentiment analysis (Chakraborty, Goyal, and Mukher-jee 2020), decision classification (Kang et al. 2018; Qiao, Xu, and Han 2018), automatic review generation (Wang et al. 2020; Yuan, Liu, and Neubig 2021).”
BackgroundMethodHeterogeneous Supervised Topic Models
Sridhar, D., Daumé, H., & Blei, D. M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00487 · Cited by 9 · see citing works
“This dataset comes from PeerRead, (Kang et al., 2018), a corpus of computer-science papers.”
MethodarXivEdits: Understanding the Human Revision Process in Scientific Writing
Jiang, C., Xu, W., & Stevens, S. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.641 · Cited by 9 · see citing works
Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond
Dycke, N., Kuznetsov, I., & Gurevych, I. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.23 · Cited by 10 · see citing works
“The ACL-2017 portion of the PeerRead corpus (Kang et al., 2018) includes reviews of submissions to the 55th Annual Meeting of ACL, for which both authors and reviewers agreed to share the review reports.”
BackgroundInfluentialA Neural Local Coherence Analysis Model for Clarity Text Scoring
Muangkammuen, P., Xu, S., Fukumoto, F., Saikaew, K. R., & Li, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.194 · Cited by 10 · see citing works
“The PeerRead dataset (Kang et al., 2018) was selected as a benchmark.”
MethodInfluentialOpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
Idahl, M., & Ahmadi, Z. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations) · doi:10.18653/v1/2025.naacl-demo.44 · Cited by 10 · see citing works
Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs
Kang, D., & Hovy, E. H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1589 · Cited by 8 · see citing works
Self-citationOn Positivity Bias in Negative Reviews
Aithal, M., & Tan, C. (2021) · doi:10.18653/v1/2021.acl-short.39 · Cited by 10 · see citing works
“• PeerRead (Kang et al., 2018).”
Modeling Interference for Individual Treatment Effect Estimation from Networked Observational Data
Huang, Q., Ma, J., Li, J.-D., Guo, R.-C., Sun, H., & Chang, Y. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3628449 · Cited by 21 · see citing works
“PeerRead [27] is a dataset of computer scientific peer reviews for papers.”
BackgroundWhat Makes a Scientific Paper be Accepted for Publication?
Fytas, P., Rizos, G., & Specia, L. (2021) · doi:10.18653/v1/2021.cinlp-1.4 · Cited by 11 · see citing works
“PeerRead suffers from various data quality issues, which we have resolved.”
BackgroundMethodResultInfluentialAcceptance Decision Prediction in Peer-Review Through Sentiment Analysis
Ribeiro, A. C., Sizo, A., Cardoso, H. L., & Reis, L. P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_60 · Cited by 10 · see citing works
What have we learned from OpenReview?
Wang, G., Peng, Q., Zhang, Y., & Zhang, M. (2022) · World Wide Web · doi:10.1007/s11280-022-01109-z · Cited by 9 · see citing works
“[13] predict the acceptance of a paper based on textual features and the score of each aspect in a review based on the paper and review contents.”
BackgroundCharacterizing Peer Review Comments of Academic Articles in Multiple Rounds
Han, R., Zhou, H., Zhong, J., & Zhang, C.-Z. (2022) · ASIS&T Annual Meeting · doi:10.1002/pra2.607 · Cited by 8 · see citing works
“Kang et al. (2018) defined two novel tasks based on PeerRead: (i) predicting the acceptance of a paper based on textual features and (ii) predicting the score of each aspect in a review based on the paper and review contents.”
MethodSentBS: Sentence-level Beam Search for Controllable Summarization
Shen, C., Cheng, L., Bing, L., You, Y., & Si, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.699 · Cited by 10 · see citing works
Exploring the Potential of GPT-2 for Generating Fake Reviews of Research Papers
Bartoli, A., & Medvet, E. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200717 · Cited by 8 · see citing works
Cognitive Representation Learning of Self-Media Online Article Quality
Wang, Y., Huang, S., Li, G., Deng, Q., Liao, D., Si, P., Yang, Y., & Xu, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413747 · Cited by 10 · see citing works
“[10] predicted whether an academic paper is accepted or rejected based on features such as the title length and whether specific words appear in the abstract.”
BackgroundLeveraging peer-review aspects for extractive and abstractive summarization of scientific articles
Majadly, M., & Last, M. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00665-z · Cited by 8 · see citing works
“While some datasets for sci-entific peer reviews exist, such as PeerRead [42], which is well-known and aids researchers in studying this topic, they are not directly applicable to the task of aspect extraction.”
Artificial Intelligence vs. Human: Decoding Text Authenticity with Transformers
Gîfu, D., & Silviu-Vasile, C. (2025) · Future Internet · doi:10.3390/fi17010038 · Cited by 10 · see citing works
“…(https://paperswithcode.com/datasets, accessed on 22 July 2024): This dataset contains human-written text from sources such as Wikipedia, WikiHow [45], Reddit (ELI5), arXiv, and PeerRead [46] for English, as well as news articles for Urdu, RuATD [47] for Russian, and Indonesian news articles.”
Kumar, S., Arora, H., Ghosal, T., & Ekbal, A. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3530937 · Cited by 8 · see citing works
“The first public dataset of scientific peer reviews was presented in [7] available for research purposes, providing an opportunity to study this important artifact.”
BackgroundER-AE: Differentially Private Text Generation for Authorship AnonymizationPreprint
Bo, H., Ding, S. H. H., Fung, B. C. M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08736 · Cited by 7 · see citing works
“• Academic Review Dataset: All the public reviews from NeurIPS (2013-2018) and ICLR (2017) based on the original data and the web crawler provided by [21].”
BackgroundBharti, P. K., Agarwal, M., & Ekbal, A. (2024) · Scientometrics · doi:10.1007/s11192-024-04938-z · Cited by 6 · see citing works
“Kang et al. (2018) introduced PeerRead, the first large-scale peer review dataset, which contains full-text papers from ACL, ICLR, and NIPS along with their editorial accept/reject decisions and review comments.”
Basuki, S., & Tsuchiya, M. (2022) · IEEE Access · doi:10.1109/access.2022.3225871 · Cited by 6 · see citing works
“In this study area, the dataset published by [22] is the most cited work, which [22] compiled numerous peer-review datasets comprising ICLR, arXiv, Association for Computational Linguistics (ACL), and Conference on Computational Natural Language Learning (CoNLL).”
MethodReAct: A Review Comment Dataset for Actionability (and more)
Choudhary, G., Modani, N., & Maurya, N. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-91560-5_24 · Cited by 6 · see citing works
“[6] also present PeerRead dataset consolidating reviews from a lot of conferences.”
BackgroundKumar, A., Ghosal, T., Bhattacharjee, S., & Ekbal, A. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00359-0 · Cited by 6 · see citing works
“The PeerRead [10] dataset is the first publicly available peer-review resource that encouraged Natural Language Processing (NLP)/ Machine Learning (ML) research on peer review problems.”
COMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Singh, S., Singh, M., & Goyal, P. (2021) · 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl52503.2021.00068 · Cited by 7 · see citing works
“Several works focus on curation of peer review data [3], and further analysis such as sentiment analysis of reviews and decision prediction [3, 4].”
BackgroundSentiment as an Ordinal Latent Variable
Stoehr, N., Cotterell, R., & Schein, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.8 · Cited by 6 · see citing works
“These are PeerRead (Kang et al., 2018), specifically the splits ACL and ICLR, IMDB (Maas et al.”
MethodArgument Mining for Understanding Peer ReviewsPreprint
Hua, X., Nikolov, M., Badugu, N., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.10104 · Cited by 5 · see citing works
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
Self-citationLin, T.-L., Chen, W.-C., Hsiao, T.-F., Liu, H.-I., Yeh, Y.-H., Chan, Y.-K., Lien, W.-S., Kuo, P.-Y., Yu, P. S., & Shuai, H.-H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.259 · Cited by 6
An Analysis of Tasks and Datasets in Peer Reviewing
Staudinger, M., Kusa, W., Piroi, F., & Hanbury, A. (2024) · Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024) · doi:10.18653/v1/2024.sdp-1.24 · Cited by 6 · see citing works
When Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews
Kumar, S., Ghosal, T., & Ekbal, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1038 · Cited by 7 · see citing works
Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction
Muangkammuen, P., Fukumoto, F., Li, J., & Suzuki, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.164 · Cited by 5 · see citing works
“1 https://github.com/panitan-m/gamma_trans Much of the previous work on PASP is based on supervised learning (Kang et al., 2018; Li et al., 2020).”
BackgroundMethodInfluentialGORC: A large contextual citation graph of academic papersPreprint
Lo, K., Wang, L. L., Neumann, M. E., Kinney, R., & Weld, D. S. (2019) · arXiv (Cornell University) · Cited by 5 · see citing works
Large language models for peer review in biotechnology
Li, W., Zhao, R., Sun, P.-T., Lee, J., Lu, O., Yuan, S., Sureshkumar, P., Chen, Y., Xiao, Z., & Tang, Y. J. (2026) · New Biotechnology · doi:10.1016/j.nbt.2026.03.007 · Cited by 4 · see citing works
Heterogeneous hypergraph learning for literature retrieval based on citation intents
Shi, K., Liu, K., & He, X. (2024) · Scientometrics · doi:10.1007/s11192-024-05066-4 · Cited by 4 · see citing works
“The Peerread CIG has fewer links between the nodes and is a sparse graph, while the arXiv CIG is dense.”
InfluentialQuantitative Discourse Cohesion Analysis of Scientific Scholarly Texts Using Multilayer Networks
Bhatnagar, V., Duari, S., & Gupta, S. (2022) · IEEE Access · doi:10.1109/access.2022.3198952 · Cited by 4 · see citing works
“The first collection comprises 172 articles accepted in ICLR 2017 and is a part of the PeerRead collection (Kang et al., 2018).”
BackgroundInfluentialOverview of PragTag-2023: Low-Resource Multi-Domain Pragmatic Tagging of Peer Reviews
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Workshop on Argument Mining · doi:10.18653/v1/2023.argmining-1.21 · Cited by 10 · see citing works
“Yet, to date, NLP for peer review has focused on data from machine learning conferences (Kang et al., 2018; Hua et al., 2019; Cheng et al., 2020; Kennard et al., 2022), and the applications outside of this domain remain under-investigated.”
BackgroundInfluentialClaimBrush: A Novel Framework for Automated Patent Claim Refinement Based on Large Language Models
Kawano, S., Nonaka, H., & Yoshino, K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825172 · Cited by 4 · see citing works
Multi-task Neural Shared Structure Search: A Study Based on Text Mining
Li, J., & Fukumoto, F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-73197-7_13 · Cited by 4 · see citing works
- Authors
- Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R.
- Affiliations
- Carnegie Mellon University (US)
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
- Published
- 2018
- Pages
- 1647-1661
- OpenAlex
- W2964211782
- Semantic Scholar
- 53489041a08ef1a6cd19b4c95ce092a148283b6b
- Cited by
- 171
- References
- 22
- Citations by year
- 2026: 14 · 2025: 15 · 2024: 15 · 2023: 33 · 2022: 26 · 2021: 28 · 2020: 19 · 2019: 20 · 2018: 1
- FWCI
- 6.77
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- peer review, scholarly publishing, natural language processing applications, research evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review