CoChat
Sign Up
Cited by 35Publishing and Scholarly CommunicationAcademic Writing and PublishingTopic Modeling

NLPeer: A Unified Resource for the Computational Study of Peer Review

Open access
Dycke, Nils, Kuznetsov, Ilia, Gurevych, Iryna·2023·Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
DOI 10.18653/v1/2023.acl-long.277
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer review constitutes a core component of scholarly publishing; yet it demands substantial expertise and training, and is susceptible to errors and biases.Various applications of NLP for peer reviewing assistance aim to support reviewers in this complex process, but the lack of clearly licensed datasets and multi-domain corpora prevent the systematic study of NLP for peer review.To remedy this, we introduce NLPEER -the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.In addition to the new datasets of paper drafts, cameraready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task.Our work paves the path towards systematic, multi-faceted, evidencebased study of peer review in NLP and beyond.The data 1 and code 2 are publicly available.

AI snapshot

Written from the paper’s full text.

TL;DR

NLPEER is presented as the first ethically sourced multi-domain corpus of more than 5k papers and 11k review reports from five venues, with a unified data representation and task implementations. It enables systematic computational study of peer review across domains, reviewing systems, and time, including a novel guided skimming task.

Key findings

  • NLPEER contains 5,672 papers and 11,515 reviews, with more than 4.2M review tokens; it includes new ARR-22, COLING-20, and F1000-22 datasets plus ACL-17 and CONLL-16, spanning 2012–2022 and NLP/CL plus multi-domain venues.
  • Review score prediction: neural models substantially outperform the mean baseline for ARR-22 and F1000-22, but are on par or worse for ACL-17, CONLL-16, and COLING-20; best results include ARR-22 MRSE .37 ± .02 / F1 .46 ± .04, F1000-22 MRSE .21 ± .01 / F1 .41 ± .04, and ACL-17 MRSE .76 ± .06 / F1 .11 ± .04. Dataset size appears to have the strongest impact.
  • Pragmatic labeling: in-dataset macro-F1 is 0.72 ± 0.01 for ARR-22 and 0.87 ± 0.01 for F1000RD; cross-dataset transfer reaches 0.50 ± 0.01 (ARR-22→F1000RD) and 0.54 ± 0.01 (F1000RD→ARR-22), above majority baselines of 0.11 and 0.152.
  • Guided skimming: SciBERT achieves MRR 0.41 ± 0.05 on ACL-17 and 0.34 ± 0.03 on ARR-22, outperforming random by 0.23 and 0.18; at k = 3 roughly 50% of relevant passages are retrieved, while about 21% of retrieved paragraphs are actually linked by reviewers.

Methods

  • Data collection used an ethically and legally compliant workflow: donation-based collection for ARR-22 and COLING-20, an F1000Research API crawl for F1000-22, and openly licensed PeerRead portions for ACL-17 and CONLL-16. Papers and reviews were unified into intertextual graphs (ITG), parsed with GROBID, and supplemented with metadata, revisions, and licenses.
  • Experiments fine-tuned RoBERTa, BioBERT, and SciBERT on three tasks: review score prediction (regression using review text plus paper abstract, truncated to 512 tokens, evaluated by MRSE and F1), pragmatic labeling (sentence classification into Strength, Weakness, Request, Neutral), and guided skimming (paragraph ranking from explicit review-paper links).
  • Data were split 70/10/20, experiments repeated ten times with different random seeds, and median ± standard deviation reported; all experiments ran on a single NVIDIA A100 GPU, totaling roughly seven days of computing time.

Limitations

  • Stated: NLPEER covers only a fraction of peer reviewing across science, is English-only, and under-represents arts and humanities; donation-based ARR-22 and COLING-20 data carry participation bias. Task implementations are meant to exemplify feasibility, not claim model superiority, and were not extensively hyperparameter-tuned.
  • Stated: sentence-level pragmatic labels derived from ARR structured forms may contain noise; guided skimming ignores implicit links, and explicit links are mostly line numbers and quotes, limiting recall. The paper also warns that high intrinsic task performance does not necessarily translate into real-world utility and discusses risks such as bias, review automation, and “lazy reading.”
  • Evident from the text: ACL-17, CONLL-16, and COLING-20 have the lowest sample counts and score prediction performs on par or worse than the mean baseline there; paragraph length is a hard-to-beat spurious feature for guided skimming, and cross-dataset pragmatic labeling shows confusion between weakness and request classes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.acl-long.277

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 26 cited sources checked · 6 supported · 1 flagged · accuracy not yet measured

  • Interpretationcitation

    For donation-based datasets (all except F1000-22), the positive skew is likely due to participation bias.

    “For the donation-based datasets (all except F1000-22), this is likely due to participation bias (Dycke et al., 2022);”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dycke et al., 2022 source

      Section 6.2 (page not supplied) finds that reviewers’ donation decisions did not introduce substantial polarity bias and authors’ donation decisions introduced no substantial further positive bias. It identifies conditioning on paper acceptance, rather than donation participation, as the major source of positive score bias in its dataset. The paper does not assess the other datasets named in the claim. (LLM, conf=0.87)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Findingnone given

    NLPEER is the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.

    “we introduce NLPEER -the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues.”
    Quote in the paperSubstantial claim
  • Findingown data

    LLM perform substantially above random despite discarding all context information of a paragraph.

    “Overall, the LLM perform substantially above random despite discarding all context information of a paragraph (see appendix C for a detailed analysis).”
    Quote in the paperSubstantial claim
  • Findingnone given

    NLPEER is complemented with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task.

    “We complement our resource with implementations and analysis of three reviewing assistance tasks, including a novel guided skimming task.”
    Quote in the paperSubstantial claim
  • Findingown data

    Neural models outperform the majority baseline in-dataset by a large margin for pragmatic labeling.

    “Expectedly, neural models outperform the majority baseline in-dataset by a large margin.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 66
  • May combine two claims. 60

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Review score prediction has been first introduced in Kang et al. 2018 and further explored in Ghosal et al. 2019.

    “Review score prediction has been first introduced in (Kang et al., 2018) and further explored in (Ghosal et al., 2019).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Kang et al., 2018 source
    • cites Ghosal et al., 2019 source

  • May combine two claims.

    Study design and conductcitation

    Parts of the PeerRead data have been created with explicit consent for publication and processing by both reviewers and authors, and are included in NLPEER.

    “Parts of the PeerRead data (Kang et al., 2018) have been created with explicit consent for publication and processing by both reviewers and authors, and we include them into our resource.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kang et al., 2018 source

  • May combine two claims.

    Premisecitation

    The computational study of peer review lacks a solid data foundation because reviewing data is rarely public and existing sources are not licensed, preventing reuse and redistribution.

    “The computational study of peer review lacks a (1) solid data foundation: reviewing data is rarely public and comes with legal and ethical challenges; existing sources of peer reviewing data and the derivative datasets are not licensed, which legally prevents reuse and redistribution (Dycke et al., 2022).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dycke et al., 2022 source

      The Introduction says a solid data foundation is lacking, and §2.3 says publicly available peer-review data is scarce and published datasets lack clear licenses. However, §2.3 says the missing licenses leave reuse conditions unspecified; it does not say they legally prevent reuse and redistribution. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The guided skimming for peer review task is novel and builds upon recent work in cross-document modeling for peer review by Kuznetsov et al. (2022) and other related works.

    “The guided skimming for peer review task is novel and builds upon the recent work in cross-document modeling for peer review by Kuznetsov et al. (2022) and other related works (Qin and Zhang, 2022;Ghosal et al., 2022a).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Kuznetsov et al., 2022 source
    • cites Qin and Zhang, 2022 source
    • cites Ghosal et al., 2022a source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Pragmatic labeling has been previously explored in Hua et al. 2019, Kuznetsov et al. 2022, and Kennard et al. 2022.

    “Pragmatic labeling has been previously explored in (Hua et al., 2019;Kuznetsov et al., 2022;Kennard et al., 2022) and is usually cast as a discourse labeling task on free-form peer review text.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Hua et al., 2019 source

      The available copy contains bibliographic metadata identifying Hua et al. (2019) as a paper on argument mining for peer reviews, but no passage describing its labeling task. Searches for pragmatic labeling and discourse labeling of peer-review text returned only metadata, so the attributed methodological claim cannot be checked against this copy. (LLM, conf=0.69)

      Experimental — accuracy not yet measured
    • cites Kuznetsov et al., 2022 source
    • cites Kennard et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Experiments were based on well-established large language models from the general and scientific domain - RoBERTa, BioBERT and SciBERT - with the recommended default hyperparameter settings.

    “Hence, we base our experiments on well-established large language models (LLM) from the general and scientific domain -RoBERTa (Liu et al., 2019), BioBERT (Lee et al., 2020) and SciBERT ( Beltagy et al., 2019) -with the recommended default hyperparameter settings.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Liu et al., 2019 source
    • cites Lee et al., 2020 source
    • cites Beltagy et al., 2019 source

  • May combine two claims.

    Study design and conductcitation

    Only reviews and paper drafts for which both authors and reviewers agreed to donation were included, avoiding any possibility of leaking confidential research ideas.

    “we only include reviews and paper drafts for which both authors and reviewers agreed to donation; hereby avoiding any possibility of leaking confidential research ideas from the papers.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Dycke et al. (2022) source

      Section 3Y: Yes by the Authors says authors may donate a draft alone or a draft with reviews whose reviewers agreed to donation. Section 4.1 Design says public release of reviews also requires author permission and is limited to accepted publications. These safeguards address confidentiality, but the paper does not claim they eliminate every possibility of leaking research ideas; it also does not require reviewer agreement for a draft donated alone. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Training a RoBERTa model on 30% of the ARR-22 training set resulted in a median 0.44 MRSE and 0.24 F1-macro score on the test set, on par with the mean baseline, and comparable to the similarly-sized ACL-17.

    “To investigate a potential connection, we have trained a RoBERTa model on 30% of the ARR-22 training set, resulting in a median 0.44 MRSE and 0.24 F1-macro score on the test set -on par with the mean baseline, and comparable to the similarly-sized ACL-17.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both recall and precision exceed the random baseline by a large margin and for all considered k in guided skimming.

    “Both recall and precision exceed the random baseline by a large margin and for all considered k.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Every paper in NLPEER is associated with at least one review and one revision, making it suitable for the study of cross-document relationships.

    “Every paper in NLPEER is associated with at least one review and one revision, making it suitable for the study of cross-document relationships.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    NLPEER establishes a unified data representation and augments previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.

    “In addition to the new datasets of paper drafts, cameraready versions and peer reviews from the NLP community, we establish a unified data representation and augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Neural models substantially outperform the mean score baseline for ARR-22 and F1000-22, yet for ACL-17, CONLL-16 and COLING-20 they perform on par or worse.

    “Neural models substantially outperform the mean score baseline for ARR-22 and F1000-22, yet for ACL-17, CONLL-16 and COLING-20 they perform onpar or worse.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    This work contributes a descriptive analysis of the resulting data.

    “and complemented by a (3) descriptive analysis of the resulting data”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At around k = 3 roughly 50% of the relevant passages are retrieved, while at the same rank around 21% of the retrieved paragraphs are actually linked by the reviewers.

    “At around k = 3 roughly 50% of the relevant passages are retrieved; while at the same rank around 21% of the retrieved paragraphs are actually linked by the reviewers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Table 1 reports textual statistics of NLPEER, comprising more than 4M peer review tokens in total, with more than 400K review tokens in the NLP/CL domain.

    “Table 1 reports the textual statistics of NLPEER, comprising more than 4M peer review tokens in total, with more than 400K review tokens in the NLP/CL domain.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Scoring scales and semantics differ across datasets: the NLP and CL conferences employ fine-grained scales (5- and 9-point), while F1000 Research uses a very coarse scale (3-point).

    “However, scoring scales and semantics differ across datasets: the NLP and CL conferences employ fine-grained scales (5-and 9-point), while F1000 Research uses a very coarse scale (3-point).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Reviews from the NLP/CL communities (ARR-22, ACL-17, COLING-20) are most similar (0.37-0.53), while F1000-22 is most similar to ARR-22 with a notably lower score than the within-community comparison (∆ ≈ 0.25).

    “As Figure 4 demonstrates, reviews from the NLP/CL communities (ARR-22, ACL-17, COLING-20) are most similar (0.37-0.53), while F1000-22 is most similar to ARR-22 with a notably lower score than the within-community comparison (∆ ≈ 0.25).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The resource is diverse, and there is high variability in review and paper composition among the datasets.

    “The resource is diverse, and we observe high variability in review and paper composition among the datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    NLPEER includes datasets with linguistically diverse review reports while maintaining a domain-independent base vocabulary characteristic for the genre of peer reviews.

    “NLPEER includes datasets with linguistically diverse review reports while maintaining a domain-independent base vocabulary characteristic for the genre of peer reviews.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Papers in F1000-22 come from a wide range of domains and cover a wide range of article types from case reports to position papers, reflected in the low number of sentences per paper (158) with high variance (±90.3) compared to the other, NLP-based datasets with roughly 200 sentences per paper and lower variance.

    “For example, papers in F1000-22 come from a wide range of domains and cover a wide range of article types from case reports to position papers, reflected in the low number of sentences per paper (158) with high variance(±90.3)compared to the other, NLP-based datasets with roughly 200 sentences per paper and lower variance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Review lengths exhibit smaller variance across the datasets.

    “Yet, we note that review lengths exhibit smaller variance across the datasets.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Scores near the borderline to acceptance are well-represented, with a skew towards the positive end of the review score scale otherwise.

    “We observe that scores near the arguably most interesting region around the borderline to acceptance are well-represented, with a skew towards the positive end of the review score scale otherwise.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    SciBERT achieves an MRR of 0.41 ± 0.05 on ACL-17 and 0.34 ± 0.03 on ARR-22, outperforming the random baseline by 0.23 and 0.18, respectively.

    “SciBERT achieves an MRR of 0.41 ± 0.05 on ACL-17 and 0.34 ± 0.03 on ARR-22, outperforming the random baseline by 0.23 and 0.18, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    For RoBERTa on F1000-RD, the highest model confusion is for the neutral class; strengths, weaknesses, and requests are most commonly confused with a neutral statement.

    “We see the highest model confusion for the neutral class; strengths, weaknesses and requests are all most commonly confused with a neutral statement”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Sentence-level pragmatic labels derived from structure-based ARR forms might contain noise because ARR forms group text on section level.

    “sentence-level pragmatic labels derived from structure-based ARR forms might contain noise since ARR forms group text on section level”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For ARR-22, 64% of the relevant paragraphs are extracted from line-mentions in the reviews; for ACL-17, 57%.

    “For ARR-22 64% of the relevant paragraphs are extracted from line-mentions in the reviews, while for ACL-17 these amount to 57%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The model performs substantially above the random baseline, but the margin towards the length baseline is small, especially for the AUROC.

    “While the model performs substantially above the random baseline, the margin towards the length baseline is small, especially for the AUROC.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Other datasets include at least the paper draft and if the paper was eventually published a camera-ready version.

    “Other datasets include at least the paper draft and if the paper was eventually published a camera-ready version of the paper.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    F1000-22 is collected from F1000Research, a publishing platform with an open post-publication reviewing workflow.

    “F1000- 22 is collected from F1000Research -a publishing platform with an open post-publication reviewing workflow.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For RoBERTa on ARR-22, the neutral class is most often confused with the other classes, similar to F1000-RD.

    “Generally, we seem a similar pattern as for F1000-RD: the neutral class is most often confused with the others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Explicit links are mostly based on line numbers and quotes, limiting the recall.

    “explicit links are mostly based on line numbers and quotes, limiting the recall.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    During a pilot study, relatively unstable training was observed, rendering extensive random hyperparameter search infeasible.

    “During a pilot study we observed relatively unstable training, which rendered extensive random hyperparameter search infeasible for the scope of this work.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Guided skimming does not make use of implicit links.

    “guided skimming does not make use of implicit links”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The ranking produced by SciBERT leads to a higher precision in the top ranks, but seemingly performs on-par with the length baseline for the overall ranking performance.

    “Overall, the ranking produced by SciBERT leads to a higher precision in the top ranks, but seemingly performs on-par with the length baseline for the overall ranking performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For ARR-22, around half of the paragraphs originate from the level of sections rather than sub-sections; for ACL-17, roughly two thirds lie on section level.

    “For ARR-22 around half of the paragraphs originate from the level of sections rather than sub-sections, while for ACL-17 roughly two thirds lie on section level.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the model trained on ARR-22 and tested on F1000-RD, weaknesses are frequently confused with requests and vice versa, unlike the in-domain trained model on F1000-RD.

    “We see that unlike the in-domain trained model on F1000-RD weaknesses are frequently confused with requests and vice versa.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    For F1000-22, each paper may have multiple revisions.

    “For F1000-22, each paper may have multiple revisions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The initial analysis identified 25 potentially problematic cases, of which 10 were deemed relevant upon a second check.

    “The initial analysis has identified 25 potentially problematic cases, of which 10 were deemed relevant upon a second check.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Some reviews included in the ARR-22 dataset are revisions of previous reviews or may contain references to those.

    “Some reviews included in the dataset are therefore revisions of previous reviews or may contain references to those.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some reviewers mention strengths in the summary section of a review.

    “some reviewers do mention strengths in the summary section of a review.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For ARR-22, the phenomenon of introduction having the most links is less pronounced, while the results section is slightly more often referenced than in ACL-17.

    “For ARR-22 this phenomenon is less pronounced, while the results section is slightly more often referenced than in ACL-17.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Errors in GROBID and ITG parsing are inevitable.

    “errors are inevitable and we provide papers both in raw and parsed form.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Manual checks show that explicit links are identified with high precision but comparatively low recall.

    “Manual checks support this observation, in particular we see that explicit links seem to be identified with high precision, but comparatively low recall.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Only explicit links that can be mapped to one specific paragraph are included, leaving only quotes and line references in practice.

    “we only include those that can be mapped to one specific paragraph leaving only quotes and line references in practice.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The best model according to MRSE tends to have one of the lowest KL-divergence scores, but this is not consistently true (e.g., RoBERTa on F1000-22).

    “While the best model according to MRSE tends to have one of the lowest scores, this is not consistently true (e.g.RoBERTa on F1000-22).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    NLPEER unites openly licensed datasets from different research communities, reviewing systems and time periods, including three previously unreleased text collections: ARR-22, COLING-20 and F1000-22.

    “NLPEER unites openly licensed datasets from different research communities, reviewing systems and time periods, including three previously unreleased text collections: ARR-22, COLING-20 and F1000-22.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    Several neutral, factual sentences exist in the strength and weaknesses sections.

    “Likewise several neutral, factual sentences exist in the strength and weaknesses sections.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The collected ARR-22 dataset contains reviews for the final draft of a paper, but none of the previous revisions.

    “The collected dataset contains reviews for the final draft of a paper, but none of the previous revisions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Implicit links in reviews are not considered because they are not readily available at scale.

    “We also highlight that implicit links in reviews, i.e. discussions of paper aspects that do not use explicit identifiers for paper regions, are not considered, as they are not readily available at scale.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The guided skimming task assists during the effortful and time-intensive, yet crucial step of reading the paper under review.

    “our newly introduced guided skimming task assists during the effortful and time-intensive, yet crucial step of reading the paper under review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Similar to the in-domain trained model, the neutral class is the hardest to predict for the model.

    “Similar to the in-domain trained model the neutral class is the hardest to predict for the model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    Sentences of ARR-22 are extracted from review forms which do not enforce full consistency with the respective section, implying certain levels of noise in labels.

    “the sentences of ARR-22 are just extracted from the review forms which do not enforce full consistency with the respective section implying certain levels of noise in labels.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    34 articles with invalid file formatting and roughly 2000 articles that lacked reviews for the first version were discarded from F1000-22.

    “We discarded 34 articles with invalid file formatting and roughly 2000 articles that lacked reviews for the first version, as these indicate stale submissions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The paper and review data are distributed under CC-BY license, which we preserve.

    “The paper and review data are distributed under CC-BY license, which we preserve.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    F1000Research uses unstructured peer reviewing forms coupled with a single 3-point overall score (approve, reject, approve-with-reservations).

    “F1000Research uses unstructured peer reviewing forms coupled with a single 3-point overall score(approve, reject, approve-with-reservations).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The reviewing process at F1000Research is fully open, with reviewer and author identities known throughout the process.

    “The reviewing process at F1000Research is fully open, with reviewer and author identities known throughout the process, contributing to the diversity of NLPEER.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Unlike other datasets in NLPEER, F1000-22 covers a wide range of research communities from scientific policy research to medicine and public health.

    “Unlike other datasets in NLPEER, F1000-22 covers a wide range of research communities from scientific policy research to medicine and public health.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingown data

    The paper vocabulary overlap follows a similar trend (see Appendix Figure 7).

    “The paper vocabulary overlap follows a similar trend, see Appendix (Figure7).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The ARR-22 data collection process allowed reviewers to explicitly request attribution, and this information is included in the dataset.

    “The ARR-22 data collection process allowed reviewers to explicitly request attribution, and this information is included in the dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    A multi-domain perspective on peer review is missing, and the transferability of findings between different communities and reviewing workflows remains unclear.

    “A (2) multi-domain perspective on peer review is thus missing, and the transferability of findings between different communities and reviewing workflows remains unclear.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This makes the investigation of cross-domain NLP for reviewing assistance a promising research avenue.

    “This makes the investigation of cross-domain NLP for reviewing assistance a promising research avenue.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Peer review reports can serve as a valuable resource for distant or direct supervision of general-purpose machine learning models.

    “Hereby, they can serve as a valuable resource for distant or direct supervision of generalpurpose machine learning models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Peer review reports are expert-written texts that typically reflect deep understanding of the underlying paper.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Datasets are split by paper while ensuring a similar distribution of passages per paper across splits.

    “Datasets are split by paper while ensuring a similar distribution of passages per paper across splits (see app.B).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data

    Guided skimming is a non-trivial task, but the above-random performance of the given LLM baseline shows promise for further research in this direction.

    “While guided skimming is a non-trivial task, the above-random performance of the given LLM baseline shows promise for further research in this direction, which could also include an in-depth study considering the context of paragraphs -e.g.their position in the logical structure of the paper.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The gap to the in-distribution supervised model remains, constituting a promising target for follow-up work.

    “Nevertheless, the gap to the in-distribution supervised model remains, constituting a promising target for the follow-up work, that would need to disentangle the effects of domain, task and reviewing system shift on pragmatic labeling performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Over-focusing on few selected conferences in machine learning limits the utility of NLP approaches to peer review for other areas of science and reviewing systems, and leaves cross-domain applicability open.

    “In addition, over-focusing on few selected conferences in machine learning limits the utility of NLP approaches to peer review for other areas of science and reviewing systems, and leaves the question of cross-domain applicability open.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    F1000Research uses open reviewer and author identities throughout the reviewing process, and this information is preserved in F1000-22.

    “F1000Research uses open reviewer and author identities throughout the reviewing process; this information is preserved in F1000-22.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All reviews of the same paper belong to the same data split.

    “The same paper can receive multiple reviews; we ensure that all reviews of the same paper belong to the same data split.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Paper summaries in review reports can be used as a basis for a challenging document summarization task.

    “paper summaries in review reports can be used as a basis for a challenging document summarization task”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The F1000RD corpus lacks data from NLP and CS communities, and does not contain blind reviewing data.

    “While the recent F1000RD corpus (Kuznetsov et al., 2022) addresses some of these issues, it lacks data from NLP and CS communities, and does not contain blind reviewing data, although single-and double-blind review are arguably standard in most research fields (Johnson et al., 2018).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Kuznetsov et al., 2022 source
    • cites Johnson et al., 2018

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Study design and conductnone given

    All datasets included in NLPEER are distributed under an open Creative Commons license and were collected based on explicit consent, or an open license attached to the source data.

    “All datasets included in NLPEER are distributed under an open Creative Commons license and were collected based on explicit consent, or an open license attached to the source data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    NLPEER provides the NLP community a head-start in ethically sound, cross-domain and cross-temporal study of peer review.

    “The purpose of NLPEER is to provide the NLP community a head-start in ethically sound, cross-domain and cross-temporal study of peer review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Aspect score disagreements between reviews may serve as a weak supervision signal for aligning arguments for and against the paper across reviews.

    “aspect score disagreements between reviews may serve as a weak supervision signal for aligning arguments for and against the paper across reviews.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The fundamental difference between peer reviewing systems in NLPEER makes it an interesting and challenging target for computational study of reviewing scores across reviewing systems and research communities.

    “This fundamental difference between the peer reviewing systems in NLPEER makes it an interesting and challenging target for the computational study of reviewing scores across reviewing systems and research communities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationown data

    Results pave the path towards exploiting cross-document links between peer reviews and research papers for language model benchmarking.

    “pave the path towards exploiting cross-document links between peer reviews and research papers for language model benchmarking.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The authors of camera-ready publications included in NLPEER are attributed.

    “Finally, the authors of camera-ready publications included in NLPEER are attributed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The paper revisions make NLPEER well-suited for the study of collaborative text work.

    “The paper revisions make NLPEER well-suited for the study of collaborative text work.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    For representative sampling, the distribution of reviews per paper is similar across splits.

    “For representative sampling, we ensure that the distribution of reviews per paper is similar across splits (see Appendix B).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Experiments were repeated ten times with different random seeds, and the median and standard deviation of each performance metric across the runs were reported.

    “We repeat the experiments ten-times with different random seeds and report the median and standard deviation of each performance metric across the runs.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The three tasks - review score prediction, pragmatic labeling and guided skimming - are anchored in realistic assistance scenarios and targeted towards helping junior reviewers to improve their review.

    “To demonstrate its versatility, we define three tasks -review score prediction, pragmatic labeling and guided skimming -that are anchored in realistic assistance scenarios and targeted towards helping junior reviewers to improve their review.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    NLPEER is a unique resource for the study of computational assistance during peer review authoring.

    “NLPEER is a unique resource for the study of computational assistance during peer review authoring.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The resulting labels contain certain levels of noise and reflect only a subset of the regions of the paper that are actually discussed in the reviews.

    “Consequently, the resulting labels do contain certain levels of noise and reflect only a subset of the regions of the paper that are actually discussed in the reviews.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The results should not be interpreted as a claim towards superiority of a particular model, approach or reviewing system.

    “our results should not be interpreted as a claim towards superiority of a particular model, approach or reviewing system.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Especially in the top ranks for ACL-17, the LLM seem to pick up information beyond mere paragraph length.

    “However, especially in the top ranks for ACL-17 (see 12) the LLM seem to pick up information beyond mere paragraph length.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Fine-tuning was repeated 10 times for each model and task with different random seeds.

    “We repeat fine-tuning including model selection in total 10 times for each model and task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The authors omitted extensive fine-tuning of the LLMs and focused on a base set of hyperparameters close to recommended ones.

    “we omit extensive fine-tuning of the LLMs and instead focus on a base set of hyperparameters close to the recommended ones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    There appear to exist natural areas of focus for reviewers as approximated by explicit links, but paragraphs of many different section types are covered in both datasets.

    “Overall, there appear to exist natural areas of focus for reviewers as approximated by explicit links, but paragraphs of many different section types are in fact covered in both datasets.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    It is likely that strengths and weaknesses are already reported in the summary section correlating to the neutral label, encouraging this confusion.

    “As the review forms are not strictly enforced, it is likely that strengths and weaknesses are already reported in the summary section correlating to the neutral label encouraging this confusion.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Review forms are not strictly enforced.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    For guided skimming, the splits have a similar distribution of relevant paragraphs per paper.

    “For guided skimming, we make sure that the splits have a similar distribution of relevant paragraphs per paper.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    Models were fine-tuned with small learning rates ([1, 3 × 10 -5 ]) and a linear warm-up schedule, and training was allowed for up to 20 epochs, following existing recommendations.

    “To account for small dataset sizes, we fine-tune each model with small learning rates ([1, 3 × 10 -5 ]) and a linear warm-up schedule, and allow training for up to 20 epochs, following existing recommendations (Mosbach et al., 2021).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mosbach et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Interpretationreasoning

    Many sentences in the weakness section of ARR are actually requests.

    “This supports the hypothesis that many sentences in the weakness section of ARR are actually requests, as the model trained on F1000-RD is based off of human gold annotations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Paragraph length seems a relevant feature that might be exploited as a spurious decision criterion by the models.

    “The length of the paragraphs seems a relevant feature that might be exploited as a spu-”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Kuznetsov et al. (2022) report an F1 score of 0.77 (precision 0.81) for anchor identification and 0.64 (precision 0.66) for matching explicit anchors to regions in the paper compared to human annotations.

    “The authors report an F1 score of 0.77 (with a precision of 0.81) for anchor identification and 0.64 (with a precision of 0.66) for their approach at matching explicit anchors to regions in the paper compared to human annotations.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kuznetsov et al. (2022) source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Exact matches were included without manual verification, while for the others the authors checked if the papers plausibly align.

    “Exact matches were included without manual verification, for the others the authors checked if the papers plausibly align.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    For COLING-20 and CONLL-16, the models perform on par with the mean baseline, suggesting they converged to predicting scores very close to the mean.

    “For COLING-20 and CONLL-16 the models perform on-par with mean baseline suggesting that they converged to predicting scores very close to the mean.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    A targeted study is necessary to substantiate the assessment that opportunities outweigh risks.

    “yet highlight that a targeted study is necessary to substantiate this assessment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    For review score prediction, the distribution of reviews per paper is similar across splits.

    “For review score prediction, we require that the distribution of reviews per paper is similar across splits.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The authors did not perform extensive hyperparameter search and tuning of the models.

    “Since we did not perform extensive hyperparameter search and tuning of the models”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Manual spot checks suggest a high quality of GROBID and ITG parsing.

    “Although manual spot checks suggest a high quality of GROBID and ITG parsing”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    High intrinsic task performance does not necessarily translate into the extrinsic utility of NLP support in real-world reviewing environments.

    “high intrinsic task performance does not necessarily translate into the extrinsic utility of NLP support in real-world reviewing environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The layout information extracted heuristically from the PDFs allows a reliable mapping of paragraphs to line ranges.

    “we rely on the layout information extracted heuristically from the PDFs allowing a reliable mapping of paragraphs to line ranges.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Reviews have a wider cross-domain shared vocabulary, while papers employ a more specialized register.

    “This supports the observation that reviews have a wider crossdomain shared vocabulary, while papers apparently employ a more specialized register.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The authors followed recommendations (Mosbach et al., 2021) for fine-tuning LLM on comparatively small datasets.

    “We follow recommendations (Mosbach et al., 2021) for fine-tuning LLM on comparatively small datasets.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mosbach et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    More data needs to be collected to enable fully representative NLP-based study of peer review.

    “more data needs to be collected to enable fully representative NLP-based study of peer review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The opportunities provided by the introduced review assistance tasks outweigh the potential risks in general.

    “We assess that the opportunities provided by the introduced review assistance tasks outweigh the potential risks in general”
    Quote in the paperNot a substantial claim
105 of 105

  • The paper relies on this without support.

    Peer review reports are expert-written texts that typically reflect deep understanding of the underlying paper.

    “Peer review reports are expert-written texts that typically reflect deep understanding of the underlying paper.”

  • The authors state this as an assumption.

    Each paper is associated with at least one revision.

    “Our data model assumes that each paper is associated with at least one revision”

  • The paper relies on this without support.

    Potential biases result from the donation-based collection for ARR-22 and COLING-20.

    “Specifically, potential biases resulting from the donation-based collection for ARR-22 and COLING-20 should be taken into account.”

  • The authors state this as an assumption.

    A model that learns reviewers' rating behavior well should predict scores from a similar range of scores as humans do.

    “A model that learns reviewers' rating behavior well, should also predict scores from a similar range of scores as humans do.”

  • The paper relies on this without support.

    Review forms are not strictly enforced.

    “As the review forms are not strictly enforced, it is likely that strengths and weaknesses are already reported in the summary section correlating to the neutral label encouraging this confusion.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.

    “These datasets cover an early peer-review corpus, the ACL Rolling Review process, and public ICLR review records, respectively, and differ in field structure, text granularity, and label sources.”

    From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Dycke, Kuznetsov, and Gurevych 2023

    Section 3.1 identifies ACL-17 as part of the earlier PeerRead data, and Appendix A.3 identifies ARR-22 as data from ACL Rolling Review. The Corpus Creation Details describe differences in review structure and scores, while Appendix B.2 describes different sources of pragmatic labels. The paper’s dataset list does not include public ICLR review records, so it does not support that part of the sentence. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Existing peer-review datasets usually contain heterogeneous paper-review-decision records with differing field structures, text granularity, and label sources.

    “First, existing peer-review datasets usually contain heterogeneous paper-review-decision records.Their field structures, text granularity, and label sources differ”

    From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Dycke, Kuznetsov, and Gurevych 2023

    The Introduction (§1) says existing peer-review datasets use task-specific data models and formats, and §3.2 describes unifying papers and reviews with different structures. The pragmatic-labeling discussion (§5.3 and Appendix B.2) also identifies different label sources. The paper does not establish that existing datasets usually contain paper-review-decision records; that part of the claim goes beyond the cited evidence. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Reviewer comments are distributed across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation rather than concentrated in a single type of problem.

    “Empirical analyses show that reviewer comments are distributed across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation rather than concentrated in a single type of problem (Dycke et al., 2023;Kang et al., 2018;Stephen, 2022).”

    From Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers (2026) · cites this paper as Dycke et al., 2023

    Table 5 in “Relevant Paragraph Distribution within Papers” reports review sentences across summary, strength, weakness, and request labels, supporting the general point that comments occur in multiple categories. It does not measure the claimed distribution across methodology, theory, writing, originality, impact, clarity, soundness, and recommendation. Section 5.3 also notes that feedback can be imbalanced, including a focus on weaknesses. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • The use of peer review data for research purposes raises important ethical considerations around reviewer consent, data licensing, and responsible use.

    “The use of peer review data for research purposes raises important ethical considerations around reviewer consent, data licensing, and responsible use (Dycke et al., 2023).”

    From Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (2024) · cites this paper as Dycke et al., 2023

  • PeerRead, MOPRD, and NLPeer are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.

    “PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”

    From LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024) · cites this paper as Dycke et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • ACL-17 SciBERT MRR differs between the main text and appendix

    Major

    The main results report an ACL-17 MRR of 0.41 ± 0.05 for SciBERT, while Table 12 reports 0.35 ± 0.03 for the same model and dataset.

  • Average of dataset review lengths is misreported

    Minor

    Table 1 says totals marked with an asterisk are averaged. The five reported means for sentences per review average to 33.6, not 32.2.

    Reported 32.2 · recomputed 33.6

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Dycke, N., Kuznetsov, I., & Gurevych, I.
Affiliations
Technische Universität Darmstadt (DE)
ORCID
Venue
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2023
Pages
5049-5073
DOI
10.18653/v1/2023.acl-long.277
OpenAlex
W4385569919
Cited by
19
References
26
Citations by year
2026: 2 · 2025: 6 · 2024: 1 · 2023: 10
FWCI
69.77
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Annual Meeting of the Association for Computational Linguistics (ACL) (publishedVersion, cc-by)
  • TUbilio (Technical University of Darmstadt) (submittedVersion)
  • Technischen Universität Darmstadt (submittedVersion, cc-by)
  • Technischen Universität Darmstadt (cc-by)
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
peer review, review reports, scholarly publishing, data licensing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review