What Can Natural Language Processing Do for Peer Review?

May combine two claims.

Most modern semantic similarity measures are based on dense representations, and thus lack interpretability.

“Finally, most modern semantic similarity measures are based on dense representations, and thus lack interpretability (Kim et al., 2023).”

• cites Kim et al., 2023 sourcePartly supported

  The Related Work section says that similarity scores based on language-model representations are commonly used in document matching. The paper also discusses the need for information beyond those scores to help people make decisions, but it does not establish that most modern semantic similarity measures use dense representations or that dense representations inherently lack interpretability. Page numbers are unavailable in our copy. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

NLP-assisted keyword generation and automatic track suggestion could make the submission process and the subsequent review more efficient.

“NLP-assisted keyword generation (Caragea et al., 2014; Çano and Bojar, 2020) and automatic track suggestion could make the submission process and the subsequent review more efficient.”

• cites Caragea et al., 2014 sourceSupported
• cites Çano and Bojar, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Other AI/ML conferences have addressed incentives for strategic behavior by adding randomness to reviewer-paper matching scores to make the formation of collusion rings more difficult.

“Other AI/ML conferences have further addressed incentives for strategic behavior by adding randomness to reviewer-paper matching scores to make the formation of collusion rings more difficult (Jecmen et al., 2020).”

• cites Jecmen et al., 2020 sourcePartly supported

  The Introduction describes randomized reviewer assignments designed to reduce manipulation, including collusion-related risks. But Related Literature says the paper takes similarity-score computation as given, and the Introduction describes tests on past conference datasets—not conferences adopting randomness in matching scores. No page numbers are available in our copy. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Even with stellar scientific content, a poorly composed paper is more likely to be rejected and later resubmitted, incurring additional work for both authors and future reviewers.

“Even with stellar scientific content, a poorly composed paper is more likely to be rejected and later resubmitted (Church, 2020), incurring additional work for both authors and future reviewers.”

• cites Church, 2020 sourcePartly supported

  In §2.1.1 (page unavailable), Church describes a paper with better content receiving a lower score than one with better English; §2.1.2 says nonnative English is sometimes used as an excuse for rejection. Section 2.4 discusses repeated submissions, but attributes them to unpredictable reviewing, not poor composition. The paper therefore supports the risk of rejection despite strong content, but not the claimed composition-to-resubmission sequence. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

AAAI-2021 applied a system of this kind and found at least one unreported conflict for 78.8% of submissions-over 96,000 unreported conflicts in total.

“AAAI-2021 applied a system of this kind and found at least one unreported conflict for 78.8% of submissions-over 96,000 unreported conflicts in total (Leyton-Brown et al., 2024, Section 5.1.4).”

• cites Leyton-Brown et al., 2024 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Some checklists ask authors to indicate the sections of their papers in which certain issues are discussed, such as reproducibility efforts, compensation to human participants, and adherence to data licenses.

“In a similar vein, some checklists 6 ask authors to indicate the sections of their papers in which certain issues are discussed, e.g., reproducibility efforts, compensation to human participants, adherence to data licenses (Dodge et al., 2019; Beygelzimer et al., 2021a; Rogers et al., 2021).”

• cites Dodge et al., 2019 sourcePartly supported

  Section 5 presents an experimental-results checklist intended to improve reproducibility reporting. But the checklist in Text Box 1 asks for details such as data splits and hyperparameter trials; our copy does not say authors must identify the sections of their papers where issues are discussed, or include participant compensation and data-license adherence. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Beygelzimer et al., 2021aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Rogers et al., 2021 sourceSupported

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Automated checks for whether the checklist answers correspond to the indicated paper sections could help authors avoid accidental errors, speed up checklist completion, and facilitate reviewing.

“Automated checks for whether the checklist answers correspond to the indicated paper sections could help authors avoid accidental errors, speed up checklist completion and facilitate reviewing (Liu and Shah, 2023).”

• cites Liu and Shah, 2023 sourcePartly supported

  Section 4.3 describes using GPT-4 to verify author checklists against specified paper sections, and Section 4.2 reports that it flagged many author answers that differed from the hand-labeled ground truth. That supports the possibility of catching checklist errors and aiding review, but the passages do not say that these checks speed up authors’ checklist completion. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

May combine two claims.

More advanced processing of revisions has been approached for arXiv pre-prints and ACL publications, and in the context of peer review.

“more advanced processing of revisions has been approached for arXiv pre-prints (Jiang et al., 2022; Du et al., 2022) and ACL publications (Mita et al., 2022), and in the context of peer review (Kuznetsov et al., 2022; D'Arcy et al., 2023).”

• cites Jiang et al., 2022 sourceSupported
• cites Du et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Mita et al., 2022 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Zeng et al. (2023) cast the task as scientific opinion summarization.

“Zeng et al. (2023) cast the task as scientific opinion summarization.”

• cites Zeng et al. (2023) sourceSupported

May combine two claims.

A traditional way to enforce composition of peer review reports is to deploy structured and semi-structured reviewing forms-which has been shown to improve inter-rater agreement of decision recommendations.

“A traditional way to enforce composition of peer review reports is to deploy structured and semi-structured reviewing forms-which has been shown to improve inter-rater agreement of decision recommendations (Malički and Mehmani, 2024).”

• cites Malički and Mehmani, 2024 sourcePartly supported

  The Inter-rater agreement section reports higher agreement on final recommendations under structured review than in the same journals before implementation (41% versus 31%). But the Discussion says the pilot was not randomized and lacked a direct comparison with traditional reports for the same manuscripts. It therefore supports an association, not the claim that reviewing forms have been shown to improve agreement. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

May combine two claims.

Recent work has shown that generative AI models could be used to identify errors such as mathematical mistakes, conceptual fallacies, or problems in experimental design, but might overemphasize certain aspects such as implications of the research and adding more experiments, compared to humans.

“Recent work has shown that generative AI models could be used to identify errors such as mathematical mistakes, conceptual fallacies, or problems in experimental design (Nuijten and Polanin, 2020; Liu and Shah, 2023), but might overemphasize certain aspects such as implications of the research and adding more experiments, compared to humans (Liang et al., 2023).”

• cites Nuijten and Polanin, 2020 sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Liu and Shah, 2023 sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites Liang et al., 2023 sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Conducting reviewer surveys is commonplace at AI conferences.

“Conducting reviewer surveys is commonplace at AI conferences (Névéol et al., 2017; Rastogi et al., 2022; Shah, 2023).”

• cites Névéol et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Rastogi et al., 2022 sourceSupported
• cites Shah, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Several algorithms have been developed to address this issue but have not performed well in practice.

“Several algorithms have been developed to address this issue (Roos et al., 2012; Ge et al., 2013) but have not performed well in practice (see Shah, 2022, Section 5 for more details on this miscalibration).”

• cites Roos et al., 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Ge et al., 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Shah, 2022 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Zeng et al. (2023) propose a checklist-based approach to prompt an LLM to write summaries that fulfill meta-review requirements of self-consistency, faithfulness, and active engagement.

“They propose a checklist-based approach to prompt an LLM to write summaries that fulfill meta-review requirements of self-consistency, faithfulness, and active engagement.”

• cites Zeng et al. (2023) sourceSupported

May combine two claims.

As part of assessing the paper, reviewers often use search engines to find related literature-and might encounter public versions of the manuscript under review, which can inadvertently expose the authors' identities and affect the evaluation.

“As part of assessing the paper, reviewers often use search engines to find related literature-and might encounter public versions of the manuscript under review, which can inadvertently expose the authors' identities and affect the evaluation (Rastogi et al., 2022).”

• cites Rastogi et al., 2022 sourceSupported

May combine two claims.

Human evaluation in NLP for peer review requires effort and expertise, and might be challenging to reproduce.

“A further drawback of human evaluations in NLP for peer review is that they require effort and expertise, and might be challenging to reproduce (Belz et al., 2023).”

• cites Belz et al., 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

GDPR is more permissive to research use than commercial use.

“While GDPR is more permissive to research use than commercial use (Kamocki et al., 2018)”

• cites Kamocki et al., 2018 sourcePartly supported

  In the section headed “5.1(f) of the GDPR” (page unavailable), the paper says research can arguably be a legitimate interest that permits processing, subject to data subjects’ rights. It does not say GDPR is more permissive for research use than for commercial use; the comparative part of the claim is unsupported. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Estimates repeatedly arrive at millions of human-hours spent reviewing yearly.

“Estimates repeatedly arrive at millions of human-hours spent reviewing yearly (AJE, 2013; GSPR, 2018)”

• cites AJE, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites GSPR, 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Low-quality reviewing includes generic, hastily written reviews and the use of fast-reject heuristics like 'not state of the art', 'too niche', or 'writing too bad' instead of a thorough evaluation.

“This includes both generic, hastily written reviews, and the use of fast-reject heuristics like 'not state of the art', 'too niche', or 'writing too bad' instead of a thorough evaluation”

May combine two claims.

Reviewer abuse includes downscoring others' work to reduce potential competition, establishing collusion rings, and other behaviors.

“This includes downscoring others' work to reduce potential competition, establishing collusion rings in which a group of reviewers conspire to pick each others' work for review and give it a favourable evaluation (Section 4.3.2), and other behaviors (Shah, 2022, Section 4).”

• cites Shah, 2022 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Li et al. (2023) found that the model struggles with recognizing and resolving conflicts.

“Li et al. (2023) explore a summarization model that incorporates inter-document relationships between reviews, author responses, and discussions, finding that the model struggles with recognizing and resolving conflicts.”

• cites Li et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Peer-reviewing data covers a range of document types: the manuscript itself, its review reports, rebuttals, discussion threads, meta-reviews, and potentially a camera-ready revision.

“It covers a range of document types: the manuscript itself, its review reports, rebuttals, discussion threads, meta-reviews, and-potentially-a camera-ready revision.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Peer-reviewing data includes many complex document types that are semantically and structurally interrelated.

“Peer-reviewing data includes many complex document types that are semantically and structurally interrelated.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Many peer-reviewing documents are multimodal, combining written language with categorical metadata, structure, non-linguistic elements, and numerical scores.

“Many of these documents are multimodal in a broad sense, combining written language with categorical metadata, structure (e.g., manuscript layout), non-linguistic elements (e.g., figures, tables, formulae, formatting), and numerical scores (e.g., overall and confidence scores).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Peer-reviewing data includes diverse document types authored by multiple parties.

“Peer-reviewing data includes diverse document types authored by multiple parties”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Peer-reviewing data is complex and diverse.

“Peer-reviewing data is complex and diverse (Figure 5).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The ACL-2023 conference alone generated over 12,000 reviews for over 4,500 full-text manuscripts, together with meta-reviews, discussions, and revisions.

“the ACL-2023 conference alone generated over 12,000 reviews for over 4,500 full-text manuscripts (Rogers et al., 2023b), together with the accompanying meta-reviews, discussions, and revisions.”

• cites Rogers et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Peer-reviewing data is interconnected.

“An additional challenge stems from the fact that peer-reviewing data is interconnected.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 'Yes-Yes-Yes' workflow used at ACL Conferences through the ACL Rolling Review system is an example of a peer-reviewing data collection process that takes confidentiality, personal data, and licensing into account.

“One example of a peer-reviewing data collection process that takes confidentiality, personal data, and licensing into account is the 'Yes-Yes-Yes' workflow used at ACL Conferences through the ACL Rolling Review system.”

• AssumptionRests on an assumption: Peer-reviewing data in a closed-review system is confidential.
• AssumptionRests on an assumption: Some peer-reviewing data relates to natural persons (authors, reviewers, meta-reviewers, program chairs).
• AssumptionRests on an assumption: Peer-reviewing data constitutes intellectual property of its respective authors.

Probably not a substantial claim: may not pass S3 (load-bearing).

In the reviewing campaigns of September to November 2021, over 1,900 reviewers and the authors of 235 submissions contributed their data, later published as part of the NLPeer corpus.

“In the reviewing campaigns of September to November 2021, over 1,900 reviewers and the authors of 235 submissions contributed their data, later published as part of the NLPeer corpus (Dycke et al., 2023a).”

• cites Dycke et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The number of scientific articles produced every year is growing rapidly.

“The number of scientific articles produced every year is growing rapidly.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Peer review is widely used, yet it is hard, time-consuming, and prone to error.

“Peer review is widely used, yet it is hard, time-consuming, and prone to error.”

Probably not a substantial claim: may not pass S2 (checkable).

Natural Language Processing has great potential to improve reviewing.

“Natural Language Processing (NLP) has great potential to improve reviewing.”

Probably not a substantial claim: may not pass S4 (based).

The artifacts involved in peer review-manuscripts, reviews, discussions-are largely text-based.

“Since the artifacts involved in peer review-manuscripts, reviews, discussions-are largely text-based”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Bidding should be understood as a measure of reviewer interest rather than expertise.

“Bidding should be understood as a measure of reviewer interest rather than expertise; for example, bidders might be more likely to bid on submissions that they believe will be high quality, justifying the effort of reading them carefully.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Reviewer-paper matching is a crucial step: it is at this stage that both the qualification gap and strategic behavior are best mitigated.

“Reviewer-paper matching is a crucial step: it is at this stage that both the qualification gap and strategic behavior are best mitigated.”

Probably not a substantial claim: may not pass S4 (based).

Simple NLP automation would be sufficient to detect the presence of these sections in the submission.

“simple NLP automation would be sufficient to detect the presence of these sections in the submission.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Although there are differences between specific conferences and editions, the underlying peer-reviewing process at AI conferences is by and large similar.

“Although there are differences between specific conferences (and even between different editions of the same conference), the underlying peer-reviewing process is by and large similar.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Peer review provides both a new testing ground for approaches that emerge elsewhere, and can serve as a unique source of new tasks, data, and methods that can benefit other application areas.

“Peer review provides both a new testing ground for approaches that emerge elsewhere, and can serve as a unique source of new tasks, data, and methods that can benefit other application areas.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Such analysis would help program chairs and meta-reviewers identify submissions that could be manually reviewed as candidates for desk rejection and mentoring programs, as well as help balance the reviewing load and study the effects of manuscript quality on the subsequent process even if desk rejection does not take place.

“Such analysis would help program chairs and meta-reviewers identify submissions that could be manually reviewed as candidates for desk rejection and mentoring programs, as well as help balance the reviewing load and study the effects of manuscript quality on the subsequent process even if desk rejection does not take place.”

Probably not a substantial claim: may not pass S2 (checkable).

NLP techniques could be leveraged to support peer review.

“Thus, NLP techniques could be leveraged to support peer review.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

At a large conference it is virtually guaranteed that some reviewers will have incomplete and inaccurate profiles.

“First, the publication history of a potential reviewer does not necessarily reflect their expertise or their current interests-and at a large conference it is virtually guaranteed that some reviewers will have incomplete and inaccurate profiles.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Addressing individual well-defined problems in peer review could make peer review more efficient and robust, save researcher time, and increase satisfaction and trust in the peer review process.

“addressing them could make peer review more efficient and robust, save valuable researcher time, and increase satisfaction and trust in the peer review process.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Some individual well-defined problems within peer review can be addressed.

“some individual well-defined problems within peer review can and should be addressed”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Tools like ACL pubcheck offer some of this functionality and can be extended.

“Tools like ACL pubcheck 4 offer some of this functionality and can be extended.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Before a submission enters the review process, it undergoes a range of additional checks to make sure it adheres to the standards and requirements of the venue.

“Before a submission enters the review process, it undergoes a range of additional checks to make sure it adheres to the standards and requirements of the venue.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

To a large extent, reviewing work is text work, and some parts of it bear resemblance to annotation work.

“To a large extent, reviewing work is text work , and some parts of it bear resemblance to annotation work (Rogers and Augenstein, 2020).”

• cites Rogers and Augenstein, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Many of the challenges in NLP for peer review are special cases of general NLP challenges familiar from other domains.

“As we shall see, many of the challenges in NLP for peer review are special cases of general NLP challenges familiar from other domains.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Applying NLP to peer review meets a range of serious challenges that need to be addressed before full automation of peer review can be considered.

“As we show, applying NLP to peer review meets a range of serious challenges that need to be addressed before any approaches to full automation of peer review can even be considered .”

Probably not a substantial claim: may not pass S4 (based).

The publication history of a potential reviewer does not necessarily reflect their expertise or their current interests.

“First, the publication history of a potential reviewer does not necessarily reflect their expertise or their current interests-and at a large conference it is virtually guaranteed that some reviewers will have incomplete and inaccurate profiles.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Keywords are more interpretable than text-based similarity scores, and can serve as explanations for the matches.

“Keywords are also more interpretable than text-based similarity scores, and can serve as explanations for the matches.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Failing to pass these checks can lead to desk rejection without review.

“Failing to pass these checks can lead to desk rejection without review.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

TL;DRs, graphical abstracts, and video previews are used to make the final conference program easier to navigate, are shown in user interfaces for reviewers and chairs, and are published as part of a centralized effort to promote conference submissions via social media.

“Such materials are used to make the final conference program easier to navigate, are shown in the user interfaces for reviewers and chairs, and are published as part of a centralized effort to promote conference submissions via conference social media accounts.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Assistance with adjusting a given work to other presentation modes can additionally speed up the submission process for the authors.

“Assistance with adjusting a given work to other presentation modes can additionally speed up the submission process for the authors.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

While particular artifacts and roles might be specific to AI conferences, most concepts involved are discipline-agnostic and can be easily adapted to other research communities and non-academic use cases.

“We also note that while particular artifacts and roles might be specific to AI conferences, most concepts involved are discipline-agnostic and can be easily adapted to other research communities and non-academic use cases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Using NLP systems to improve writing and clarity without altering the intellectual contribution of the manuscript is allowed.

“However, using NLP systems to improve writing and clarity without altering the intellectual contribution of the manuscript is allowed.”

Probably not a substantial claim: may not pass S2 (checkable).

Clarity issues can make manuscripts harder to read and evaluate, steering the precious resource of peer review away from the main scientific content of the submission.

“Clarity issues can make manuscripts harder to read and evaluate, steering the precious resource of peer review away from the main scientific content of the submission.”

Probably not a substantial claim: may not pass S4 (based).

Conflicts arising from supervisory relationships, co-holding grants, working at the same institution, being students of the same supervisor, etc. are hard to infer, as there exists no single, comprehensive and reliable source of such data.

“First, conflicts arising from supervisory relationships, co-holding grants, working at the same institution, being students of the same supervisor, etc. are hard to infer, as there exists no single, comprehensive and reliable source of such data.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Initial manuscript quality plays a large role in the subsequent process, motivating the deployment of advanced assessment tools already at the screening stage.

“Initial manuscript quality plays a large role in the subsequent process, motivating the deployment of advanced assessment tools already at the screening stage.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Mistakenly rejected good work delays the dissemination of findings, causes unnecessary resubmission effort, and has a direct impact on researchers' careers.

“Mistakenly rejected good work, in turn, delays the dissemination of findings, causes unnecessary resubmission effort, and has a direct impact on researchers' careers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Peer review is commonly used in scenarios that require non-trivial assessment of complex products at scale.

“Peer review is commonly used in scenarios that require non-trivial assessment of complex products at scale.”

• cites Lee et al., 2013 sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Texts are central to the peer-reviewing process: submitted manuscripts largely consist of text, and so do reviewing guidelines, organizers' communications, peer review reports, rebuttals, discussion comments, meta-reviews, and resulting publications along with amendment notes.

“We note that texts are central to the peer-reviewing process: submitted manuscripts largely consist of text, and so do the reviewing guidelines, organizers' communications, peer review reports, rebuttals, discussion comments, meta-reviews, and the resulting publications along with their amendment notes (Figure 1b).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

While some venues allow authors to add arbitrary keywords, these need to be concise yet descriptive, and distilling a paper to a good set of keywords requires thought and knowledge of the field.

“While some venues allow the authors to add arbitrary keywords to their submissions, these need to be concise yet descriptive, and distilling a paper to a good set of keywords requires thought and knowledge of the field.”

Probably not a substantial claim: may not pass S4 (based).

Passing peer review gives manuscripts the special status of a 'peer-reviewed scientific publication', which often plays a large role in formal evaluations, researchers' standing, and perception of the work.

“A lot is at stake: passing peer review gives manuscripts the special status of a 'peer-reviewed scientific publication', which often-for the better or worse-plays a large role in formal evaluations ( ' x peer-reviewed papers to get a PhD' ), researchers' standing within the field ( 'over 100 papers in A* venues' ), and perception of the work both by fellow researchers and by non-experts and media”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

NLP assistance could help analyze the submissions for coherence, citation coverage, and readability.

“For example, NLP assistance could help analyze the submissions for coherence, citation coverage, and readability (see Section 5.1).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

At ACL-2023 most submissions focused on English, and relatively few focused on other languages.

“At ACL-2023 most submissions focused on English, and relatively few focused on other languages.”

May combine two claims.

Organizational measures such as reviewer mentoring and training courses exist, but low-quality reviewing remains a persistent issue.

“While there exist organizational measures to improve reviewing quality, such as reviewer mentoring (Stelmakh et al., 2021d) and training courses 1 , low-quality reviewing remains a persistent issue.”

• cites Stelmakh et al., 2021dNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

As conferences grow, curating sets of reviewers becomes increasingly difficult for the program chairs.

“As conferences grow, curating sets of reviewers becomes increasingly difficult for the program chairs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The description of peer review is modular, and many challenges and solutions outlined in the paper are applicable to a wider range of reviewing systems, such as journal review and open post-publication review.

“We note that our description is modular , and many challenges and solutions outlined in this paper will be applicable to a wider range of reviewing systems, e.g., journal review and open post-publication review.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Reviewer-paper matching is a critical step, and it can materially improve matching quality and consequently the quality of the overall peer review process.

“It is a critical step, and it can materially improve matching quality and consequently the quality of the overall peer review process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Coordinating massive reviewing efforts poses new logistical challenges for the organizers of the peer-reviewing process.

“Coordinating these massive efforts poses new logistical challenges for the organizers of the peer-reviewing process”

• AssumptionRests on an assumption: Major AI conferences such as AAAI regularly attract over ten thousand submissions each.

Probably not a substantial claim: may not pass S3 (load-bearing).

Deciding on some submission metadata can be challenging.

“Yet, deciding on some of this metadata can be challenging.”

Classified as probably not a substantial claim.

Often, a submission would not directly fit any of the available tracks and keywords.

“Often, a submission would not directly fit any of these tracks and keywords.”

• AssumptionRests on an assumption: The ACL-2023 conference featured 26 distinct tracks, and AAAI-2024 used 14 primary and 282 secondary keywords as metadata.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Increasing submission rates make it necessary to lower the barrier of entry for reviewing, with more junior researchers joining the reviewer pool, especially in rapidly growing fields like machine learning and NLP.

“Increasing submission rates make it necessary to lower the barrier of entry for reviewing, with more and more junior researchers joining the reviewer pool-especially in rapidly growing fields like machine learning and NLP.”

May combine two claims.

Most conferences ask reviewers to specify their COI manually, but this approach can fail if reviewers are forgetful, have too many conflicts, or strategically choose to under-report their COIs.

“Most conferences ask reviewers to specify their COI manually, but this approach can fail if reviewers are forgetful, have too many conflicts, or strategically choose to under-report their COIs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Such functionality is already offered by many commercial tools such as Grammarly and WordTune, as well as specialized academic writing support systems like Curie, PaperPal, and Writefull.

“Such functionality is already offered by many commercial tools such as Grammarly ( https://www.grammarly.com ) and WordTune ( https://www.wordtune.com ), as well as specialized academic writing support systems like Curie ( https://www.aje.com/curie ), PaperPal ( https://paperpal.com ), and Writefull ( https://www.writefull.com ).”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Peer review is not one isolated action and not a single NLP task; it is a complex process with dozens of tasks, all requiring human effort and prone to failure.

“Yet, peer review is not one isolated action and not a single NLP task. It is a complex process with dozens of tasks, which all require human effort, and are all prone to failure.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Tools to support academic writing are abundant, although currently there exist no ready-to-use solutions that can do it all.

“Tools to support academic writing are abundant, although currently there exist no ready-to-use solutions that can do it all (Altmäe et al., 2023).”

• cites Altmäe et al., 2023 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The choice of the best method for combining individual scores is not clear.

“In general, the choice of the best method for combining the individual scores is not clear, and PCs commonly use heuristics to decide on the combination method.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Technologies that introduce new intellectual content to the paper are not allowed, since they bear risk of plagiarizing ideas or parts of text from published sources.

“Generally, technologies that introduce new intellectual content to the paper are not allowed, since they bear risk of plagiarizing ideas or parts of text from published sources.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Mistakenly approved bad work can misguide follow-up research and misinform policymaking and public opinion.

“Mistakenly approved bad work can misguide follow-up research and misinform policymaking and public opinion.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Reviewers are themselves researchers, and they often have their own work under review for the same venue.

“Yet, by definition, reviewers are themselves researchers, and they often have their own work under review for the same venue.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Keyword suggestion faces challenges due to the need to disambiguate unrelated keywords and connect related keywords.

“Keyword suggestion faces additional challenges due to the need to disambiguate unrelated keywords and connect the related keywords: for example, the authors might chose 'human factors : ethics' as metadata for their submission, while the reviewer may choose 'philosophical foundations : human subject experiments'.”

Probably not a substantial claim: may not pass S3 (load-bearing).

PCs commonly use heuristics to decide on the combination method.

“In general, the choice of the best method for combining the individual scores is not clear, and PCs commonly use heuristics to decide on the combination method.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Determining how well and in what ways a given reviewer is qualified to review a given submission offers considerable scope for improvement via NLP techniques.

“In this section, we discuss a key precursor to matching that offers considerable scope for improvement via NLP techniques: determining how well and in what ways a given reviewer is qualified to review a given submission.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Several AI conferences have recently introduced compulsory discussions of limitations and broader ethical impact of the submissions.

“Several AI conferences have recently introduced compulsory discussions of limitations and broader ethical impact of the submissions;”

Probably not a substantial claim: may not pass S4 (based).

Re-using the data from prior conferences to determine a reviewer's past performance by the current program chairs meets even greater confidentiality risks.

“Yet, re-using the data from prior conferences to determine a reviewer's past performance by the current program chairs meets even greater confidentiality risks, and calls for novel applications of privacy-preserving techniques such as anonymization and differential privacy (Section 9).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Many such systems also offer a paraphrasing functionality, which currently focuses on applying scientific writing style to the text rather than improving the structure and organization of a draft.

“Many such systems also offer a paraphrasing functionality, which currently focuses on applying scientific writing style to the text rather than improving the structure and organization of a draft.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Collusion rings have become a topic of grave concern because of recent reports of their increasing prevalence at major computer science conferences.

“Collusion rings have become a topic of grave concern because of recent reports of their increasing prevalence at major computer science conferences (Vijaykumar, 2020; Littman, 2021).”

• cites Vijaykumar, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Littman, 2021 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

A poor selection of keywords will fail to adequately differentiate large groups of submissions and reviewers, which in turn will impact the quality of the subsequent matching.

“Yet, a poor selection of keywords will fail to adequately differentiate large groups of submissions and reviewers, which in turn will impact the quality of the subsequent matching.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Submission metadata is important: keywords are often used in reviewer-paper matching and later in conference program navigation; the track sets the context for the review and later presentation; and the contribution type helps reviewers with evaluation.

“This metadata is important: keywords are often used in reviewer-paper matching (Section 4.2) and later in conference program navigation; the track sets the context both for the review and for the later presentation of the accepted manuscripts; and the contribution type helps reviewers with evaluation.”

Classified as probably not a substantial claim.

This naturally leads to calibration issues.

“This naturally leads to calibration issues.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Due to the competitive environment, reviewers might behave strategically, for example, by enforcing citations of their own work or by writing 'torpedo reviews' aimed to scuttle certain topics or competing groups.

“Due to the competitive environment, reviewers might behave strategically, for example, by enforcing citations of their own work or by writing 'torpedo reviews' aimed to scuttle certain topics or competing groups.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Li et al. (2023) explore a summarization model that incorporates inter-document relationships between reviews, author responses, and discussions.

“Li et al. (2023) explore a summarization model that incorporates inter-document relationships between reviews, author responses, and discussions”

• cites Li et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Each reviewer normally evaluates only a small subset of submissions.

“Given the high effort associated with reviewing, each reviewer normally evaluates only a small subset of submissions.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Due to the fast pace and increasing topical diversity of modern research, such in-depth understanding might be lacking even for experienced reviewers.

“Due to the fast pace and increasing topical diversity of modern research, such in-depth understanding might be lacking even for experienced reviewers.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The utility of the discussion phase is often debated: while some studies find that the discussion phase did little to alter reviewers' opinion, others reveal favorable outcomes.

“The utility of the discussion phase is often debated: while some studies find that the discussion phase did little to alter reviewers' opinion (Daumé III, 2013; Shah et al., 2018), others reveal favorable outcomes (Parno et al., 2017; Frachtenberg and Koster, 2020).”

• cites Daumé III, 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Shah et al., 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Parno et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Kuznetsov et al. (2022) investigate the connections between review comments and manuscript edits.

“Kuznetsov et al. (2022) investigate the connections between review comments and manuscript edits”

• cites Kuznetsov et al., 2022 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Peer review is an argumentative text that pursues a range of communicative goals.

“Peer review is an argumentative text that pursues a range of communicative goals.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Shen et al. (2022) and Kumar et al. (2021) explore meta-review generation in a controlled generation setting conditioned on the meta-review score.

“Shen et al. (2022) and Kumar et al. (2021) explore meta-review generation in a controlled generation setting conditioned on the meta-review score.”

• cites Shen et al. (2022) sourceNot checked — no full text of the source
• cites Kumar et al. (2021) sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based).

Existing works on aspect-based analysis of peer reviews operate with fairly coarse-grained aspect labels.

“We note that existing works on aspect-based analysis of peer reviews operate with fairly coarse-grained aspect labels.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The meta-reviewer's task of aggregating information from prior peer-reviewing data involves three core challenges: the volume of information to process, its varying quality, and the argumentative nature of the underlying texts.

“The meta-reviewer's task of aggregating information from the prior peer-reviewing data meets three core challenges: the amount of information that needs to be processed, the varying quality of this information, and the inherently argumentative nature of the underlying texts.”

• AssumptionRests on an assumption: Authors want their manuscript to be accepted and will argue in favor of their paper, possibly trying to minimize issues identified by the reviewers or to provide easy solutions to reviewers' concerns.
• AssumptionRests on an assumption: Reviewers want the manuscript critically assessed and will focus on perceived weaknesses and might disagree with each other.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hua et al. (2019) use argumentative structure to compare review composition in different computer science venues and investigate the relationships between review argumentation and scores.

“Hua et al. (2019) use argumentative structure to compare review composition in different computer science venues and investigate the relationships between review argumentation and scores.”

• cites Hua et al., 2019 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Santu et al. (2024) conduct in-depth experiments with prompting LLMs to perform the task.

“Santu et al. (2024) conduct in-depth experiments with prompting LLMs to perform the task.”

• cites Santu et al. (2024)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

While reviews must be substantiated based on the manuscript and related work, meta-reviews must additionally take into account the content of author responses and discussions.

“While reviews must be substantiated based on the manuscript and related work, meta-reviews must additionally take into account the content of author responses and discussions, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The task of meta-review generation is challenging due to the need to process diverse, interconnected texts which contain differing, often antagonistic opinions.

“This task is challenging due to the need to process diverse, interconnected texts which contain differing, often antagonistic opinions.”

• AssumptionRests on an assumption: Authors want their manuscript to be accepted and will argue in favor of their paper, possibly trying to minimize issues identified by the reviewers or to provide easy solutions to reviewers' concerns.
• AssumptionRests on an assumption: Reviewers want the manuscript critically assessed and will focus on perceived weaknesses and might disagree with each other.

Probably not a substantial claim: may not pass S4 (based).

Unlike research papers that generally follow an established discourse structure, the standards of peer review writing are less codified.

“Yet, unlike research papers that generally follow an established discourse structure, the standards of peer review writing are less codified.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A meta-review aims to facilitate decision making by the program chairs.

“a meta-review aims to facilitate decision making by the program chairs”

Probably not a substantial claim: may not pass S3 (load-bearing).

Guided reviewing interfaces could improve review quality for novice reviewers.

“Guided reviewing interfaces are another subclass of reading assistance for review: since novice reviewers would be unfamiliar with the process of writing a review, a tool that guides them through the process could improve review quality (Stelmakh et al., 2021d).”

• cites Stelmakh et al., 2021dNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Past studies also hypothesize and experiment with anchoring and herding effects among reviewers, wherein a reviewer might be (disproportionally) influenced by other reviewers or might follow the reviewer who initiates the discussion.

“Past studies also hypothesize and experiment with anchoring and herding effects among reviewers, wherein a reviewer might be (disproportionally) influenced by other reviewers (Teplitskiy et al., 2019) or might follow the reviewer who initiates the discussion (Stelmakh et al., 2023a).”

• cites Teplitskiy et al., 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Stelmakh et al., 2023aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

NLP has the potential to assist participants of the peer-reviewing process by helping authors revise manuscripts based on feedback and by helping reviewers, meta-reviewers and program chairs analyze resulting changes.

“NLP has the potential to assist the participants of the peer-reviewing process in two core ways: by helping authors revise the manuscript based on feedback, and by helping reviewers, meta-reviewers and program chairs analyze the resulting changes.”

Classified as probably not a substantial claim.

Rude, vague and unsubstantiated reviews are not uncommon.

“As a result, rude, vague and unsubstantiated reviews are not uncommon.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Reading for peer review is a hard expert task that often requires reviewers to build a deep understanding of the work and draw upon their background knowledge and analytical skills.

“Reading for peer review is a hard expert task that often requires reviewers to build a deep understanding of the work and draw upon their background knowledge and analytical skills.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Post-conference reports cover various aspects and changes to the review process for a given conference.

“Examples of such analysis include post-conference reports (Shah et al., 2018; Rogers et al., 2023b) that cover various aspects and changes to the review process for a given conference”

• cites Shah et al., 2018Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Rogers et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Candidate submissions for best paper awards are usually determined either by asking reviewers to nominate papers they review, or by considering manuscripts with very high reviewer scores.

“Given the large submission numbers, candidate submissions are usually determined either by explicitly asking reviewers to nominate the papers they are reviewing, or by considering manuscripts that have received a very high score among reviewers.”

• cites [11]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Two possible approaches to promote discussion among reviewers are generic reminders-which are often ineffective and ignored-and personalized reminders-which are generally known to be more effective, but require effort by the meta-reviewers.

“Two possible approaches to promote discussion among reviewers are generic reminders-which are often ineffective and ignored-and personalized reminders-which are generally known to be more effective (Vervloet et al., 2012), but require effort by the meta-reviewers.”

• cites Vervloet et al., 2012 sourceNot checked — no full text of the source

• AssumptionRests on an assumption: Generic reminders to promote discussion among reviewers are often ineffective and ignored.

Probably not a substantial claim: may not pass S2 (checkable).

Peer reviews afford analysis across multiple dimensions, which can support meta-reviewers in their work.

“We note that peer reviews afford analysis across multiple dimensions, which can all support meta-reviewers in their work.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Evaluations of interventions in the peer-review process exist, such as the reviewer training experiment of Stelmakh et al., 2021d.

“or evaluations of interventions in the peer-review process, such as the reviewer training experiment of Stelmakh et al., 2021d.”

• cites Stelmakh et al., 2021dNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Lack of evaluation in the 'technical soundness' aspect can prompt the meta-reviewer to investigate the manuscript more carefully, and ultimately help prevent accidental acceptance of technically flawed work.

“For example, lack of evaluation in the 'technical soundness' aspect can prompt the meta-reviewer to investigate the manuscript more carefully, and ultimately help prevent accidental acceptance of technically flawed work.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Existing results in NLP-supported aggregated analysis of peer review are preliminary.

“Yet, existing results are preliminary and motivate future work.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Low-quality reviews are frustrating and commonplace, indicating the potential for review writing assistance.

“Low-quality reviews are frustrating and commonplace, indicating the potential for review writing assistance.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Reviewing requires in-depth understanding of prior work.

“Reviewing requires in-depth understanding of prior work.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Meta-reviews and review reports serve different purposes, and many reviewing assistance techniques would need adjustment to help meta-reviewers.

“It is important to note, however, that meta-reviews and review reports serve different purposes, and many of the reviewing assistance techniques would need adjustment to help meta-reviewers.”

May combine two claims.

Existing NLP approaches help align manuscript revisions on sentence and paragraph level, detect new and changed content, and can automatically label edits by their intent.

“Existing NLP approaches help align manuscript revisions on sentence and paragraph level, detect new and changed content, and can automatically label the edits by their intent, e.g., grammar, clarity or factual adjustment.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The main purpose of a review report is to evaluate the work as thoroughly as possible.

“the main purpose of a review report is to evaluate the work as thoroughly as possible”

Classified as probably not a substantial claim.

Because of the time lag between initial reviewing and discussion, reviewers may struggle to interpret parts of the discussion pertaining to the submitted manuscript, other reviews, and author responses.

“Because of this time lag reviewers may struggle to interpret parts of the discussion pertaining to the submitted manuscript, other reviews, and author responses.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Peer review is a hard task prone to heuristics, such as 'results are not surprising', 'results do not surpass state of the art', 'the method is too simple', 'the authors should have done [X] instead', and others.

“Peer review is a hard task prone to heuristics, such as 'results are not surprising' , 'results do not surpass state of the art' , 'the method is too simple' , 'the authors should have done [X] instead' , and others.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Peer-reviewing data has unique features that set it apart from other NLP domains such as Wikipedia, newswire, social media, and scientific publications.

“Several unique features of peer-reviewing data set it apart from more traditional and well-covered application domains in NLP like Wikipedia, newswire, social media, and even the closely related domain of scientific publications.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Often the decision-making process requires a calibration step prior to ranking papers on any aspect of the meta-reviews.

“Often the decision-making process requires a calibration step prior to ranking papers on any aspect of the meta-reviews.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Reviewing takes substantial time and effort, and is rarely the reviewers' primary professional occupation.

“Reviewing takes substantial time and effort, and is rarely the reviewers' primary professional occupation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

At many AI conferences automated reviewer-paper assignment is only the first step, followed by a manual adjustment of the results.

“At many AI conferences automated reviewer-paper assignment is only the first step, followed by a manual adjustment of the results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Plagiarized and fake reviews are also an issue that NLP tools can help detect and prevent.

“Finally, as with manuscripts, plagiarized and fake reviews are also an issue (Piniewski et al., 2024) that NLP tools can help detect and prevent.”

• cites Piniewski et al., 2024 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Meta-review generation is related to aspect-based sentiment analysis and multi-document summarization, and has recently received increased attention.

“Meta-review generation is related to aspect-based sentiment analysis and multi-document summarization, and has recently received increased attention.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of citations a manuscript accumulates is a simple yet imperfect metric of impact.

“One simple-yet imperfect-metric of impact is the number of citations that a manuscript accumulates (Plank and van Dalen, 2019)”

• cites Plank and van Dalen, 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Cases found to have large disagreements in this sense might point to situations that need careful consideration as they may indicate unethical meta-reviewing.

“Cases found to have large disagreements in this sense might point to situations that need careful consideration as they may indicate unethical meta-reviewing.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Reviewer anonymity aims to elicit objective reviews and protect the reviewers from potential backlash-yet anonymity also reduces accountability on the reviewer side.

“Reviewer anonymity aims to elicit objective reviews and protect the reviewers from potential backlash-yet anonymity also reduces accountability on the reviewer side.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Earlier meta-scientific studies of peer review used closed data.

“as it has been done in earlier meta-scientific studies of peer review (Bornmann, 2011; Tomkins et al., 2017, etc.)”

• cites Bornmann, 2011 sourceNot checked — no full text of the source
• cites Tomkins et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

As of today, we possess neither the technology capable of performing peer review in a satisfactory manner, nor the methodology to evaluate its effects on the affected community and the scientific process.

“As of today, we possess neither the technology capable of performing peer review in a satisfactory manner, nor the methodology to evaluate its effects on the affected community and the scientific process.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

NLP for peer review suffers from the lack of domain diversity.

“and NLP for peer review suffers from the lack of domain diversity.”

Probably not a substantial claim: may not pass S2 (checkable).

Any evaluation of reviewer expertise, such as a confidence score or ability to detect technical flaws, is only a proxy for actual reviewer expertise.

“Like language proficiency tests serve as one imperfect measure for the actual 'language proficiency,' any evaluation of 'reviewer expertise'-be it a confidence score, the ability to detect technical flaws, or something else-is similarly just a proxy.”

Probably not a substantial claim: may not pass S2 (checkable).

Many variables in NLP for peer review, including review quality, manuscript quality, and reviewer expertise, are constructs that cannot be observed directly rather than hard objective properties.

“Many variables in NLP for peer review are not hard objective properties. Instead, they are constructs -hypothetical variables that we can't observe directly.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Using open-source models promotes scientific openness and reproducibility, yet increases the risks of malicious behavior by adversarial parties who can exploit vulnerabilities in these tools.

“Using open-source models promotes scientific openness and reproducibility-yet increases the risks of malicious behavior by adversarial parties who can exploit vulnerabilities in these tools.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

It is not possible to foresee all potential consequences of NLP assistance for peer review.

“We stress that it is not possible to foresee all potential consequences of NLP assistance for peer review.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Closed data is problematic in terms of reproducibility, makes follow-up research and deployment of derivative NLP systems challenging, and brings new administrative and technical challenges of redistributing the data and tracking its use.

“Yet, closed data is problematic in terms of reproducibility, makes follow-up research and deployment of derivative NLP systems challenging, and brings new administrative and technical challenges of redistributing the data and tracking its use.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

New algorithms require transparency, but the human process of decision making also requires transparency, which can be improved in existing processes as well.

“Similarly, in terms of transparency, new algorithms require transparency, but the human process of decision making also requires transparency, which can be improved in existing processes as well.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Authors of papers are biased by the positivity of reviews, while third-party evaluators are biased by the length of reviews.

“Goldberg et al. (2023b) find that authors of the papers are biased by the positivity of reviews, while third-party evaluators are biased by the length, etc.”

• cites Goldberg et al. (2023b)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The disadvantage of donation-based data collection is that it takes time and effort, and needs to take into account confidentiality, personal data regulations, and licensing.

“The disadvantage is that it takes time and effort, and needs to take into account the confidentiality, personal data regulations, and licensing.”

May combine two claims.

GDPR imposes requirements on the collection of reviewing data, including informed consent, clear reflection on data collected and purpose, storage and management, withdrawal and consent revocation.

“it still imposes requirements on the collection of reviewing data, including informed consent, a clear reflection on what exact data is collected and for what purpose, how the data is stored and managed, whether the data can be withdrawn and consent revoked, etc.”

• cites Kamocki et al., 2018 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

May combine two claims.

Data from open reviewing may not be representative of closed-reviewing scenarios and may carry biases that influence observed outcomes.

“Further, even if well-suited for extracting high-level insights about reviewing processes in a given community, the data might be not representative of closed-reviewing scenarios and carry potential biases that influence the observed outcomes.”

May combine two claims.

The sharing of peer-reviewing data and the use of models to assist in the peer review process could potentially be abused to infer personal information about reviewers, threatening the integrity of the process.

“The sharing of peer-reviewing data and the use of models to assist in the peer review process could potentially be abused to infer personal information about the reviewers, threatening the integrity of the process.”

May combine two claims.

The performance of AI tools can be misleading, and automated systems that seem to work well in the majority of cases can induce complacency, bringing new risks in terms of bias propagation, systematic error, outcome disparity, and safety.

“Yet, the performance of AI tools can be misleading, and automated systems that seem to work well in the majority of cases can induce complacency, bringing new risks in terms of bias propagation, systematic error, outcome disparity, and safety.”

• cites ProPublica, 2016Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Deploying an off-site LLM to process closed (or not-yet-open) peer-reviewing data raises ethical and legal concerns.

“deploying an off-site LLM to process closed (or not-yet-open) peer-reviewing data raises ethical and legal concerns.”

• AssumptionRests on an assumption: Peer-reviewing data in a closed-review system is confidential.
• AssumptionRests on an assumption: Some peer-reviewing data relates to natural persons (authors, reviewers, meta-reviewers, program chairs).
• AssumptionRests on an assumption: Peer-reviewing data constitutes intellectual property of its respective authors.
• AssumptionRests on an assumption: The use of third-party services to process peer-reviewing data became increasingly common with the advent of closed-source LLMs accessible via APIs.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Peer review reports form clusters with respect to their quality, with agreement on clearly high- and low-quality reports and a broad class of borderline cases where indicators of review quality are less reliable.

“Like manuscripts that typically fall into 'clear accept', 'clear reject', and 'borderline' categories, peer review reports form clusters with respect to their quality. Thus, while we can expect agreement on review reports that are clearly high- and low-quality, we can also expect a broad class of borderline cases, where our indicators of review quality would be less reliable.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The advantage of donation-based data collection is that it theoretically can be applied to any community and any venue.

“The advantage of data collection on a donation basis is that it theoretically can be applied to any community and any venue.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Unfair decisions will not magically go away, and authors will always disagree with rejections.

“Unfair decisions will not magically go away, and authors will always disagree with rejections (Goldberg et al., 2023b, Section 4.2).”

• cites Goldberg et al., 2023b, Section 4.2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Open reviewing data comes from only few communities, so it might not be representative of the diverse reviewing practices in a new community or discipline.

“Since open reviewing data comes from only few communities, it might not be representative of the diverse reviewing practices in a new community or discipline.”

Probably not a substantial claim: may not pass S2 (checkable).

The complexity and inter-connectedness of reviewing data bring new challenges.

“At the same time, the complexity and inter-connectedness of the reviewing data bring new challenges.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Public availability of reviews might affect the tone and composition of the review reports.

“The sole fact that reviews are made publicly available (sometimes with reviewer identity disclosed, such as at F1000Research) might affect the tone and composition of the review reports”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Once NLP systems are deployed, users will likely adjust their behavior over time to gain further benefits from the system, intentionally or not.

“Given the high-stakes nature of the process, once NLP systems are deployed, the users will likely adjust their behavior over time to gain further benefits from the system-intentionally or not.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Algorithms may introduce biases, but even without algorithms, people already have biases encoded in their evaluation and opinions.

“For example, algorithms may introduce biases, but even without algorithms, people already have biases encoded in their evaluation and opinions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GDPR regulates the use of personal data of EU subjects anywhere in the world.

“which regulates the use of personal data of EU subjects anywhere in the world (Rogers et al., 2021)”

• cites Rogers et al., 2021 sourceNot checked yet

  Checked before — its earlier verdict is held back until a new check runs.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

In order to make peer-reviewing data openly available for follow-up research, it must be associated with a license that regulates its downstream use.

“Thus, in order to make peer-reviewing data openly available for follow-up research, it must be associated with a license that regulates its downstream use.”

• AssumptionRests on an assumption: Peer-reviewing data constitutes intellectual property of its respective authors.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Making peer-reviewing data open via donation-based data collection requires explicit permission from the participants.

“Making this data open via donation-based data collection requires explicit permission from the participants.”

• AssumptionRests on an assumption: Peer-reviewing data in a closed-review system is confidential.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

To judge measurement validity, multiple alternative ways to measure the variable of interest and other related variables are needed.

“We note that to judge measurement validity, we need multiple alternative ways to measure the variable of interest and other related variables.”

Probably not a substantial claim: may not pass S4 (based).

NLP assistance models can pick up on idiolects of people who frequently published in the past.

“NLP assistance models can pick up on idiolects of people who frequently published in the past.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Objective evaluation and measurement of peer review variables meets two core challenges: not every variable can be objectively defined to yield a single ground truth, and even if defined, constructing an extensive gold dataset often requires substantial effort, prohibiting large-scale experimentation.

“Objective evaluation and measurement of peer review variables meets two core challenges: not every variable can be objectively defined to yield a single ground truth; and even if it is defined, constructing an extensive gold dataset often requires substantial effort, thereby prohibiting large-scale experimentation.”