LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Open accessAbstract
This work is motivated by two key trends. On one hand, large language models (LLMs) have shown remarkable versatility in various generative tasks such as writing, drawing, and question answering, significantly reducing the time required for many routine tasks. On the other hand, researchers, whose work is not only time-consuming but also highly expertise-demanding, face increasing challenges as they have to spend more time reading, writing, and reviewing papers. This raises the question: how can LLMs potentially assist researchers in alleviating their heavy workload? This study focuses on the topic of LLMs assist NLP Researchers, particularly examining the effectiveness of LLM in assisting paper (meta-)reviewing and its recognizability. To address this, we constructed the ReviewCritique dataset, which includes two types of information: (i) NLP papers (initial submissions rather than camera-ready) with both human-written and LLM-generated reviews, and (ii) each review comes with "deficiency" labels and corresponding explanations for individual segments, annotated by experts. Using ReviewCritique, this study explores two threads of research questions: (i) "LLMs as Reviewers", how do reviews generated by LLMs compare with those written by humans in terms of quality and distinguishability? (ii) "LLMs as Metareviewers", how effectively can LLMs identify potential issues, such as Deficient or unprofessional review segments, within individual paper reviews? To our knowledge, this is the first work to provide such a comprehensive analysis.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces ReviewCritique, a dataset of 100 NLP submissions with human reviews and 20 papers with LLM-generated reviews, annotated at sentence level for review deficiencies. It finds LLM reviews are more often deficient and less diverse than human reviews, and that even top LLMs struggle as meta-reviewers to identify deficient review segments.
Key findings
- LLM-generated reviews contained more Deficient instances than human-written reviews: 100% vs. 71.57% at the review level and 13.97% vs. 6.27% at the segment level.
- The top human error types were Misunderstanding (22.86%), Neglect (19.64%), and Inexpert Statement (18.23%); the top LLM error type was Out-of-scope (30.49%), followed by Misunderstanding (9.87%) and Superficial Review (9.42%).
- LLMs gave high recommendation scores regardless of acceptance status: averages of 7.43 for accepted and 7.47 for rejected papers, versus human averages of 6.41 for accepted and 4.81 for rejected submissions.
- Review diversity was lower for LLMs: ITF-IDF full-review scores were Human 6.04, Claude Opus 4.09, Gemini 3.82, and GPT-4 3.34; inter-LLM BERTScore was 70.33–71.17, while inter-human BERTScore was 59.15. As meta-reviewers, closed-source models generally outperformed open-source models, with Claude Opus achieving the highest F1 scores, but F1 scores remained relatively low.
Methods
- ReviewCritique was curated from OpenReview: 100 NLP papers submitted to ICLR and NeurIPS between 2020 and 2023, with equal accepted/rejected distribution, complete individual reviews (3–5 per submission), meta-reviews, and author rebuttals.
- A subset of 20 papers was used to collect LLM-generated reviews from GPT-4, Gemini-1.5, and Claude Opus, with three reviews per LLM using ICLR review guidelines and templates.
- Expert annotators (40 members; 16 with Ph.D. degrees, 11 university faculty, 15 former area chairs) labeled sentence-level Deficient segments and explanations; two annotators independently reviewed each paper, with disagreements resolved by a senior expert with area chair experience. Experiments compared LLMs as reviewers and as meta-reviewers using error-type analysis, component-level analysis, ITF-IDF diversity, BERTScore complementarity, and F1/ROUGE/BERTScore evaluation with Labeling-All and Select-Deficient prompts.
Limitations
- Stated: ReviewCritique focuses primarily on textual information from submissions and does not include figures, tables, or other visual elements.
- Stated: the dataset is limited to the NLP domain, and the work focuses on the pre-rebuttal phase, excluding the multi-turn aspects of author rebuttals and post-rebuttal reviews.
- Evident and acknowledged: the authors note they cannot guarantee complete annotator unawareness of LLM-generated reviews, and the LLM-generated review subset is limited to 20 papers because annotation was time-consuming.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 1 cited sources checked · 0 supported · 0 flagged
- Findingreasoning
LLMs are more prone to generating contradictory claims in their reviews.
“LLMs are more prone to generating contradictory claims in their reviews”
Quote in the paperSubstantial claim - Findingreasoning
LLMs are more prone to making claims that the authors never stated in the submission.
“making claims that the authors never stated in the submission”
Quote in the paperSubstantial claim - Findingown data
LLM-generated reviews contain more Deficient instances than human-written reviews (100% vs. 71.57% at the review level, and 13.97% vs. 6.27% at the segment level).
“LLM-generated reviews contain more Deficient instances compared to humanwritten reviews (100% vs. 71.57%at the review level, and 13.97% vs. 6.27% at the segment level).”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Rests on an assumption: The prompts used to generate LLM reviews (including ICLR review guidelines, randomly chosen human-written reviews, and a generation template) are appropriate and do not bias the comparison.
- Rests on an assumption: The three closed-source LLMs (GPT-4, Gemini-1.5, Claude Opus) are the models most likely to be used by humans seeking AI assistance in their reviews.
- Findingown data
GPT-4 and Gemini 1.5 show similar performance to Claude Opus.
“GPT-4 and Gemini 1.5 show similar performance to Claude Opus.”
Quote in the paperNot a substantial claim - Findingown data
In all LLM-generated reviews, LLMs consistently praise the writing of the papers, stating that they are well-written and easy to follow.
“In all LLMgenerated reviews, LLMs consistently praise the writing of the papers, stating that they are wellwritten and easy to follow.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
Among the papers used for generating LLM reviews, 15% of the papers had both the meta-reviewer and human reviewers agree that the writing was unclear and difficult to follow.
“However, among the papers used for generating LLM reviews, 15% of the papers had both the meta-reviewer and human reviewers agree that the writing was unclear and difficult to follow.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
LLMs often highlight weaknesses such as the need for more experiments, lack of generalizability, additional tasks, more analysis, evaluation on languages beyond English, etc.
“LLMs often highlight weaknesses such as the need for more experiments, lack of generalizability, additional tasks, more analysis, evaluation on languages beyond English, etc.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
Overall scores for all LLMs on explaining correctly identified Deficient segments are relatively low.
“The results in Table 5 show that overall scores for all LLMs are relatively low”
Quote in the paperSubstantial claim - Findingown data
Open-source models Llama3 (8B and 70B) and Qwen2-72B generally score lower than the closed-source models.
“The open-source models, Llama3 (8B and 70B) and Qwen2-72B, generally score lower than the closed-source models.”
Quote in the paperNot a substantial claim - Findingown data
LLMs struggle to accurately judge the paper writing quality submission.
“LLMs struggle to accurately judge the paper writing quality submission”
Quote in the paperSubstantial claim - Findingown data
For ITF-IDF, human reviews score the highest (6.04), followed by Claude Opus (4.09), Gemini (3.82), and GPT-4 (3.34).
“For ITF-IDF, from the full review perspective, human reviews score the highest (6.04), followed by Claude Opus (4.09), Gemini (3.82), and GPT-4 (3.34).”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
Even state-of-the-art LLMs struggle to assess review deficiencies effectively.
“even state-of-the-art LLMs struggle to assess review deficiencies effectively.”
Quote in the paperSubstantial claim - Findingown data
Among the LLMs, Claude Opus achieves the highest scores across all metrics on explanation evaluation.
“Among the LLMs, Claude Opus achieves the highest scores across all metrics”
Quote in the paperNot a substantial claim - Findingown data
Closed-source models (GPT-4, Claude Opus, and Gemini 1.5) generally outperform open-source models (Llama3-8B and 70B, Qwen2-72B) in F1 score.
“Closed-source models (GPT-4, Claude Opus, and Gemini 1.5) generally outperform open-source models (Llama3-8B and 70B, Qwen2-72B) in F1 score.”
Quote in the paperNot a substantial claim - Findingown data
A major reason for Deficient reviews from human reviewers is misunderstanding the paper submission and raising unnecessary concerns by neglecting information already stated.
“From Table 3, a major reason for Deficient reviews from human reviewers is misunderstanding the paper submission and raising unnecessary concerns by neglecting information already stated.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
Despite consensus among human experts that the writing was unclear for 15% of papers, the LLMs still provided positive feedback on the writing quality of these papers.
“Despite this consensus among human experts, the LLMs still provided positive feedback on the writing quality of these papers, failing to accurately assess the writing quality.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
GPT-4 tends to have the lowest ITF-IDF scores, suggesting more repetitive segments compared to other LLMs.
“The scores are relatively consistent across different sections, but GPT-4 tends to have the lowest scores, suggesting more repetitive segments compared to other LLMs.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
The performance rank of models on ITF-IDF scores remains the same across different similarity threshold t values.
“The performance rank remains the same across different t values.”
Quote in the paperNot a substantial claim - Findingown data
The Summary section in LLM-generated reviews exhibits relatively better quality compared to other aspects.
“The Summary section in LLM-generated reviews exhibits relatively better quality compared to other aspects.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingnone given
Compared to humans, LLMs are more likely to suggest out-of-scope experiments or analyses.
“Compared to humans, LLMs are more likely to suggest out-of-scope experiments or analyses.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
Human reviewers scored averages of 6.41 for accepted and 4.81 for rejected submissions.
“In contrast, human reviewers scored averages of 6.41 for accepted and 4.81 for rejected submissions.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingreasoning
Another significant error in human reviews is making inexpert critiques or statements due to insufficient domain knowledge.
“Another significant error is making inexpert critiques or statements due to insufficient domain knowledge, potentially from unqualified reviewers being involved due to the increasing number of submissions to AI/NLP conferences and the need to recruit more reviewers.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Findingown data
Claude Opus excels in identifying Deficient segments.
“Claude Opus also excels in identifying Deficient segments, as shown previously.”
Quote in the paperNot a substantial claim - Findingown data
ReviewCritique includes expert-annotated LLM-generated reviews, enabling direct comparison between human and LLM-generated reviews at a granular level.
“Additionally, ReviewCritique includes expert-annotated LLMgenerated reviews, enabling direct comparison between human and LLM-generated reviews at a granular level.”
Quote in the paperSubstantial claim - Findingown data
Human reviews maintain high diversity across all sections.
“Human reviews maintain high diversity across all sections.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Interpretationreasoning
LLMs have a tendency towards hallucination.
“indicating a tendency towards hallucination.”
Quote in the paperSubstantial claim - Interpretationown data
LLMs may lack the ability to accurately judge the writing quality of a paper submission.
“Our analysis suggests that LLMs may lack the ability to accurately judge the writing quality of a paper submission.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Interpretationown data
Human reviewers tend to produce more diverse reviews compared to the LLMs.
“This indicates that human reviewers tend to produce more diverse reviews compared to the LLMs.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Interpretationown data
LLMs, and AI in general, are currently inadequate for performing tasks that require a high level of expertise and nuanced judgment.
“Increase community awareness that LLMs, and AI in general, are currently inadequate for performing tasks that require a high level of expertise and nuanced judgment.”
Quote in the paperSubstantial claim- Rests on an assumption: The 100 NLP papers and their reviews from OpenReview are representative of NLP papers and reviewing practices.
- Interpretationown data
LLMs tend to accept authors' claims in submissions without much critical evaluation.
“LLMs tend to accept authors' claims in submissions without much critical evaluation.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Interpretationown data
The use of multiple LLMs may not necessarily lead to a significant increase in the diversity of perspectives and insights in the review process.
“In addition, this finding implies that the use of multiple LLMs may not necessarily lead to a significant increase in the diversity of perspectives and insights in the review process.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Interpretationown data
The six types Inaccurate Summary, Writing, Superficial Review, Experiment, Contradiction and Unstated Statement are particularly difficult for LLMs to detect.
“suggesting that these types of Deficient are particularly difficult for LLMs to detect: Inaccurate Summary, Writing, Superficial Review, Experiment, Contradiction and Unstated Statement”
Quote in the paperSubstantial claim- Rests on an assumption: The error type taxonomy defined by the annotation team is valid for categorizing Deficient segments.
- Interpretationown data
Human reviewers scrutinize the claimed strengths and provide their expert opinions on the validity and significance of the contributions.
“In contrast, human reviewers scrutinize the claimed strengths and provide their expert opinions on the validity and significance of the contributions.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Interpretationreasoning
ReviewCritique is the first dataset to benchmark LLMs' capability as a responsible meta-reviewer.
“In contrast, ReviewCritique is the first dataset to benchmark LLMs' capability as a responsible meta-reviewer.”
Quote in the paperNot a substantial claim - Premisecitation
Llama3-8B and Llama3-70B have a maximum supported context length limited to 8k tokens.
“the maximum supported context length is limited to 8k tokens 3”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
LLMs' failure to capture nuanced aspects leads to their inability to identify inaccurate summary errors.
“leading to their inability to identify inaccurate summary errors.”
Quote in the paperSubstantial claim - Interpretationreasoning
LLMs are currently insufficient to replace human reviewers, especially in tasks that require expert judgment and nuanced understanding.
“LLMs are currently insufficient to replace human reviewers, especially in tasks that require expert judgment and nuanced understanding.”
Quote in the paperSubstantial claim - Interpretationown data
LLMs do not align with human reviewers in discerning paper quality based on their internal scoring mechanism.
“Thus, LLMs do not align with the human reviewers in discerning paper quality based on their internal scoring mechanism.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
- Study design and conductnone given
For the annotation of LLM-generated reviews, a separate group of annotators were employed who were not informed that these reviews were LLM-generated.
“For the annotation of LLM-generated reviews, we employed a separate group of annotators who were not informed that these reviews were LLM-generated.”
Quote in the paperSubstantial claim - Interpretationown data
Claude Opus's explanations align best with human annotators.
“suggesting its explanations align best with human annotators.”
Quote in the paperSubstantial claim - Interpretationreasoning
These findings highlight the current limitations of LLMs in automating the peer review process.
“These findings highlight the current limitations of LLMs in automating the peer review process.”
Quote in the paperSubstantial claim
- May combine two claims. 18
- Classified as probably not a substantial claim. 9
- The quote only partly states this claim. 1
May combine two claims.
Findingown dataAmong all segments in the Strengths section of LLM-generated reviews, 53.2% are simply rephrased from the submission, while the remaining segments are mostly inferred from the introduction and abstract.
“Our analysis reveals that among all segments in the Strengths section of LLM-generated reviews, 53.2% are simply rephrased from the submission, while the remaining segments are mostly inferred from the introduction and abstract, where authors typically highlight their contributions.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
May combine two claims.
FindingreasoningAlthough LLMs can generate paper summaries with fewer errors, they may fail to capture nuanced aspects of the paper.
“although LLMs can generate paper summaries with fewer errors, they may fail to capture nuanced aspects of the paper”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor accepted papers, 34.5% of the Strength segments generated by LLMs were questioned by human experts in their corresponding human-written reviews; for rejected papers, this rose to 51.9%.
“For accepted papers, 34.5% of the Strength segments generated by LLMs were questioned by human experts in their corresponding human-written reviews.For rejected papers, this rose to 51.9%.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.
Findingown dataSix types of Deficient have a significantly lower percentage compared to the average recall of GPT-4 (47.68%), Claude Opus (42.12%), and Gemini 1.5 (50.37%).
“Red-colored types have a significantly lower percentage compared to the average recall of LLMs.”
Quote in the paperNot a substantial claim- Rests on an assumption: The error type taxonomy defined by the annotation team is valid for categorizing Deficient segments.
May combine two claims.
Findingown dataThe BERTScores between different LLM pairs are similar and high, ranging from 70.33 to 71.17, while the BERTScore between human reviewers is 59.15.
“The BERTScores between different LLM pairs are similar and high, ranging from 70.33 to 71.17.In comparison, the BERTScore between human reviewers is 59.15, which is noticeably lower than the scores between the LLMs.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
May combine two claims.
Findingnone givenThe authors created ReviewCritique, containing both human-written and LLM-generated reviews, with detailed deficiency annotations and explanations.
“We created ReviewCritique , containing both human-written and LLM-generated reviews, with detailed deficiency annotations and explanations.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataLLMs tend to give high scores to all submissions, regardless of quality or acceptance status, with averages of 7.43 for accepted and 7.47 for rejected papers.
“Experiment shows that LLMs tend to give high scores to all submissions, regardless of quality or acceptance status, with averages of 7.43 for accepted and 7.47 for rejected papers.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
May combine two claims.
Findingnone givenReviewCritique labels review deficiencies at the sentence level, demanding highly experienced annotators.
“First, ReviewCritique labels review deficiencies at the sentence level, demanding highly experienced annotators.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataLLMs tend to provide superficial reviews, often failing to offer constructive suggestions on experiments.
“tend to provide superficial reviews, often failing to offer constructive suggestions on experiments.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataWhile LLMs can generate reviews, they often produce Deficient and paper-unspecific segments, lacking the diversity and constructive feedbacks.
“while LLMs can generate reviews, they often produce Deficient and paper-unspecific segments, lacking the diversity and constructive feedbacks.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataClaude Opus achieves the highest F1 scores, with GPT-4 and Gemini 1.5 performing slightly worse.
“Claude Opus achieves the highest F1 scores, with GPT-4 and Gemini 1.5 performing slightly worse.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataThe annotator team consisted of 40 members from the NLP community, all with multiple first-authored publications in top-tier NLP venues and extensive reviewing experience.
“Our annotator team consisted of 40 members from the NLP community, all with multiple first-authored publications in top-tier NLP venues and extensive reviewing experience.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataDespite the superior performance of closed-source models, their F1 scores remain relatively low even with different prompt strategies, highlighting the challenges LLMs face in such expertise-intensive tasks.
“Despite the superior performance of the closedsource models, their F1 scores remain relatively low even with different prompt strategies, highlighting the challenges LLMs face in such expertiseintensive tasks and emphasizing the importance of human expertise in the meta-reviewing process.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningFor Llama3-8B and Llama3-70B, the full paper body text was truncated while the other prompt components were kept intact.
“we truncate the full paper body text while keeping the other components of the prompt intact.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataLLMs often accept authors' claims without thorough verification, treating strengths as a text summarization task.
“These findings suggest that LLMs often accept authors' claims without thorough verification, treating strengths as a text summarization task.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataDisagreements between the two annotators were resolved by a senior expert with area chair experience.
“Disagreements between the two annotators were resolved by a senior expert with area chair (AC) experience, who examined the conflicting annotations and resolved discrepancies by removing or rewriting the explanations for the unconvincing annotations.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe suggestions provided by LLMs in the Weaknesses section tend to be paper-unspecific and superficial, making them applicable to most NLP papers without offering actionable insights.
“Moreover, the suggestions provided by LLMs in the Weaknesses section tend to be paper-unspecific and superficial (e.g,The paper's focus on pretrained models might limit its applicability to domains where such models are not available or suitable.),making them applicable to most NLP papers without offering actionable insights to either authors or area chairs.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe dataset is currently limited to the NLP domain.
“the dataset is currently limited to the NLP domain.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenReviewCritique primarily focuses on the textual information from the submissions and does not include figures, tables, or other visual elements.
“ReviewCritique primarily focuses on the textual information from the submissions and does not include figures, tables, or other visual elements.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
Interpretationown dataThese suggestions often fall outside the paper's scope and shouldn't be considered weaknesses.
“While occasionally relevant, these suggestions often fall outside the paper's scope and shouldn't be considered weaknesses.”
Quote in the paperNot a substantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
May combine two claims.
Interpretationown dataLLMs can identify some Deficient segments but struggle to articulate their reasoning.
“indicating they can identify some Deficient segments but struggle to articulate their reasoning.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoningTwo annotators independently reviewed each paper's reviews without access to each other's annotations to prevent bias.
“To maintain annotation quality, two annotators independently reviewed each paper's reviews without access to each other's annotations to prevent bias.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe work focuses on the pre-rebuttal phase of the peer review process, assessing reviews based on the original submission.
“our work focuses on the pre-rebuttal phase of the peer review process, assessing reviews based on the original submission.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe approach of concealing submission information cannot guarantee complete unawareness of LLM-generated reviews.
“We acknowledge that this approach cannot guarantee complete unawareness.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataHuman reviewers are often able to sense the overall quality of a paper; if they believe a submission is of poor quality and intend to reject it, they tend to collect more weaknesses to justify their decision, which sometimes leads to overemphasis on the paper's flaws.
“human reviewers are often able to sense the overall quality of a paper.If they believe a submission is of poor quality and intend to reject it, they tend to collect more weaknesses to justify their decision, which sometimes leads to overemphasis on the paper's flaws.”
Quote in the paperSubstantial claim- Rests on an assumption: The collected individual reviews are written by human experts rather than AI.
The paper relies on this without support.
The collected individual reviews are written by human experts rather than AI.
“Question: How can we ensure that the collected individual reviews are written by human experts rather than AI?During the subsequent annotation process, we instruct annotators to notify us if they suspect that a review collected here was likely generated by AI; if any doubts arise, we will discard the paper and all its metadata.”
The paper relies on this without support.
The 100 NLP papers and their reviews from OpenReview are representative of NLP papers and reviewing practices.
“From the OpenReview website, we gathered 100 NLP papers (submitted to top-tier AI conferences ICLR and NeurIPS between 2020 and 2023) along with their complete individual reviews (3-5 for each submission), meta-reviews, and author rebuttals.”
The paper relies on this without support.
The prompts used to generate LLM reviews (including ICLR review guidelines, randomly chosen human-written reviews, and a generation template) are appropriate and do not bias the comparison.
“Each LLM generated three reviews using prompts that included the ICLR review guidelines, randomly chosen human-written reviews for both accepted and rejected papers, and a generation template in ICLR 2024 format.”
The paper relies on this without support.
The three closed-source LLMs (GPT-4, Gemini-1.5, Claude Opus) are the models most likely to be used by humans seeking AI assistance in their reviews.
“We utilized three of the most powerful closed-source LLMs, namely GPT-4 (OpenAI, 2023), Gemini-1.5 (Google, 2023), and Claude Opus (Anthropic, 2024), as these are the models most likely to be used by humans seeking AI assistance in their reviews.”
The paper relies on this without support.
The error type taxonomy defined by the annotation team is valid for categorizing Deficient segments.
“Each error type is accompanied by an explanation defined by our annotation team.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
GPT-4 full-review ITF-IDF score differs between text and Table 6
MinorThe text reports GPT-4's full-review score as 3.34, while Table 6 reports 3.37 at the stated threshold of 0.5.
“For ITF-IDF, from the full review perspective, human reviews score the highest (6.04), followed by Claude Opus (4.09), Gemini (3.82), and GPT-4 (3.34).”
“| GPT4 | 0.77 | 3.37 | 6.46 | 8.42 |”
Qwen model size differs between the evaluation and implementation details
MinorThe evaluation identifies the model as Qwen2-72B, but the listed model identifier is Qwen2-7B-Instruct. These specify different model sizes, leaving unclear which model produced the reported results.
“For open-source models, we evaluate Llama3-8B and -70B (AI@Meta, 2024) and Qwen2-72B (Bai et al., 2023).”
“Llama3-8B (Meta-Llama-3-8B-Instruct), Llama3-70B (Meta-Llama-3-70B-Instruct), and Qwen2-72B (Qwen/Qwen2-7B-Instruct).”
Reported review count conflicts with the stated generation procedure
MinorThe paper describes 20 selected papers and three LLMs, then says each LLM generated three reviews. That would produce 180 reviews if the three reviews were generated for each paper, whereas Table 1 reports only 60 LLM-generated reviews-one per model per paper. The procedure and count cannot both be correct as stated.
“Each LLM generated three reviews using prompts that included the ICLR review guidelines, randomly chosen human-written reviews for both accepted and rejected papers, and a generation template in ICLR 2024 format.”
“| #Reviews | 380 | 195 | 185 | 60 | 30 | 30 |”
Cites
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1× · see citing works
“In our experiments, we use SentenceBERT (Reimers and Gurevych, 2019) to calculate the similarity between segments.”
MethodBERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · cited 7× · see citing works
“We report the ROUGE and BERTScore values for each LLM and prompt combination in Table 10 and 11.”
MethodInfluentialA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023) · arXiv · doi:10.48550/arxiv.2307.06435 · Cited by 1,259 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Self-citationChatGPT and large language models in academia: opportunities and challenges
Meyer, J. G., Urbanowicz, R. J., Martin, P. C. N., O’Connor, K., Li, R., Peng, P.-C., Bright, T. J., Tatonetti, N., Won, K. J., Gonzalez-Hernandez, G., et al. (2023) · BioData Mining · doi:10.1186/s13040-023-00339-9 · Cited by 569 · see citing works
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 243 · cited 1× · see citing works
“Recently, researchers have also explored the evaluation of LLMs’ deficiency and limitations in automatic paper reviewing tasks (Zhou et al., 2024; Liu and Shah, 2023; Robertson, 2023; Liang et al., 2023; Lin et al., 2023a).”
BackgroundChallenges and Applications of Large Language Models
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., & McHardy, R. (2023) · arXiv · doi:10.48550/arxiv.2307.10169 · Cited by 196 · see citing works
Artificial intelligence in scientific writing: a friend or a foe?
Altmäe, S., Sola-Leyva, A., & Salumets, A. (2023) · Reproductive BioMedicine Online · doi:10.1016/j.rbmo.2023.04.009 · Cited by 174 · cited 1× · see citing works
“Interestingly, while researchers, the creators of AI/LLMs, benefit from LLMs for simple tasks (Meyer et al., 2023; Altmäe et al., 2023), it still takes years to train a qualified researcher due to the domain-specific and expertise-demanding nature of their work.”
BackgroundA Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · cited 1× · see citing works
“PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”
BackgroundCan We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96 · see citing works
Argument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 1× · see citing works
“Other datasets focus on specific aspects of peer reviews, such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et…”
BackgroundPeer review analyze: A novel benchmark resource for computational analysis of peer reviews
Ghosal, T., Kumar, S., Bharti, P. K., & Ekbal, A. (2022) · PLOS ONE · doi:10.1371/journal.pone.0259238 · Cited by 66 · cited 1× · see citing works
“Peer Review Analyze (Ghosal et al., 2022a) annotates reviews across four facets: paper section correspondence, aspect, functionality, and significance.”
BackgroundAPE: Argument Pair Extraction from Peer Review and Rebuttal via Multi-task Learning
Cheng, L., Bing, L., Yu, Q., Lu, W., & Si, L. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.569 · Cited by 58 · see citing works
Self-citationRasool, Z., Kurniawan, S., Balugo, S., et al. (2023) · Natural Language Processing Journal · doi:10.48550/arxiv.2311.07878 · Cited by 49 · cited 1×
“Artificial intelligence (AI), particularly through the recent development of large language models (LLMs), has demonstrated remarkable versatility in tasks such as writing, drawing, and question answering (Naveed et al., 2023; Rasool et al., 2024; Kaddour et al., 2023).”
BackgroundReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis
Wang, Q., Zeng, Q., Huang, L., Knight, K., Ji, H., & Rajani, N. F. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.44 · Cited by 38 · cited 2× · see citing works
“…area of interest is the use of AI to assist in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and…”
BackgroundAutomated scholarly paper review: Concepts, technologies, and challenges
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101830 · Cited by 37 · see citing works
Rasool, Z., Kurniawan, S., Balugo, S., Barnett, S., Vasa, R., Chesser, C., Hampstead, B. M., Belleville, S., Mouzakis, K., & Bahar-Fuchs, A. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100083 · Cited by 34 · see citing works
Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks
Zhou, R., Chen, L., & Yu, K. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/48d359hjdvog · Cited by 28 · cited 2× · see citing works
“One area of interest is the use of AI to assist in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and…”
BackgroundA Dataset of Argumentative Dialogues on Scientific Papers
Ruggeri, F., Mesgar, M., & Gurevych, I. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.425 · Cited by 28 · cited 1× · see citing works
“…datasets focus on specific aspects of peer reviews, such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et al.,…”
BackgroundReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · cited 2× · see citing works
“…in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and meta-reviews (Li et al., 2023; Lin et al., 2023b).”
BackgroundMReD: A Meta-Review Dataset for Structure-Controllable Text Generation
Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., & Si, L. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.198 · Cited by 23 · cited 1× · see citing works
“This contrasts with other works treating paper meta-review as a text summarization task given 3+ individual reviews (Li et al., 2023; Shen et al., 2022; Pradhan et al., 2021).”
ResultSelf-citationGregory, R. (1955) · AIEE-IRE '55 (Eastern) · doi:10.1145/1455319.1455328 · Cited by 20
Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
Deng, Z., Peng, H., Xia, C., Li, J., He, L., & Yu, P. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.555 · Cited by 19 · cited 1× · see citing works
“…the use of AI to assist in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and meta-reviews (Li et al.,…”
MethodSelf-citationNLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.277 · Cited by 19 · see citing works
DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions
Kennard, N., O’Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., & McCallum, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.89 · Cited by 17 · cited 1× · see citing works
“Other datasets focus on specific aspects of peer reviews, such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et…”
BackgroundPradhan, T., Bhatia, C., Kumar, P., & Pal, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.11.004 · Cited by 16 · cited 1× · see citing works
“This contrasts with other works treating paper meta-review as a text summarization task given 3+ individual reviews (Li et al., 2023; Shen et al., 2022; Pradhan et al., 2021).”
ResultGPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study
Robertson, Z. (2023) · arXiv · doi:10.48550/arxiv.2307.05492 · Cited by 15 · cited 1× · see citing works
“Recently, researchers have also explored the evaluation of LLMs’ deficiency and limitations in automatic paper reviewing tasks (Zhou et al., 2024; Liu and Shah, 2023; Robertson, 2023; Liang et al., 2023; Lin et al., 2023a).”
BackgroundGhosal, T., Varanasi, K. K., & Kordoni, V. (2022) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3529372.3533300 · Cited by 15 · cited 2× · see citing works
“…such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et al., 2023), and substantiation (Guo et al., 2023).”
BackgroundMulti-task Peer-Review Score Prediction
Li, J., Sato, A., Shimura, K., & Fukumoto, F. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.14 · Cited by 15 · cited 1× · see citing works
“One area of interest is the use of AI to assist in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and…”
PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews
Bharti, P. K., Navlakha, M., Agarwal, M., & Ekbal, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09662-3 · Cited by 12 · cited 3× · see citing works
“…such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et al., 2023), and substantiation (Guo et al., 2023).”
BackgroundMOPRD: A multidisciplinary open peer review dataset
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2022) · arXiv · doi:10.48550/arxiv.2212.04972 · Cited by 11 · cited 2× · see citing works
“…in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and meta-reviews (Li et al., 2023; Lin et al., 2023b).”
BackgroundSummarizing Multiple Documents with Conversational Structure for Meta-Review Generation
Li, M., Hovy, E. H., & Lau, J. H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.472 · Cited by 7 · cited 2× · see citing works
“…in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and meta-reviews (Li et al., 2023; Lin et al., 2023b).”
BackgroundResultWhen Reviewers Lock Horns: Finding Disagreements in Scientific Peer Reviews
Kumar, S., Ghosal, T., & Ekbal, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1038 · Cited by 7 · see citing works
Automatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · arXiv · doi:10.48550/arxiv.2311.11967 · Cited by 5 · cited 2×
“…such as argument (Kennard et al., 2022; Hua et al., 2019; Yuan et al., 2022; Cheng et al., 2020; Ruggeri et al., 2023), politeness (Bharti et al., 2023), uncertainty detection (Ghosal et al., 2022b), contradictions in review pairs (Kumar et al., 2023), and substantiation (Guo et al., 2023).”
BackgroundAutomatic Analysis of Substantiation in Scientific Peer Reviews
Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., & Clavel, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.684 · Cited by 5 · see citing works
Reviewer2: Optimizing Review Generation Through Prompt Generation
Gao, Z., Brantley, K., & Joachims, T. (2024) · arXiv · doi:10.48550/arxiv.2402.10886 · Cited by 3 · cited 1× · see citing works
“…in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and meta-reviews (Li et al., 2023; Lin et al., 2023b).”
NLPeer: A Unified Resource for the Computational Study of Peer Review
Dycke, N., Kuznetsov, I., & Gurevych, I. (2022) · arXiv · doi:10.48550/arxiv.2211.06651 · Cited by 1 · cited 1× · see citing works
“PeerRead (Kang et al., 2018), MOPRD (Lin et al., 2023b), and NLPeer (Dycke et al., 2023) are datasets containing a large number of peer reviews and their corresponding papers but without expert annotations.”
BackgroundOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
2 of the 41 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Wenting Xiong and Diane Litman. 2011. Automatically predicting peer-review helpfulness. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, pages 502-507.
Cited by
Sorted by citations
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., & Wang, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.70 · Cited by 33 · see citing works
Self-citationAAAR-1.0: Assessing AI's Potential to Assist Research
Lou, R., Xu, H., Wang, S., Du, J., Kamoi, R., Lu, X., Xie, J., Sun, Y., Zhang, Y., Ahn, J. J., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.22394 · Cited by 3 · see citing works
“For example, Si, Yang, and Hashimoto (2024) conducted a large-scale human investigation on LLM-generated research ideas and found that LLMs can generate novel ideas compared with humans while lacking feasibility.”
Self-citationGenerative Artificial Intelligence Policies under the Microscope
Nahar, M., Lee, S., Guillen, R., & Lee, D. (2025) · Communications of the ACM · doi:10.1145/3720536 · Cited by 4 · see citing works
Cogburn, D. L., Ochieng, T., Shikako‐Thomas, K., Woods, J., & Aydin, M. (2025) · Data & Policy · doi:10.1017/dap.2025.10017 · Cited by 4 · see citing works
SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
Xiang, Y., Yan, H., Ouyang, S., Gui, L., & He, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.00255 · Cited by 1 · see citing works
“For instance, Si et al. (2025); Gu & Krenn (2025) demonstrated how LLMs can be prompted to generate novel research ideas, while Yuan et al. (2022); Du et al. (2024) explored their use in producing literature reviews for idea evaluation.”
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning
Zou, Z., Ansari, A., Zhang, D. C., Lee, D., & Yin, W. (2026) · arXiv · doi:10.48550/arxiv.2601.07611
Self-citation
- Authors
- Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., Zhang, H. R., Gupta, V., Li, Y., Li, T., Wang, F., Liu, Q., Liu, T., Gao, P., Xia, C., Xing, C., Cheng, J., Wang, Z., Su, Y., Shah, R. S., Guo, R., Gu, J., Li, H., Wei, K., Wang, Z., Cheng, L., Ranathunga, S., Fang, M., Fu, J., Liu, F., Huang, R., Blanco, E., Cao, Y., Zhang, R., Yu, P. S., & Yin, W.
- Affiliations
- University of Illinois Chicago (US)
- Hong Kong Polytechnic University (HK)
- Pennsylvania State University (US)
- Tsinghua University (CN)
- Fudan University (CN)
- University of Southern California (US)
- University of California, Davis (US)
- University of Basel (CH)
- Xiaomi (China) (CN)
- Salesforce (United States) (US)
- Emory University (US)
- Georgia Institute of Technology (US)
- University of California, Santa Cruz (US)
- Singapore University of Technology and Design (SG)
- Massey University (NZ)
- University of Arizona (US)
- ORCID
- Du, Jiangshu
- Wang, Yibo
- Zhao, Wenting
- Deng, Zhongfen
- Liu, Shuaiqi
- Zou, Henry Peng
- Venkit, Pranav Narayanan
- Zhang, Nan
- Srinath, Mukund
- Gupta, Vipul
- Li, Yinghui
- Li, Tao
- Wang, Fei
- Liu, Qin
- Liu, Tianlin
- Xia, Congying
- Wang, Zhaowei
- Su, Ying
- Shah, Raj Sanjay
- Guo, Ruohao
- Li, Haoran
- Ranathunga, Surangika
- Fang, Meng
- Fu, Jie
- Huang, Ruihong
- Blanco, Eduardo
- Cao, Yixin
- Zhang, Rui
- Yu, Philip S.
- Yin, Wenpeng
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2406.16253
- OpenAlex
- W4400023672
- Semantic Scholar
- f2c7f69177d7972510e7a026c7fa20c26905a3df
- Cited by
- 2
- References
- 41
- Citations by year
- 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2346158, 2106758)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, review quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
