DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning

Prior systems often assume that identified weaknesses are inherently correct.

“Second, prior systems often assume that identified weaknesses are inherently correct (Yu et al., 2024;Chamoun et al., 2024), overlooking the fact that even human reviews can be biased, or mistaken.”

• cites Yu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chamoun et al., 2024 sourceCould not verify

  The Evaluation setup and Human vs LLM-generated reviews sections describe models hallucinating weaknesses, but neither says prior systems assume identified weaknesses are correct. The Human-written reviews section says human reviews are not treated as a gold standard; it does not establish the claimed assumption by prior systems. Searched for that assumption and for descriptions of how prior systems handle identified weaknesses. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

DIAGPaper achieves state-of-the-art performance across benchmarks.

“DIAGPaper achieves state-of-the-art performance across benchmarks.”

DIAGPaper's dominant failure mode (71.4%) involves weaknesses that are factually plausible but judged invalid by annotators because they impose unrealistic expectations beyond the paper's intended scope.

“The dominant failure mode for DIAGPaper (71.4%) involves weaknesses that are factually plausible but judged invalid by annotators because they impose unrealistic expectations beyond the paper's intended scope.”

For the case-study paper, 36 initial weaknesses generated by 12 reviewer roles were reduced to 14 high-confidence ones.

“For this paper, 36 initial weaknesses generated by 12 reviewer roles were reduced to 14 high-confidence ones.”

In a case study, an initial reviewer critique regarding missing alternative pre-training objectives was challenged by the rebuttal agent based on the paper's explicit justification, experimental validation, and a clarification that the critique conflates non-exhaustiveness with poor design.

“an initial reviewer critique regarding missing alternative pre-training objectives was challenged by the rebuttal agent based on the paper's explicit justification, experimental validation, and a clarification that the critique conflates non-exhaustiveness with poor design.”

Top-performing systems-GPT-4o and multi-agent approaches (Reviewer2, AgentReview, MARG, and DIAGPaper)-exhibit low overlap with invalid reviews because they tend to generate valid, well-grounded weaknesses that do not align with invalid human reviews.

“We observe that topperforming systems-GPT-4o and multi-agent approaches (Reviewer2, AgentReview, MARG, and DIAGPaper)-exhibit low overlap with these invalid reviews for a desirable reason: they tend to generate valid, well-grounded weaknesses that do not align with invalid human reviews.”

Multi-agent versions built on open-source LLMs (Llama, Mistral, and Qwen) achieve scores above 47.21%, approaching or matching the performance of the closed-source single-agent GPT-4o (47.43%).

“Notably, multi-agent versions built on open-source LLMs (Llama, Mistral, and Qwen) achieve scores above 47.21%, approaching or matching the performance of the closed-source single-agent GPT-4o (47.43%).”

DIAGPaper occasionally anchors a weakness to an incorrect retrieved passage, leading to mismatches between the critique and the cited context.

“DIAGPaper occasionally anchors the weakness to an incorrect retrieved passage, leading to mismatches between the critique and the cited context.”

Removing the author rebuttal mechanism leads to the largest performance drop.

“removing the author rebuttal mechanism leads to the largest performance drop”

Systems that avoid overlap with known invalid weaknesses also tend to perform poorly on ReviewCritique(Valid), as their outputs lack coherence and grounding.

“We observe that such systems, while avoiding overlap with known invalid weaknesses, also tend to perform poorly on ReviewCritique(Valid), as their outputs lack coherence and grounding.”

DIAGPaper achieves state-of-the-art performance across semantic recall, precision, F1, and Specificity on AAAR, ReviewCritique(Valid), and ReviewCritique(Invalid).

“First, on AAAR, ReviewCritique(Valid), and Re-viewCritique(Invalid), DIAGPaper achieves stateof-the-art performance across semantic recall, precision, F1, and Specificity, demonstrating both strong alignment with valid human reviews and the ability to produce paper-specific weaknesses.”

The Rebuttal process removes approximately 40%-60% of initially identified weaknesses.

“Empirically, this process removes ∼ 40%-60% of initially identified weaknesses.”

Multi-agentization of a single LLM under the DIAGPaper framework yields consistent performance gains across all tested LLMs (GPT-4o, Llama, Mistral, and Qwen).

“Across all tested LLMs-including GPT-4o, Llama, Mistral, and Qwen-multi-agentization yields consistent performance gains.”

All components of DIAGPaper contribute positively to overall performance.

“We observe that all components contribute positively to overall performance.”

DIAGPaper's Realism score is lower than some baselines.

“However, its Realism score is lower than some baselines.”

DIAGPaper can elevate open-source LLMs to performance levels approaching those of closed-source models such as GPT-4o.

“Notably, DIAGPaper can elevate open-source LLMs to performance levels approaching those of closed-source models such as GPT-4o.”

General-purpose LLMs mostly produce noisy and unseen outputs that fail to meaningfully correspond to either valid or invalid weaknesses.

“because they mostly produce noisy and unseen outputs that fail to meaningfully correspond to either valid or invalid weaknesses.”

The gap in Realism primarily arises from DIAGPaper's tendency to generate overly strict but factually grounded critiques.

“Further analysis reveals that this gap primarily arises from DIAGPaper's tendency to generate overly strict but factually grounded critiques.”

Lower semantic overlap with ReviewCritique(Invalid) indicates better performance.

“Importantly, since the ReviewCritique(Invalid) split consists of human-written but invalid weakness statements, lower semantic overlap (e.g., lower F 1 invalid ) indicates better performance.”

The DIAGPaper framework has strong cross-LLM generalization.

“These results highlight the framework's strong cross-LLM generalization”

Explicit reviewer-author interaction is critical for filtering invalid or weakly supported critiques.

“indicating that explicit reviewer-author interaction is critical for filtering invalid or weakly supported critiques.”

By focusing on roles rather than explicitly modeling and controlling such criteria, prior systems fail to capture how expert reviewers selectively assess different intellectual aspects of a paper.

“By focusing on roles rather than explicitly modeling and controlling such criteria, prior systems fail to capture how expert reviewers selectively assess different intellectual aspects of a paper.”

• AssumptionRests on an assumption: Human review diversity stems from reviewers' heterogeneous backgrounds and expertise, which lead them to apply different criteria when evaluating the same submission.

DIAGPaper has reliable state-of-the-art performance.

“confirms the reliable state-of-the-art performance of DIAGPaper.”

F1_inv is effective in distinguishing genuinely high-quality review behavior from artifactually low overlap caused by noisy or non-review-like outputs.

“This result validates the effectiveness of F 1 inv in distinguishing genuinely high-quality review behavior from artifactually low overlap caused by noisy or non-review-like outputs”

The naive metric of lower F1_invalid is brittle because a poor-quality LLM may generate incoherent or previously unseen invalid statements that do not semantically match any annotated invalid weaknesses, thereby achieving an artificially low F1_invalid despite producing outputs that are not review-like or meaningful.

“However, this naive metric is brittle: a poor-quality LLM may generate incoherent or previously unseen invalid statements that do not semantically match any annotated invalid weaknesses, thereby achieving an artificially low F 1 invalid despite producing outputs that are not review-like or meaningful.”

The threshold τ = 0.4 used in the rebuttal is empirically calibrated using large-scale review-revision pairs, leveraging submission editing behavior as a proxy signal for author acceptance of reviewer-identified weaknesses.

“The threshold τ = 0.4 used in the rebuttal (Section 3.2) is empirically calibrated using largescale review-revision pairs, leveraging submission editing behavior as a proxy signal for author acceptance of reviewer-identified weaknesses.”