AgentReview: Exploring Peer Review Dynamics with LLM Agents

“We quantify the alignment between reviews and meta-reviews using BERTScore [31] and sentence embedding similarity [32] in Table 2, and measure the agreement of final decisions between baseline and each setting in Figure 5.”

“Social Influence Theory [27] suggests that individuals in a group tend to revise their beliefs towards a common viewpoint.”

“We quantify the alignment between reviews and meta-reviews using BERTScore [31] and sentence embedding similarity [32] in Table 2, and measure the agreement of final decisions between baseline and each setting in Figure 5.”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“3.1.2); • Groupthink and Echo Chamber Effects [22, 23].”

“This can lead to a 0.17 drop in ratings among biased reviewers and cause a spillover effect , influencing the judgments of unbiased reviewers and leading to a 0.25 decrease in ratings; • Authority Bias and Halo Effects [24].”

“Our findings are as follows, which could inspire future design of peer review systems: • Social Influence [20].”

“Our study aligns with recent works in agent-based modeling (ABM) [12, 58–61] that leverage the capabilities of LLM agents to simulate realistic environments for scientific research.”

“Altruism Fatigue & Peer Effect [21] Paper review is typically unpaid and time-consuming [28], requiring substantial time investment beyond reviewers’ regular professional duties.”

“Our A GENT R EVIEW mirrors real-world academic review practices to ensure the authenticity and relevance of our findings.”

“PeerRead enabled data-driven study of 14.7K drafts and 10.7K expert reviews [5]; later work uses LLM agents to simulate reviewer behavior and decisions [4].”

“21,22,23,24 Much work has been conducted studying the quality and nature of reviews generated by such automated review architectures.”

“AgentReview (Jin et al., 2024) is closest to our work in simulating the multi-party review process with LLM agents.”

“[18], [19] Embedding – – AgentReview / Reviewer2 [6], [11] – – – MARG [10] Prompt/Keyword Centralized – ERC-8004 Draft [8] – On-chain Basic ◦ Ours Taxonomy On-chain Hybrid ✓ ✓ : core focus; ◦ : partial support; –: out of scope.”

“Training-free methods [29–34] rely on prompting powerful closed-source models within agentic frameworks; however, these models are not inherently optimized for academic rigor.”

“This “score-then-select” design is intended to reduce anchoring compared to direct pairwise comparison or multi-candidate ranking in a single context window [24], echoing the concern raised in AgentReview that early reviewer impressions can dominate later evidence and anchor decisions [10].”

“AGENTREVIEW [4] simulates three roles—reviewers, authors, and area chairs—in a five-stage interaction process, expanding review dimensions and capturing more factors that influence decisions.”

“A major line of research evaluates the intrinsic quality of a paper by using LLM-based review agents to simulate peer review and produce automated pointwise scores (Jin et al. 2024; Lu et al. 2024).”

“As technology evolves [42, 43] 2 , adoption accelerates and baseline use rises, the absolute magnitudes of our effects may evolve, but the underlying mechanisms we document, including expanded access to critique and subsequent tool adoption, offer a timely baseline for understanding how human-AI…”

“This suggests that reproducibility assessment may be tractable as a document-level reasoning task rather than requiring full experimental replication [24, 25].”

“Systems such as AgentReview [19], SWIF2T [20], MARG [21], and SEA [3] assign distinct roles to the models and generate feedback that is more precise and more similar to human reviewing.”

“Aiming at the above problems, inspired by references [7-8], Joint prompt engineering [9] proposed a frontier topics mining method via AI-agent (D-Agents).”

“Additionally, multi-agent frameworks in related domains often emphasize the collaboration between agents with different roles, while the design of agent roles and collaboration mechanisms in the medical field needs to be tailored to the characteristics of clinical tasks [14–18].”

“Simulation frameworks such as AGENTREVIEW [27] model peer review as a multi-agent system and demonstrate that reviewer biases alone can lead to substantial variability in editorial decisions.”

“Other studies examined score calibration and related interventions that improve local consistency [10].”

“Author rebut-tals are omitted from the context, as our focus is on peer-to-peer reviewer interaction and prior work has shown that rebuttals play a limited role in LLM peer review simulations (Jin et al., 2024).”

“-References [7] and [13], both titled "Overview of Intelligent Vehicle Multi target Detection Technology Based on Multi sensor Fusion", were published in 2021 in the same journal (Journal of Automotive Safety and Energy), and were identified as duplicate references.”

“Meanwhile, Jin et al. [17] developed a peer-review simulation framework that leverages LLMs, effectively decoupling the influences of multiple potential factors and offering valuable insights for improving the design of peer-review mechanisms.”

“To address LTR tasks, [Szummer and Yilmaz, 2011; Jin et al. , 2024c; Jin et al. , 2024b; Jin et al. , 2024a] proposes a parameter learning-based approach, which has shown that over-parameterization can achieve superior performance on test data under certain conditions in the ”interpolation regime”…”