AgentReview: Exploring Peer Review Dynamics with LLM Agents

“We quantify the alignment between reviews and meta-reviews using BERTScore [31] and sentence embedding similarity [32] in Table 2, and measure the agreement of final decisions between baseline and each setting in Figure 5.”

“Social Influence Theory [27] suggests that individuals in a group tend to revise their beliefs towards a common viewpoint.”

“We quantify the alignment between reviews and meta-reviews using BERTScore [31] and sentence embedding similarity [32] in Table 2, and measure the agreement of final decisions between baseline and each setting in Figure 5.”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“3.1.2); • Groupthink and Echo Chamber Effects [22, 23].”

“This can lead to a 0.17 drop in ratings among biased reviewers and cause a spillover effect , influencing the judgments of unbiased reviewers and leading to a 0.25 decrease in ratings; • Authority Bias and Halo Effects [24].”

“Our findings are as follows, which could inspire future design of peer review systems: • Social Influence [20].”

“Our study aligns with recent works in agent-based modeling (ABM) [12, 58–61] that leverage the capabilities of LLM agents to simulate realistic environments for scientific research.”

“Altruism Fatigue & Peer Effect [21] Paper review is typically unpaid and time-consuming [28], requiring substantial time investment beyond reviewers’ regular professional duties.”

“Our A GENT R EVIEW mirrors real-world academic review practices to ensure the authenticity and relevance of our findings.”

“We consider two types of baselines: (1) Prompt-based baselines including AI Scien-tist (Chris et al., 2024) and AgentReview (Jin et al., 2024a) implemented with various back-bone models (GPT-o1-2024-12-17, Claude-3.”

“Single-aspect measurements are insufficient for interactive evaluation, so many studies adopt evaluated objectives that range from specific actions to hybrid actions [110], or from single-turn interactions to multi-turn dialogues [10].”

“This evaluation mirrors the rigorous peer-review process typical at prestigious venues like ICLR and NeurIPS [18], where an expert review agent systematically examines each paper pair through three fundamental scientific dimensions: • (1) innovation and novelty of research contributions, • (2)…”

“Despite these limitations, we hope that the datasets and library we have constructed will facilitate broader research and applications in multimodal time series analysis, such as benchmark (27) and agent (28).”

“LLM-powered agents have demonstrated impressive capabilities in generating research ideas [16, 3, 28, 17], conducting experiments and analyses [11, 34, 4, 40, 13, 41, 15], drafting scientific articles [25, 35, 38, 1] and automated review [36, 14, 32, 6, 42].”

“Recent advancements in Large Language Models (LLMs) such as GPT-4 [20], LLaMA [42], Mistral Large 2 [17], and Gemini [43] have established new benchmarks in natural language processing (NLP), offering enhanced language comprehension and reasoning [44–48].”

“Many current benchmarks rely on subjective evaluations, such as LLM-based judging [Jin et al., 2024] or peer-review simulations [Russo Latona et al., 2024], which often suffer from inconsistency or misalignment with actual code execution.”

“Jin et al. (2024) introduce AgentReview, a framework that models the conference peer review process using LLM agents.”

“…et al., 2024), agentic frameworks (D’Arcy et al., 2024), and comparative analyses of LLM-generated reviews (Liu & Shah, 2023), along with broader explorations of LLMs’ roles and limitations in peer review (Jin et al., 2024; Santu et al., 2024; Sukpanichnant et al., 2024; Liang et al., 2024b).”

“Existing LLM-assisted peer review efforts largely replicate traditional workflows, framing evaluation as absolute scoring followed by aggregation [18, 27, 49, 55, 61].”

“Meanwhile, LLM-assisted peer review automation has attracted increasing attention in the research community, with several studies developing frameworks based on prompt engineering, multi-agent architectures, and knowledge enhancement methods [4]–[6].”

“Jin et al. [15] present AgentReview, a simulation framework where LLMs emulate the roles of reviewers, authors, and area chairs to analyze biases and decision variability in scientific peer review.”

“For example, AgentReview (Jin et al., 2024) uses LLM agents to investigate review dynamics and latent biases; ReviewAgents (Gao et al., 2025) proposes a multi-role framework to emulate human reasoning processes; and MARG (D’Arcy et al., 2024) distributes the paper among specialized agents to…”

“The rapid advancement of large language models (LLMs) (Jin et al., 2024b; Ouyang et al., 2025; Cheng et al., 2025) has led to the proliferation of “model arenas”—platforms designed to compare and evaluate multiple models, identifying their relative strengths and weaknesses (Chiang et al., 2024).”

“…emergence and rapid development of large language models (LLMs), including ChatGPT and its contemporaries, new approaches to improve the peer review process are being explored (Couto et al., 2024; Fecher et al., 2023; Hosseini & Horbach, 2023; Li et al., 2024a; Thelwall, 2024a; Zhao et al., 2024).”

“These advances (Jin et al. 2024; Wu et al. 2023) reveal LLMs’ potential to reproduce complex human behaviors in controlled, repeatable settings.”

“Sensor count is assigned as follows: • Small ( ≤ 30 m 2 ): 1 sensor • Medium ( 30 < area ≤ 60 m 2 ): 2 sensors • Large ( > 60 m 2 ): 3 sensors 3.”

“Regarding authorship manipulation, tactics include adding prestigious coauthors or inflating citations to top-tier and eminent scholars, leveraging the model’s tendency to associate prestige with quality 12 .”

“Current training for AI Scientists is mainly targeting literature search [3–5, 35 ] and Recent studies have explored leveraging LLMs to evaluate academic manuscripts, predict review scores, and generate feedback [42–47].”

“Current systems go beyond single-prompt generation to adopt multi-agent designs that emulate panel workflows (Jin et al., 2024a; Durante et al., 2024) and reinforcement learning (Ziegler et al., 2020; Christiano et al., 2023) to align outputs with nuanced human preferences.”

“Further researches on software development, gaming, employment consultant and social science area also validate novel application values of LLM-based agent system on real world simulation (Qian et al. 2023; Xu et al. 2023; Tang et al. 2024; Xie et al. 2024; Jin et al. 2024).”

“To this end, Jin et al. [140] introduced an LLM-based agent peer review simulation framework.”

“Modern Large Language Models have enabled automated or semi-automated pipelines for academic reviewing [10, 15, 29], already adopted in up to 15.8% of AI-conference reviews [13, 14].”

“This paradigm has been expanded to diverse applications to judge various types of candidates, including paper quality assessing (Jin et al., 2024), document relevance measurement (Gao et al., 2023; Rahmani et al., 2024), and reasoning trace correctness verification (Zhang et al.)”

“Agentic systems have also been explored for automated paper reviewing [25, 73].”

“These results suggest that automated scientific discovery is becoming practically feasible and that LLM-based agents are approaching a level of functional completeness required for autonomous research (Jin et al., 2024; Sahu et al., 2025; Ajith et al., 2024; Zhang et al., 2025b, 2026).”

“User simulators generate synthetic profiles to simulate user-system interactions, enabling the offline evaluation of RSs. Earlier methods primarily used rule-based (Wei et al., 2018) or reinforcement learning approaches (Ie et al., 2019; Zhao et al., 2023).”

“To mitigate the computational burden of explicit counterfactual interventions, two efficiency-oriented approximations are explored that that estimate structural importance while preserving alignment with task utility.”

“The academic literature on peer review has revealed the presence of affiliation bias in open peer review [6,7,38].”

“Later systems improved specificity through fine-tuning and structured pipelines, and most recently multi-agent approaches that assign roles to reduce generic feedback and increase comment quality (Zhu et al., 2025; Jin et al., 2024”

“…in this application, and researchers are extensively investigating the feasibility of LLM-assisted peer review (e.g., Liu and Shah , 2023; Liang et al. , 2024; Roberson , 2023; Du et al. , 2024) and are developing trial tools (e.g., D’Arcy et al. , 2024; Jin et al. , 2024; Sun et al. , 2024).”

“Since LLMs are predominantly trained on web-based corpora, they struggle to emulate roles with insufficient representation online [329] and often produce conversations lacking diversity [268].”

“AgentReview (Jin et al. 2024) further explores this direction by prompting LLMs to assume diverse reviewer personalities and decision strategies.”

“In real-world applications such as robotics, virtual agents, and interactive assistants (Jiang et al., 2024b; Jin et al., 2024; Liu et al., 2025), effective planning demands not only long-horizon decision-making but also strict adherence to logical consistency and domain-specific rules.”

“Prompt-Based Multi-Agent Frameworks: AI-Scientist and AgentReview (Jin et al., 2024).”

“It is Inconclusive when evidence is insufficient, unavailable, or too weakly aligned to justify a stronger label.”

“Review responsibility can be conceptualized along three interdependent dimensions [2], [3]: commitment (the extent of substantive engagement with the manuscript), constructive intention (a fair-minded desire to improve the work) and domain knowledgeability (sufficient expertise to make an informed…”

“Meanwhile, AgentReview (Jin et al., 2024) employed LLM agents to simulate peer reviews, rebuttals, and committee discussions, offering insights into the dynamics of academic decision-making.”

“Previous works have demonstrated the usefulness of the LLM-based review feedback (Gao, Brantley, and Joachims 2024; Jin et al. 2024; Lu et al. 2024).”

“Submitting such materials to third-party LLM systems raises confidentiality risks [25, 65], including data retention, logging, unauthorized reuse, or unintended exposure of novel findings or confidential data.”

“When agents engage in self-correction or peer-review (Shinn et al., 2023; Jin et al., 2024), the assigned role functions as a rigid cognitive prior that skews agents’ judgment.”

“Automated evaluation systems could enable rapid iteration and improvement (Jin et al., 2024), but creating trustworthy automated reviewers for scientific papers remains an open problem.”

“Despite their growing role, it remains challenging to assess the significance or scientific plausibility of hypotheses generated by LLMs (Lin et al., 2025), especially when another LLM is used as a judge.”

“It is worth noting that alternative ranking evaluation measures, such as Spearman’s rank correlation coefficient [52] and Kendall’s tau [53, 54], exist.”

“Stanford Agent Review [21]: Multi-agent system simulating reviewer deliberation.”

“This substantial cost has driven a transformation toward integrating LLMs to improve reviewing efficiency [9, 19, 37, 56].”

“Researchers have developed benchmarks and methods for academic peer review [Jin et al., 2024, Li et al., 2025] and rebuttals [He et al., 2026, Ma et al., 2026, Zhang et al., 2025] to simulate iterative scientific communication.”

“Jin et al. (2024) introduce an LLM-based peer review simulation framework AGENTREVIEW that addresses the complexities and privacy concerns of traditional peer review analysis, revealing significant biases in paper decisions.”

“For example, evaluations of ICLR 2024 submissions found that LLM reviewers hallucinated details, favored verbose papers, and exhibited institution-based biases [22].”

“Despite extensive research in scientometrics–scrutinizing the creation [2], diffusion [3], and association [4] of academic knowledge–many studies focus on limited timespans [5, 6], venues [7, 8], or particular areas like natural language processing [3, 9, 10] and human-computer interaction [11].”

“Even when equipped with the exact same Google Search tool, ScholarPeer outperforms AI Scientist v2 and AgentReview with search access by significant margins.”

“Current agents either write the paper for the author [2, 34] or grade it at the door [18, 20, 21, 27, 33].”

“A key limitation is that existing LLM-based systems predominantly generate descriptive rather than prescriptive feedback: they can identify issues, but often fail to specify how those issues should be addressed [Jin et al., 2024, Gao et al., 2025, Wu et al., 2026b].”

“Jin et al. [16] introduced AgentReview , an LLM-based peer-review simulation framework that models reviewer and decision-making roles and enables controlled analysis of latent behavioral factors and their effects on reviews and paper decisions.”

“Jin et al. (2024) proposed the Agent Review framework, which applies a multi-agent system to the analysis of peer-review mechanisms.”

“For the Artificial Intelligence and Machine Learning category, we collect five years’ ICLR publications from PeerRead (Kang et al., 2018a) and AgentReview (Jin et al., 2024).”

“Various authors have since suggested improvements using thinking processes or agentic approaches to the task (Jin et al., 2024; Zhu et al., 2025; Idahl and Ahmadi, 2025; Bougie and Watan-abe, 2025; Sahu et al., 2025).”

“These approaches introduce multiple LLM agents with assigned roles to simulate the collaborative nature of real conference peer review, thereby generating more specific and comprehensive feedback (Jin et al., 2024; Chamoun et al., 2024; Yu et al., 2024; D’Arcy et al., 2024; Wang et al., 2024b).”

“Studies document that 6.5–16.9% of AI conference reviews show substantial LLM involvement [16], and that, in LLM-based agent simulations of peer review, up to 37.1% of variance in decisions is attributable to modeled reviewer biases [12].”

“Frontier LLMs such as GPT 13 , Gemini 14 , Claude, Grok, Qwen 15 , and Kimi 16 , have demonstrated remarkable capabilities in natural language processing tasks 17–20 , including summarization, sentiment analysis, and content generation, making them suitable for evaluating complex scientific texts.”

“AgentReview, a novel LLM ‐ based simulation framework to analyze peer review dynamics, revealed a 37.1% decision variance due to reviewer biases while addressing privacy concerns and latent factors in the process [3].”

“Reference [28] introduced AGENTREVIEW, a simulation framework leveraging LLMs to systematically examine factors influencing peer review outcomes.”

“, hospital settings where medical staff and patients engage in treatment simulations (Li et al. 2024; Schmidgall et al. 2024), news interviews (Lu et al. 2024), and academic peer review (Jin et al. 2024).”

“…tasks, including review and meta-review generation [Shen et al., 2021, Zhang et al., 2022, Yuan et al., 2022, Wu et al., 2022, D’Arcy et al., 2023, Jin et al., 2024, Gao et al., 2024, Zhou et al., 2024, Weng et al., 2024, Zhu et al., 2025]; and (3) analytical tasks related to review content, like…”

“To enhance feedback comprehensiveness and consistency, systems like AgentReview [5] and MARG [17] simulate multi-agent re-viewer deliberations to mirror real-world committee dynamics.”

“With increasing paper submissions, conferences and reviewers have explored the use of LLMs to assist with peer review (Jin et al., 2024; Demetrio et al., 2025; Thakkar et al., 2026; Zhu et al., 2025; Huang et al., 2025).”

“Camera-ready formatting, author-identifying text, source-specific artifacts, and the natural majority-reject rate can all leak the label (Jin et al., 2024).”

“MetaGPT, ChatDev, AutoGen, and SWE-agent supply the orchestration primitives we reuse [13, 31, 47, 50], while LLM-as-a-judge, MARG, and AgentReview inform our reviewer-driven re-experimentation but treat review as terminal rather than a gate that can demand new substrate experiments [54, 8, 18].”

“…therefore assess our method under our model assumptions with a larger LLM-simulated review study, following precedent in prior work on review tools (Jin et al., 2024; Erturk and Durmaz, 2026) and evidence that LLM simulations can often replicate aggregate-level human population dynamics…”

“In addition, authors sometimes provide generic or irrelevant limitations that fail to convey the study’s real constraints [7], or use them as a form of hedging, presenting minor weaknesses while avoiding deeper flaws to reduce the risk of rejection [25].”

“Other work simulates peer review dynamics using LLM agents to study latent factors and emergent behaviors in review processes [18].”

“Recent systems use LLM agents for idea generation [1, 19], experimentation [2], manuscript preparation [6], and peer review [3, 20].”

“Subsequent studies improve review generation by fine-tuning (Wei et al., 2023; Gao et al., 2024; Idahl & Ahmadi, 2025; Yu et al., 2024b; Zhu et al., 2025) or multi-agent collaboration (Jin et al., 2024; Lu et al., 2025; D’Arcy et al., 2024; Taechoyotin et al., 2024).”

“Recent work has used LLM agents to simulate town life [8], resource dilemmas [9], peer review [10], opinion dynamics [11], and open-ended social interaction [12].”