AgentReview: Exploring Peer Review Dynamics with LLM Agents

“We quantify the alignment between reviews and meta-reviews using BERTScore [31] and sentence embedding similarity [32] in Table 2, and measure the agreement of final decisions between baseline and each setting in Figure 5.”

“Social Influence Theory [27] suggests that individuals in a group tend to revise their beliefs towards a common viewpoint.”

“We quantify the alignment between reviews and meta-reviews using BERTScore [31] and sentence embedding similarity [32] in Table 2, and measure the agreement of final decisions between baseline and each setting in Figure 5.”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“3.1.2); • Groupthink and Echo Chamber Effects [22, 23].”

“This can lead to a 0.17 drop in ratings among biased reviewers and cause a spillover effect , influencing the judgments of unbiased reviewers and leading to a 0.25 decrease in ratings; • Authority Bias and Halo Effects [24].”

“Our findings are as follows, which could inspire future design of peer review systems: • Social Influence [20].”

“Our study aligns with recent works in agent-based modeling (ABM) [12, 58–61] that leverage the capabilities of LLM agents to simulate realistic environments for scientific research.”

“Altruism Fatigue & Peer Effect [21] Paper review is typically unpaid and time-consuming [28], requiring substantial time investment beyond reviewers’ regular professional duties.”

“Our A GENT R EVIEW mirrors real-world academic review practices to ensure the authenticity and relevance of our findings.”

“Following [17], we ask LLM agents to generate reviews that comprise four sections, including significance and novelty , potential reasons for acceptance , potential reasons for rejection , and suggestions for improvement .”

“Research has also delved into the operational aspects, such as reviewer assignments [39, 30, 40] and author rebuttals [41], identifying areas for improvement in transparency, fairness, and accountability [2].”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“Despite its importance, peer reviews often face several challenges, such as biases [1], variable review quality [1], unclear reviewer motives [2], and imperfect review mechanism [3], exacerbated by the ever-growing number of submissions.”

“…knowledgeable reviewers are 24% more likely to mention insufficient discussion of limitations , whereas expert reviewers not only address these basic aspects but also provide 6.8 % more critiques on experimental validation, resulting in more concrete and beneficial feedback for improving the paper.”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“3.3); • Anchoring Bias [25].”

“Conflict Theory [29] states that societal interactions are often driven by conflict rather than consensus.”

“Table 6 presents the generated review, rebuttal, and meta-review for the paper “Image as Set of Points” [64] in our simulation.”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“Recent machine learning conferences have shown an increase in reviews suspected to be AI-generated [63].”

“The rise of open science and preprint platforms has further complicated these systems, which may disclose author identities under double-blind policies [4].”

“These capabilities have facilitated their adoption as autonomous agents that interact with each other to complete tasks [58, 59].”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“3.1.2); • Groupthink and Echo Chamber Effects [22, 23].”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“Several studies have scrutinized various challenges within peer review, such as bias [1, 35, 36], conflict of interests [37], and the broader issues of review quality and fairness [1, 37, 38].”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“Research has also delved into the operational aspects, such as reviewer assignments [39, 30, 40] and author rebuttals [41], identifying areas for improvement in transparency, fairness, and accountability [2].”

“The inherent variability and arbitrariness in human peer reviews [26] add complexity to direct comparisons between simulated and real outcomes.”

“Large Language Models (LLMs) such as GPT-4 [8], Claude 3 [42], and Gemini [43] have not only demonstrated sophisticated language understanding [44–46], reasoning [47–49] and generation skills [50–55], but also exhibit planning, collaboration, and competitive behaviors [14, 56, 57].”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“Efforts to mitigate these problems have focused on enhancing fairness [2], reducing biases among novice reviewers [1], calibrating noisy peer review ratings [5], and refining mechanisms for paper assignment and reviewer expertise matching [6, 7].”

“Caution about Use of LLMs.”

“Notable exceptions occur under the irresponsible and malicious settings, where the ratings exhibit a bimodal distribution with peaks at [5 , 5 .”

“Research has also delved into the operational aspects, such as reviewer assignments [39, 30, 40] and author rebuttals [41], identifying areas for improvement in transparency, fairness, and accountability [2].”

“While A GENT R EVIEW uses LLMs to generate paper reviews, there are ethical concerns regarding their use in actual peer review processes [62].”

“Recent conferences have increasingly permitted the release of preprints, potentially impacting paper acceptance [33].”

“Several studies have scrutinized various challenges within peer review, such as bias [1, 35, 36], conflict of interests [37], and the broader issues of review quality and fairness [1, 37, 38].”

“Peer review serves as the backbone of academic research, ensuring the integrity and quality of published work [34].”

“Efforts to mitigate these problems have focused on enhancing fairness [2], reducing biases among novice reviewers [1], calibrating noisy peer review ratings [5], and refining mechanisms for paper assignment and reviewer expertise matching [6, 7].”

“Research has also delved into the operational aspects, such as reviewer assignments [39, 30, 40] and author rebuttals [41], identifying areas for improvement in transparency, fairness, and accountability [2].”

“As shown in Figure 1, A GENT R EVIEW is built upon the capabilities of LLMs to perform realistic simulations of societal environments [11–15] and provide high-quality feedback on academic literature comparable to or exceeds human levels [16–19].”

“Several studies have scrutinized various challenges within peer review, such as bias [1, 35, 36], conflict of interests [37], and the broader issues of review quality and fairness [1, 37, 38].”

“Despite its importance, peer reviews often face several challenges, such as biases [1], variable review quality [1], unclear reviewer motives [2], and imperfect review mechanism [3], exacerbated by the ever-growing number of submissions.”

“Several studies have scrutinized various challenges within peer review, such as bias [1, 35, 36], conflict of interests [37], and the broader issues of review quality and fairness [1, 37, 38].”

“Altruism Fatigue & Peer Effect [21] Paper review is typically unpaid and time-consuming [28], requiring substantial time investment beyond reviewers’ regular professional duties.”

“We introduce A GENT R EVIEW , the first framework that integrates large language models (LLMs) [8, 9] with agent-based modeling [10] to simulate the peer review process (Sec.”

“We consider two types of baselines: (1) Prompt-based baselines including AI Scien-tist (Chris et al., 2024) and AgentReview (Jin et al., 2024a) implemented with various back-bone models (GPT-o1-2024-12-17, Claude-3.”

“Single-aspect measurements are insufficient for interactive evaluation, so many studies adopt evaluated objectives that range from specific actions to hybrid actions [110], or from single-turn interactions to multi-turn dialogues [10].”

“This evaluation mirrors the rigorous peer-review process typical at prestigious venues like ICLR and NeurIPS [18], where an expert review agent systematically examines each paper pair through three fundamental scientific dimensions: • (1) innovation and novelty of research contributions, • (2)…”

“Despite these limitations, we hope that the datasets and library we have constructed will facilitate broader research and applications in multimodal time series analysis, such as benchmark (27) and agent (28).”

“LLM-powered agents have demonstrated impressive capabilities in generating research ideas [16, 3, 28, 17], conducting experiments and analyses [11, 34, 4, 40, 13, 41, 15], drafting scientific articles [25, 35, 38, 1] and automated review [36, 14, 32, 6, 42].”