CoChat
Sign Up
PreprintCited by 3Ethics and Social Impacts of AIAcademic Writing and PublishingTopic Modeling

When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review

Open access
Zhu, Changjia, Xiong, Junjie, Ma, Renkai, Lu, Zhicong, Liu, Yao, Li, Lingyao·2025·arXiv
DOI 10.48550/arxiv.2509.09912arXiv 2509.09912
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer review is the cornerstone of academic publishing, yet the process is increasingly strained by rising submission volumes, reviewer overload, and expertise mismatches. Large language models (LLMs) are now being used as "reviewer aids," raising concerns about their fairness, consistency, and robustness against indirect prompt injection attacks. This paper presents a systematic evaluation of LLMs as academic reviewers. Using a curated dataset of 1,441 papers from ICLR 2023 and NeurIPS 2022, we evaluate GPT-5-mini against human reviewers across ratings, strengths, and weaknesses. The evaluation employs structured prompting with reference paper calibration, topic modeling, and similarity analysis to compare review content. We further embed covert instructions into PDF submissions to assess LLMs' susceptibility to prompt injection. Our findings show that LLMs consistently inflate ratings for weaker papers while aligning more closely with human judgments on stronger contributions. Moreover, while overarching malicious prompts induce only minor shifts in topical focus, explicitly field-specific instructions successfully manipulate specific aspects of LLM-generated reviews. This study underscores both the promises and perils of integrating LLMs into peer review and points to the importance of designing safeguards that ensure integrity and trust in future review processes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2509.09912

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs often underperform in identifying substantive weaknesses and show limited sensitivity to paper quality.

    “Although LLMs can provide useful paper-level feedback, they often underperform in identifying substantive weaknesses and show limited sensitivity to paper quality (Liang et al., 2024; Li et al., 2025a; Zhu et al., 2025).”

    From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Zhu et al., 2025

    Section 4.1 (p. 11) supports limited sensitivity to paper quality: LLM ratings aligned less well with human judgments for rejected papers, and 479 of 500 rejected papers were rated as acceptable. Section 4.2 (pp. 14–15) finds only moderate overlap with human-identified weaknesses on individual papers, but it does not establish that LLMs generally underperform at identifying substantive weaknesses; they emphasize some technical weaknesses more than human reviewers do. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 77

  1. [1][1] ACL Rolling Review. 2025. Authors Guidelines. https://aclrollingreview.org/authors. -
  2. [2][2] Association for Computing Machinery. 2025. Peer Review Policy FAQ. https://www.acm.org/publications/policies/peer-review-faq. -
  3. [3][3] Md Ahsan Ayub and Subhabrata Majumdar. 2024. Embedding-based classifiers can detect prompt injection attacks. arXiv preprint arXiv:2410.22284 (2024). - · arXiv
  4. [4][4] Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. 2019. Toward algorithmic accountability in public services: A qualitative study of affected community perspectives on algorithmic decision-making in child welfare services. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1-12. -
  5. [5][5] Amy S Bruckman, Casey Fiesler, Jeff Hancock, and Cosmin Munteanu. 2017. CSCW research ethics town hall: Working towards community norms. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing . 113-115. -
  6. [6][6] Ángel Alexander Cabrera, Adam Perer, and Jason I Hong. 2023. Improving human-AI collaboration with descriptions of AI behavior. Proceedings of the ACM on Human-Computer Interaction 7 (2023), 1-21. -
  7. [7][7] Alexander J Carroll and Joshua Borycz. 2024. Integrating large language models and generative artificial intelligence tools into information literacy instruction. The Journal of Academic Librarianship 50, 4 (2024), 102899. -
  8. [8][8] Shiping Chen, Duncan Brumby, and Anna Cox. 2025. Envisioning the Future of Peer Review: Investigating LLM-Assisted Reviewing Using ChatGPT as a Case Study. In Proceedings of the 4th Annual Symposium on Human-Computer Interaction for Work . 1-18. -
  9. [9][9] Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. Struq: Defending against prompt injection with structured queries. In 34rd USENIX Security Symposium (USENIX Security 25) . -
  10. [10][10] Yulin Chen, Haoran Li, Zihao Zheng, Yangqiu Song, Dekai Wu, and Bryan Hooi. 2024. Defense against prompt injection attack by leveraging attack techniques. arXiv preprint arXiv:2411.00459 (2024). - · arXiv
  11. [11][11] CHI 2025 conference. 2024. CHI 2025 - Papers Track, post-review report (Round 1). https://chi2025.acm.org/chi-2025-papers-track-post-reviewreport-round-1/. -
  12. [12][12] CHI 2026 Conference. 2025. Guide to Reviewing Papers. https://chi2026.acm.org/guide-to-reviewing-papers/. -
  13. [13][13] Valdemar Danry, Pat Pataranutaporn, Matthew Groh, and Ziv Epstein. 2025. Deceptive explanations by large language models lead people to change their beliefs about misinformation more often than honest explanations. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems . 1-31. -
  14. [14][14] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. Advances in Neural Information Processing Systems 37 (2024), 82895-82920. -
  15. [15][15] Rebecca Dorn, Lee Kezar, Fred Morstatter, and Kristina Lerman. 2024. Harmful speech detection by language models exhibits gender-queer dialect bias. In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization . 1-12. -
  16. [16][16] [Eric Price. 2014. The NIPS experiment. https://blog.mrtz.org/2014/12/15/the-nips-experiment.html.](https://blog.mrtz.org/2014/12/15/the-nips-experiment.html) -
  17. [17][17] Lizhou Fan, Lingyao Li, Zihui Ma, Sanggyu Lee, Huizi Yu, and Libby Hemphill. 2024. A bibliometric review of large language models research from 2017 to 2023. ACM Transactions on Intelligent Systems and Technology 15, 5 (2024), 1-25. -
  18. [18][18] Christopher Flathmann, Wen Duan, Nathan J Mcneese, Allyson Hauptman, and Rui Zhang. 2024. Empirically understanding the potential impacts and process of social influence in human-AI teams. Proceedings of the ACM on Human-Computer Interaction 8 (2024), 1-32. -
  19. [19][19] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM workshop on artificial intelligence and security . 79-90. -
  20. [20][20] Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022). - · arXiv
  21. [21][21] William Hackett, Lewis Birch, Stefan Trawicki, Neeraj Suri, and Peter Garraghan. 2025. Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails. arXiv preprint arXiv:2504.11168 (2025). - · arXiv
  22. [22][22] David Hartmann, Amin Oueslati, Dimitri Staufer, Lena Pohlmann, Simon Munzert, and Hendrik Heuer. 2025. Lost in moderation: How commercial content moderation apis over-and under-moderate group-targeted hate speech and linguistic variations. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems . 1-26. -
  23. [23][23] Mohammadreza Hojat, Joseph S Gonnella, and Addeane S Caelleigh. 2003. Impartial judgment by the 'gatekeepers' of science: fallibility and accountability in the peer review process. Advances in Health Sciences Education 8, 1 (2003), 75-96. -
  24. [24][24] [ICLR. 2025. ICLR 2025 Reviewer Guide. https://iclr.cc/Conferences/2025/ReviewerGuide.](https://iclr.cc/Conferences/2025/ReviewerGuide) -
  25. [25][25] Yvonne Jansen, Kasper Hornbæk, and Pierre Dragicevic. 2016. What Did Authors Value in the CHI'16 Reviews They Received?. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems . 596-608. -
  26. [26][26] Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli. 2014. Peer review in scientific publications: benefits, critiques, & a survival guide. Ejifcc 25, 3 (2014), 227. -
  27. [27][27] Tine Köhler, M Gloria González-Morales, George C Banks, Ernest H O'Boyle, Joseph A Allen, Ruchi Sinha, Sang Eun Woo, and Lisa MV Gulick. 2020. Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency framework for peer review. Industrial and Organizational Psychology 13, 1 (2020), 1-27. -
  28. [28][28] Michelle S Lam, Mitchell L Gordon, Danaë Metaxa, Jeffrey T Hancock, James A Landay, and Michael S Bernstein. 2022. End-user audits: A system empowering communities to lead large-scale investigations of harmful algorithmic behavior. Proceedings of the ACM on Human-Computer Interaction 6 (2022), 1-34. -
  29. [29][29] Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R Davidson, Veniamin Veselovsky, and Robert West. 2024. The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates. arXiv preprint arXiv:2405.02150 (2024). - · arXiv
  30. [30][30] Rena Li, Sara Kingsley, Chelsea Fan, Proteeti Sinha, Nora Wai, Jaimie Lee, Hong Shen, Motahhare Eslami, and Jason Hong. 2023. Participation and Division of Labor in User-Driven Algorithm Audits: How Do Everyday Users Work together to Surface Algorithmic Harms?. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1-19. -
  31. [31][31] Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. 2024. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. In International Conference on Machine Learning . PMLR, 29575-29620. -
  32. [32][32] Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. 2024. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI 1, 8 (2024), AIoa2400196. -
  33. [33][33] Xinrui Lin, Heyan Huang, Kaihuang Huang, Xin Shu, and John Vines. 2025. Seeking Inspiration through Human-LLM Interaction. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems . 1-17. -
  34. [34][34] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 (2023). - · arXiv
  35. [35][35] Xiang Liu, Torsten Suel, and Nasir Memon. 2014. A robust model for paper reviewer assignment. In Proceedings of the 8th ACM Conference on Recommender systems . 25-32. -
  36. [36][36] Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. 2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499 (2023). - · arXiv
  37. [37][37] Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. 2024. Layoutllm: Layout instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 15630-15640. -
  38. [38][38] Richard Mallett, Jessica Hagen-Zanker, Rachel Slater, and Maren Duvendack. 2012. The benefits and challenges of using systematic reviews in international development research. Journal of development effectiveness 4, 3 (2012), 445-455. -
  39. [39][39] Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018). - · arXiv
  40. [40][40] David Mimno and Andrew McCallum. 2007. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining . 500-509. -
  41. [41][41] Miryam Naddaf. 2025. AI is transforming peer review - and many scientists are worried. https://www.nature.com/articles/d41586-025-00894-7. -
  42. [42][42] Belal Abdullah Hezam Murshed, Suresha Mallappa, Jemal Abawajy, Mufeed Ahmed Naji Saif, Hasib Daowd Esmail Al-Ariki, and Hudhaifa Mohammed Abdulwahab. 2023. Short text topic modelling approaches in the context of big data: taxonomy, survey, and analysis. Artificial Intelligence Review 56, 6 (2023), 5133-5260. -
  43. [43][43] Nature Portfolio. 2025. Referees - Authors and Referees. https://www.nature.com/mp/authors-and-referees/referees. -
  44. [44][44] [NeurIPS. 2024. NeurIPS Fact Sheet. https://media.neurips.cc/Conferences/NeurIPS2024/NeurIPS2024-Fact\_Sheet.pdf.](https://media.neurips.cc/Conferences/NeurIPS2024/NeurIPS2024-Fact_Sheet.pdf) -
  45. [45][45] Kenneth Nugent and Christopher J Peterson. 2024. Peer review and medical journals. Journal of Primary Care & Community Health 15 (2024), 21501319241252235. -
  46. [46][46] Barbara Nussbaumer-Streit, Moriah Ellen, Irma Klerings, Raluca Sfetcu, Nicoletta Riva, Mersiha Mahmić-Kaknjo, Georgios Poulentzas, P Martinez, Eduard Baladia, Liliya Eugenevna Ziganshina, et al. 2021. Resource use during systematic review production varies widely: a scoping review. Journal of clinical epidemiology 139 (2021), 287-296. -
  47. [47][47] Bayode Ogunleye, Tonderai Maswera, Laurence Hirsch, Jotham Gaudoin, and Teresa Brunsdon. 2023. Comparison of topic modelling approaches in the banking context. Applied Sciences 13, 2 (2023), 797. -
  48. [49][49] openreview.net. 2025. https://openreview.net/. -
  49. [50][50] [Paper Copilot. 2025. ICLR 2025 Statistics. https://papercopilot.com/statistics/iclr-statistics/iclr-2025-statistics/.](https://papercopilot.com/statistics/iclr-statistics/iclr-2025-statistics/) -
  50. [51][51] Nicholas Proferes, Naiyan Jones, Sarah Gilbert, Casey Fiesler, and Michael Zimmer. 2021. Studying reddit: A systematic overview of disciplines, approaches, methods, and ethics. Social Media+ Society 7, 2 (2021), 20563051211019004. -
  51. [52][52] Javier Rando, Jie Zhang, Nicholas Carlini, and Florian Tramèr. 2025. Adversarial ml problems are getting harder to solve and to evaluate. arXiv preprint arXiv:2502.02260 (2025). - · arXiv
  52. [54][54] Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, and Jason Bennett Thatcher. 2024. An early categorization of prompt injection attacks on large language models. arXiv preprint arXiv:2402.00898 (2024). - · arXiv
  53. [55][55] Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein. 2021. Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors. Proceedings of the ACM on Human-Computer Interaction 5 (2021), 1-29. -
  54. [56][56] Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. 2025. Automatically evaluating the paper reviewing capability of large language models. arXiv e-prints (2025), arXiv-2502. -
  55. [57][57] Springer Nature. 2025. Transparent peer review to be extended to all of Nature's research papers. https://www.nature.com/articles/d41586-02501880-9. -
  56. [58][58] Chris Street and Kerry W Ward. 2019. Cognitive bias in the peer review process: Understanding a source of friction between reviewers and researchers. ACM SIGMIS Database: the DATABASE for Advances in Information Systems 50, 4 (2019), 52-70. -
  57. [59][59] Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2025. Can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025. arXiv preprint arXiv:2504.09737 (2025). - · arXiv
  58. [60][60] Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. 2024. Ai-driven review systems: evaluating llms in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365 (2024). - · arXiv
  59. [61][61] Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone. 2024. When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour 8, 12 (2024), 2293-2303. -
  60. [62][62] Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . 1-14. -
  61. [63][63] Jessica Vitak, Nicholas Proferes, Katie Shilton, and Zahra Ashktorab. 2017. Ethics regulation in social computing research: Examining the role of institutional review boards. Journal of Empirical Research on Human Research Ethics 12, 5 (2017), 372-382. -
  62. [64][64] Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z Morley Mao, Muhao Chen, and Chaowei Xiao. 2024. Fath: Authenticationbased test-time defense against indirect prompt injection attacks. arXiv preprint arXiv:2410.21492 (2024). - · arXiv
  63. [65][65] Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2024. Knowledge editing for large language models: A survey. Comput. Surveys 57, 3 (2024), 1-37. -
  64. [66][66] Maranke Wieringa. 2020. What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability. In Proceedings of the 2020 conference on fairness, accountability, and transparency . 1-18. -
  65. [67][67] Max L Wilson and Lennart Nacke. 2022. How to: Peer review for CHI (and beyond). In CHI Conference on Human Factors in Computing Systems Extended Abstracts . 1-4. -
  66. [68][68] Fangzhou Wu, Shutong Wu, Yulong Cao, and Chaowei Xiao. 2024. Wipi: A new web threat for llm-driven web agents. arXiv preprint arXiv:2402.16965 (2024). - · arXiv
  67. [69][69] Junjie Xiong, Mingkui Wei, Xiao Han, Zhuo Lu, and Yao Liu. 2025. The Implications of Insecure Use of Fonts Against PDF Documents and Web Pages. IEEE Transactions on Information Forensics and Security 20 (2025), 8773-8787. doi:10.1109/TIFS.2025.3599320 - · DOI
  68. [70][70] Junjie Xiong, Changjia Zhu, Shuhang Lin, Chong Zhang, Yongfeng Zhang, Yao Liu, and Lingyao Li. 2025. Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models. arXiv preprint arXiv:2505.16957 (2025). - · arXiv
  69. [71][71] Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. 2024. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. arXiv preprint arXiv:2412.01708 (2024). - · arXiv
  70. [72][72] Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2025. Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1 . 1809-1820. -
  71. [73][73] Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, and Phillip Howard. 2024. Is your paper being reviewed by an llm? investigating ai text detectability in peer review. arXiv preprint arXiv:2410.03019 (2024). - · arXiv
  72. [74][74] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. In Findings of the Association for Computational Linguistics ACL 2024 . 10471-10506. -
  73. [75][75] Chao Zhang, Shengqi Zhu, Xinyu Yang, Yu-Chia Tseng, Shenrong Jiang, and Jeffrey M Rzeszotarski. 2025. Navigating the fog: How university students recalibrate sensemaking practices to address plausible falsehoods in llm outputs. In Proceedings of the 7th ACM Conference on Conversational User Interfaces . 1-15. -
  74. [76][76] Rui Zhang, Nathan J McNeese, Guo Freeman, and Geoff Musick. 2021. " An ideal human" expectations of AI teammates in human-AI teaming. Proceedings of the ACM on Human-Computer Interaction 4 (2021), 1-25. -
  75. [77][77] Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC-COLING 2024) . 9340-9351. -
  76. [78][78] Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. Deepreview: Improving llm-based paper review with human-like deep thinking process. arXiv preprint arXiv:2503.08569 (2025). · arXiv
  77. [79][79] Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2024. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. World Wide Web 27, 5 (2024), 58.

Cited by

Sorted by citations

Authors
Zhu, C., Xiong, J., Ma, R., Lu, Z., Liu, Y., & Li, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2509.09912
arXiv
2509.09912
OpenAlex
W4414593097
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
peer review, large language models, prompt injection, reviewer bias, review consistency, scholarly publishing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review