Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks
Open accessPreprint — published as 10.1145/3803804
The DOI matches its DataCite record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) are increasingly being integrated into the scientific peer-review process, raising new questions about their reliability and resilience to manipulation. In this work, we investigate the potential for hidden prompt injection attacks, where authors embed adversarial text within a paper's PDF to influence the LLM-generated review. We begin by formalising three distinct threat models that envision attackers with different motivations -- not all of which implying malicious intent. For each threat model, we design adversarial prompts that remain invisible to human readers yet can steer an LLM's output toward the author's desired outcome. Using a user study with domain scholars, we derive four representative reviewing prompts used to elicit peer reviews from LLMs. We then evaluate the robustness of our adversarial prompts across (i) different reviewing prompts, (ii) different commercial LLM-based systems, and (iii) different peer-reviewed papers. Our results show that adversarial prompts can reliably mislead the LLM, sometimes in ways that adversely affect a "honest-but-lazy" reviewer. Finally, we propose and empirically assess methods to reduce detectability of adversarial prompts under automated content checks.
Overview
CoChat Summary
TL;DR
The paper formalizes three threat models—Ignore, Detect, and Exploit—for hidden prompt-injection attacks embedded in PDFs against LLMs used in scientific peer review, and shows commercial LLMs can be misled across reviewing prompts, papers, and models. It also tests evasion and defenses, finding keyword-based detection can be bypassed and a prompt-injection-aware reviewing prompt did not mitigate the attacks.
Key findings
- Against GPT-4o on 26 rejected ICLR papers, Ignore attacks had ASR=1.00; Detect-1 ASR=0.78 and Detect-2 ASR=0.35; Exploit-1 raised the average rating to 9.79±0.44 vs baseline 7.93±0.68, and Exploit-2 to 9.01±0.15.
- Against GPT-o3 (two papers), Ignore ASR=1.00, Detect-1 ASR=1.00, Detect-2 ASR=0.99; Exploit-1 averaged 9.39±0.61 vs baseline 6.40±0.55, and Exploit-2 8.22±0.98. Against Gemini-2.5-flash, Ignore ASR=0.35, Detect-1 ASR=0.63, Detect-2 ASR=0.49; Exploit-1 averaged 7.78±0.81 vs baseline 7.23±0.69, and Exploit-2 8.19±0.90. Claude Sonnet 4 Ignore ASR=1.00 across 40 requests.
- Against GPT-5.2, Ignore ASR=1.00, Detect-1 ASR=0.50, Detect-2 ASR=0.30; Exploit-1 averaged 8.98±0.16 vs baseline 7.22±0.44, and Exploit-2 8.91±0.30. A prompt-injection-aware reviewing prompt did not mitigate attacks: Ignore ASR=1.00, Exploit remained above baseline, and Detect showed no statistically significant difference.
- Evasion via keyword-splitting+homoglyphs and homoglyphs retained ASR=0.79 for Detect-1, while different-language evasion failed. In-the-wild prompts from [58] had no measurable effect (t-test, p=.83); the very long prompt from [94] improved ratings but was significantly weaker than the authors’ Exploit prompts (p<.05). Attacks also succeeded against ChatPDF.
Methods
- The authors define three threat models—Ignore, Detect, Exploit—and craft adversarial prompts with chat-markup tags, injected into PDFs as white tiny text via PhantomText; Ignore prompts were injected on each page, while Exploit and Detect prompts were injected on the front page.
- A user study with a convenience sample of four researchers from four European universities produced four reviewing prompts (R0–R3) used to elicit reviews from LLMs.
- Evaluation used manual web-interface queries to GPT-4o, GPT-o3, Gemini-2.5-flash, Claude Sonnet 4, and later GPT-5.2; the main testbed was 26 rejected ICLR’23/’24 papers, with two accepted ICLR’25 oral papers for follow-up; each query was repeated at least 5 times, mostly 10, totaling approximately 10,000 reviewing requests. Metrics were attack-success rate (ASR) for Ignore/Detect and average rating vs baseline for Exploit, with t-tests.
Limitations
- The paper states that its considered LLMs are closed-source black boxes, that manual queries and rate limits/budget constrained coverage, and that results are not claimed to be universally generalizable; secondary experiments used only two papers.
- The paper states that its threat model assumes an “honest-but-lazy” reviewer who uploads the PDF and copy-pastes LLM output, and does not cover reviewers who do not submit the PDF, check the output, or interact repeatedly; the user study had only four responses via convenience sampling.
- The paper states that its results are “expected” given prior prompt-injection findings, with novelty in the application domain; evident limitations include limited model/paper coverage, discontinued models, restrictions on Gemini/Claude testing, and dependence on correct chat-markup tags.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig. 3 Effectiveness of Exploit prompts vs GPT-3o (for [32, 77]). This figure should be compared with Figure 5 (and Figure 6).](https://figures.cochat.ai/paper-figures/7b5c78a7dae148c4aec31644be9a012c/misleading-large-language-models-used-or-misused-in-scientific-peer-reviewing-figure-3.png)
![Fig. 4 Effectiveness of 'existing' adversarial prompts. We test the five 'Wild' adversarial prompts found in arXiv preprints (according to [58]) and the 'very long' adversarial prompt used in [94] (unpublished). The test is done on GPT-4o, across our four reviewing prompts, with ten repetitions. More details in Table 16.](https://figures.cochat.ai/paper-figures/f70e708076f547b3b241b2eb33db1b45/misleading-large-language-models-used-or-misused-in-scientific-peer-reviewing-figure-4.png)

![Fig. 6 Exploit prompts against Gemini-2.5-flash (for [32, 77]).](https://figures.cochat.ai/paper-figures/bccb2b977baa43cba60a50090c440b11/misleading-large-language-models-used-or-misused-in-scientific-peer-reviewing-figure-6.png)
Verification
Claims the paper makes
0 Valid · 0 Not valid · 0 Not checked · 3 Own results and interpretations
- Probably not a substantial claim. 3
- It may state two separate claims as one. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“We injected each of the six prompts into two different papers [32,77] and submitted them to GPT-4o under four reviewing prompts, repeating each request 10 times-resulting in 240 reviews.”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > 8.1 Were In-the-Wild Prompts Effective?
Reported 240 · recomputed 480
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > 5.2.1 Overview and Major Results.
“• Ignore attacks are always successful.”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > Reviewing Prompt
“| 23 | 2406.03665v1 | 7.40 ±0.52 | 9.50 ±0.52 | 9.00 ±0.00 | 1.00 | 0.65 | 0.40 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > GPT-4o).
“We inject the prompts only on the PDF first page (consistent with the experiment in §5.1).”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > R0 R1 R2 R3
“We report the ASR of Detect-1, Detect-2, and their simultaneous execution (Detect-1 OR Detect-2) when the adversarial prompts are inserted in each page of the paper.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > Additional information
“ChatPDF [4].ChatPDF is an LLM-based service with native support for PDF interaction”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > References
“4. 2025. Stanford Agentic Reviewer. https://paperreview.ai/.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Hence, for this experiment, we perform a total of 120 queries (40 for the baseline, and 80 for each Exploit attack).”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > C.1 Unsuccessful Exploit vs reject reviewer
Reported 120 · recomputed 200
Citations
Cites
Comparison of convenience sampling and purposive sampling
Etikan, I. (2016) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20160501.11 · Cited by 11,412
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667
Bornmann, L., Haunschild, R., & Mutz, R. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00903-w · Cited by 496
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
SecAlign: Defending Against Prompt Injection with Preference Optimization
Chen, S., Zharmagambetov, A. S., Mahloujifar, S., Chaudhuri, K., Wagner, D., & Guo, C. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744836 · Cited by 366
The strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2023) · arXiv · doi:10.48550/arxiv.2309.15884 · Cited by 345
The strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00327 · Cited by 345
Research works printed but not linked: 22
- AAAI. 2025. AAAI Launches AI-Powered Peer Review Assessment System. https://aaai.org/aaai-launches-ai-powered-peer-review-assessment- system/.
- Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. {StruQ}: Defending against prompt injection with structured queries. In 34th USENIX Security Symposium (USENIX Security 25). 2383-2400.
- CVPR. 2025. CVPR 2025 Changes. https://web.archive.org/web/20250418203046/https://cvpr.thecvf.com/Conferences/2025/CVPRChanges.
- ICLR. 2024. ICLR 2024 Reviewer Guide. https://iclr.cc/Conferences/2024/ReviewerGuide.
- ICML Conference. 2026. https://x.com/icmlconf/status/2022760600477073876.
- Tadayoshi Kohno, Yasemin Acar, and Wulf Loh. 2023. Ethical frameworks and computer security trolley problems: Foundations for conversations. In 32nd USENIX Security Symposium (USENIX Security 23). 5145-5162.
- Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. 2024. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. In International Conference on Machine Learning. PMLR, 29575-29620.
- Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24). 1831-1847.
- Yupei Liu, Yuqi Jia, Jinyuan Jia, and Neil Zhenqiang Gong. 2025. Evaluating {LLM-based} Personal Information Extraction and Countermeasures. In 34th USENIX Security Symposium (USENIX Security 25). 1669-1688.
- Jonathan Lorraine. 2024. Getting harsh conference reviews from LLM-powered reviewers? Consider hiding some extra guidance for the LLM in your paper. https://x.com/jonLorraine9/status/1858592201799852115.
- Miryam Naddaf. 2025. Will AI take over peer review? Nature (2025).
- Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tramèr, and Katherine Lee. 2025. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations.
- NeurIPS. 2024. 2024 Reviewer Guidelines. https://neurips.cc/Conferences/2024/ReviewerGuidelines.
- NeurIPS. 2025. NeurIPS25 LLM Policy. https://neurips.cc/Conferences/2025/LLM.
- Pangram. 2025. Pangram Predicts 21% of ICLR Reviews are AI-Generated. https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews- are-ai-generated.
- Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2025. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse. In The Thirteenth International Conference on Learning Representations.
- Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2026. A large-scale randomized study of large language model feedback in peer review. Nature Machine Intelligence (2026), 1-11.
- Jiachen T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. 2025. Data Shapley in One Training Run. In The Thirteenth International Conference on Learning Representations.
- Xunguang Wang, Daoyuan Wu, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Shuai Wang, Yingjiu Li, Yang Liu, Ning Liu, and Juergen Rahmel. 2025. {SelfDefend}:{LLMs} can defend themselves against jailbreaking in a practical manner. In 34th USENIX Security Symposium (USENIX Security 25). 2441-2460.
- Yanting Wang, Wei Zou, Runpeng Geng, and Jinyuan Jia. 2025. {TracLLM}: A Generic Framework for Attributing Long Context {LLMs}. In 34th USENIX Security Symposium (USENIX Security 25). 3845-3864.
- jianying Zhou. 2025. Top Cyber Security Conferences Ranking. http://jianying.space/conference-ranking.html.
- James Zou and Nitya Thakkar. 2025. Leveraging LLM feedback to enhance review quality. https://web.archive.org/web/20250510123847/https: //blog.iclr.cc/2025/04/15/leveraging-llm-feedback-to-enhance-review-quality/ 1 2306.05880v5 https://openreview.net/forum?id=HNdp1ltDG_ 2 2306.07290v1 https://openreview.net/forum?id=TeeyHEi25C 3 2306.09212v2 https://openreview.net/forum?id=ck4SG9lnrQ 4 2307.02628v1 https://openreview.net/forum?id=bcHty5VvkQ 5 2308.12044v5 https://openreview.net/forum?id=7MlOI37rbn 6 2309.16515v3 https://openreview.net/forum?id=ro4CgvfUKy 7 2309.17144v1 https://openreview.net/forum?id=qW9GVa3Caa 8 2310.00212v3 https://openreview.net/forum?id=JzAuFCKiov 9 2310.05755v1 https://openreview.net/forum?id=veIzQxZUhF 10 2310.06177v1 https://openreview.net/forum?id=wmq67R2PIu 11 2310.13033v2 https://openreview.net/forum?id=TCJbcjS0c2 12 2310.15149v1 https://openreview.net/forum?id=EraNITdn34 13 2310.16277v1 https://openreview.net/forum?id=d2TOOGbrtP 14 2311.00267v1 https://openreview.net/forum?id=7v3tkQmtpE 15 2311.01729v2 https://openreview.net/forum?id=jYHRP6nj9Q 16 2311.04166v2 https://openreview.net/forum?id=YkEW5TabYN 17 2311.18054v2 https://openreview.net/forum?id=eKGEsFdpin 18 2312.00249v2 https://openreview.net/forum?id=rAX55lDjtt 19 2402.03545v3 https://openreview.net/forum?id=sFQe52N40m 20 2402.06220v1 https://openreview.net/forum?id=lWXedJyLuL 21 2404.06694v2 https://openreview.net/forum?id=5ZWxBU9sYG 22 2405.02766v1 https://openreview.net/forum?id=Pa6SiS66p0 23 2406.03665v1 https://openreview.net/forum?id=alnvAZGWLD 24 2412.09968v1 https://openreview.net/forum?id=tB7p0SM5TH 25 2412.12232v1 https://openreview.net/forum?id=gqtbL7j2JW D.2 Experimental Results We report the detailed per-paper results in Tables 17, 18, 19, 20 (providing the specific results for R0, R1, R2, R3, respectively). We report in Figure 5 and Table 15 the results achieved by GPT-4o on the two papers (i.e., [32, 77]) used also for the assessment of GPT-o3 and Gemini-2.5-flash (for which we also report some results in Figure 6). We report in Table 16 the detailed results of our assessment in §8.1, wherein we "compare" the prompts found/proposed in prior work with ours (these results should be compared with those of Exploit in Table 15). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
Cited by
The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to AnalysisPreprint
Wang, P., Li, X., Xiang, C., Zhang, J., Li, Y., Zhang, L., Wang, X., & Tian, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10453 · Cited by 46
Autonomous Research Agents: A Survey of AI Scientists and the Verification GapPreprint
Ding, T., Nannapaneni, A., Liu, B., & Zhang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05179 · Cited by 23
Jailbreaking LLMs: A Survey of Attacks, Defenses and EvaluationPreprint
Hakim, S. B., Gharami, K., Ghalaty, N. F., Moni, S. S., Xu, S., & Song, H. (2026) · doi:10.36227/techrxiv.176773228.86819800/v1 · Cited by 15
When AI reviews science: Can we trust the referee?
Wang, J., Liu, Y., Xu, H., Hu, K., Di, S., Ni, W., Yue, L., Zhang, M.-L., Ren, K., & Chen, L. (2026) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2026.100030 · Cited by 9
Torrielli, F., Locci, S., Rapp, A., & Di Caro, L. (2026) · Scientometrics · doi:10.1007/s11192-026-05695-x · Cited by 8
Metadata
- Authors
- Collu, M. G., Salviati, U., Confalonieri, R., Conti, M., & Apruzzese, G.
- Affiliations
- Örebro University (SE)
- University of Padua (IT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2508.20863
- OpenAlex
- W4415986829
- Cited by
- 22
- References
- 62
- Citations by year
- 2026: 19 · 2025: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, peer review, adversarial prompts, threat model, PDF embedding, reviewer bias, scholarly publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks. https://app.cochat.ai/papers/10.48550/arxiv.2508.20863
- The paper
- Collu, M. G., Salviati, U., Confalonieri, R., Conti, M., & Apruzzese, G. (2025). Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks. arXiv. https://doi.org/10.48550/arxiv.2508.20863
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.