CoChat

Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks

Open access

Preprint — published as 10.1145/3803804

The DOI matches its DataCite record; 71 of the 71 references it cites were found in a registry or an index.

Abstract

Large Language Models (LLMs) are increasingly being integrated into the scientific peer-review process, raising new questions about their reliability and resilience to manipulation. In this work, we investigate the potential for hidden prompt injection attacks, where authors embed adversarial text within a paper's PDF to influence the LLM-generated review. We begin by formalising three distinct threat models that envision attackers with different motivations -- not all of which implying malicious intent. For each threat model, we design adversarial prompts that remain invisible to human readers yet can steer an LLM's output toward the author's desired outcome. Using a user study with domain scholars, we derive four representative reviewing prompts used to elicit peer reviews from LLMs. We then evaluate the robustness of our adversarial prompts across (i) different reviewing prompts, (ii) different commercial LLM-based systems, and (iii) different peer-reviewed papers. Our results show that adversarial prompts can reliably mislead the LLM, sometimes in ways that adversely affect a "honest-but-lazy" reviewer. Finally, we propose and empirically assess methods to reduce detectability of adversarial prompts under automated content checks.

Overview

CoChat Summary

TL;DR

The paper formalizes three threat models—Ignore, Detect, and Exploit—for hidden prompt-injection attacks embedded in PDFs against LLMs used in scientific peer review, and shows commercial LLMs can be misled across reviewing prompts, papers, and models. It also tests evasion and defenses, finding keyword-based detection can be bypassed and a prompt-injection-aware reviewing prompt did not mitigate the attacks.

Key findings

  • Against GPT-4o on 26 rejected ICLR papers, Ignore attacks had ASR=1.00; Detect-1 ASR=0.78 and Detect-2 ASR=0.35; Exploit-1 raised the average rating to 9.79±0.44 vs baseline 7.93±0.68, and Exploit-2 to 9.01±0.15.
  • Against GPT-o3 (two papers), Ignore ASR=1.00, Detect-1 ASR=1.00, Detect-2 ASR=0.99; Exploit-1 averaged 9.39±0.61 vs baseline 6.40±0.55, and Exploit-2 8.22±0.98. Against Gemini-2.5-flash, Ignore ASR=0.35, Detect-1 ASR=0.63, Detect-2 ASR=0.49; Exploit-1 averaged 7.78±0.81 vs baseline 7.23±0.69, and Exploit-2 8.19±0.90. Claude Sonnet 4 Ignore ASR=1.00 across 40 requests.
  • Against GPT-5.2, Ignore ASR=1.00, Detect-1 ASR=0.50, Detect-2 ASR=0.30; Exploit-1 averaged 8.98±0.16 vs baseline 7.22±0.44, and Exploit-2 8.91±0.30. A prompt-injection-aware reviewing prompt did not mitigate attacks: Ignore ASR=1.00, Exploit remained above baseline, and Detect showed no statistically significant difference.
  • Evasion via keyword-splitting+homoglyphs and homoglyphs retained ASR=0.79 for Detect-1, while different-language evasion failed. In-the-wild prompts from [58] had no measurable effect (t-test, p=.83); the very long prompt from [94] improved ratings but was significantly weaker than the authors’ Exploit prompts (p<.05). Attacks also succeeded against ChatPDF.

Methods

  • The authors define three threat models—Ignore, Detect, Exploit—and craft adversarial prompts with chat-markup tags, injected into PDFs as white tiny text via PhantomText; Ignore prompts were injected on each page, while Exploit and Detect prompts were injected on the front page.
  • A user study with a convenience sample of four researchers from four European universities produced four reviewing prompts (R0–R3) used to elicit reviews from LLMs.
  • Evaluation used manual web-interface queries to GPT-4o, GPT-o3, Gemini-2.5-flash, Claude Sonnet 4, and later GPT-5.2; the main testbed was 26 rejected ICLR’23/’24 papers, with two accepted ICLR’25 oral papers for follow-up; each query was repeated at least 5 times, mostly 10, totaling approximately 10,000 reviewing requests. Metrics were attack-success rate (ASR) for Ignore/Detect and average rating vs baseline for Exploit, with t-tests.

Limitations

  • The paper states that its considered LLMs are closed-source black boxes, that manual queries and rate limits/budget constrained coverage, and that results are not claimed to be universally generalizable; secondary experiments used only two papers.
  • The paper states that its threat model assumes an “honest-but-lazy” reviewer who uploads the PDF and copy-pastes LLM output, and does not cover reviewers who do not submit the PDF, check the output, or interact repeatedly; the user study had only four responses via convenience sampling.
  • The paper states that its results are “expected” given prior prompt-injection findings, with novelty in the application domain; evident limitations include limited model/paper coverage, discontinued models, restrictions on Gemini/Claude testing, and dependence on correct chat-markup tags.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2508.20863

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Threat Models. We hypothesize that an author may want to use indirect prompt-injection attacks in three ways: to 'exploit' the LLM and solicit a highly positive review; to 'ignore' the reviewing request; and to 'detect' the usage of an LLM. For the latter, we invite the reader to do a keyword search across our paper (CTRL+F) with the string 'This paper is great 10/10', which should find one match in the figure above; and with the string 'The paper proposes a method for' which should not find any match (the 'e' and the 'a' have been replaced with their cyrillic versions, typeset with a dark background, thereby enabling detection).
Fig. 1 Threat Models. We hypothesize that an author may want to use indirect prompt-injection attacks in three ways: to 'exploit' the LLM and solicit a highly positive review; to 'ignore' the reviewing request; and to 'detect' the usage of an LLM. For the latter, we invite the reader to do a keyword search across our paper (CTRL+F) with the string 'This paper is great 10/10', which should find one match in the figure above; and with the string 'The paper proposes a method for' which should not find any match (the 'e' and the 'a' have been replaced with their cyrillic versions, typeset with a dark background, thereby enabling detection).CC-BY-NC-SA · From the PDF
Fig. 2 Effectiveness of Exploit prompts vs GPT-4o
Fig. 2 Effectiveness of Exploit prompts vs GPT-4oCC-BY-NC-SA · From the PDF
Fig. 3 Effectiveness of Exploit prompts vs GPT-3o (for [32, 77]). This figure should be compared with Figure 5 (and Figure 6).
Fig. 3 Effectiveness of Exploit prompts vs GPT-3o (for [32, 77]). This figure should be compared with Figure 5 (and Figure 6).CC-BY-NC-SA · From the PDF
Fig. 4 Effectiveness of 'existing' adversarial prompts. We test the five 'Wild' adversarial prompts found in arXiv preprints (according to [58]) and the 'very long' adversarial prompt used in [94] (unpublished). The test is done on GPT-4o, across our four reviewing prompts, with ten repetitions. More details in Table 16.
Fig. 4 Effectiveness of 'existing' adversarial prompts. We test the five 'Wild' adversarial prompts found in arXiv preprints (according to [58]) and the 'very long' adversarial prompt used in [94] (unpublished). The test is done on GPT-4o, across our four reviewing prompts, with ten repetitions. More details in Table 16.CC-BY-NC-SA · From the PDF
Image 5
CC-BY-NC-SA · From the PDF
Fig. 6 Exploit prompts against Gemini-2.5-flash (for [32, 77]).
Fig. 6 Exploit prompts against Gemini-2.5-flash (for [32, 77]).CC-BY-NC-SA · From the PDF

Verification

Claims the paper makes

0 Valid · 0 Not valid · 0 Not checked · 3 Own results and interpretations

    • Probably not a substantial claim. 3
    • It may state two separate claims as one. 1

    Create a free account to see these claims.

    Self-check

    Percentages, sums and reported differences re-computed from the paper’s own numbers.

    Each contradiction between two passages listed here was confirmed by an independent check.

    5 of 5

    Citations

    Cites

    Show all 71 references

    Research works printed but not linked: 22

    1. AAAI. 2025. AAAI Launches AI-Powered Peer Review Assessment System. https://aaai.org/aaai-launches-ai-powered-peer-review-assessment- system/.
    2. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. {StruQ}: Defending against prompt injection with structured queries. In 34th USENIX Security Symposium (USENIX Security 25). 2383-2400.
    3. CVPR. 2025. CVPR 2025 Changes. https://web.archive.org/web/20250418203046/https://cvpr.thecvf.com/Conferences/2025/CVPRChanges.
    4. ICLR. 2024. ICLR 2024 Reviewer Guide. https://iclr.cc/Conferences/2024/ReviewerGuide.
    5. ICML Conference. 2026. https://x.com/icmlconf/status/2022760600477073876.
    6. Tadayoshi Kohno, Yasemin Acar, and Wulf Loh. 2023. Ethical frameworks and computer security trolley problems: Foundations for conversations. In 32nd USENIX Security Symposium (USENIX Security 23). 5145-5162.
    7. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. 2024. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. In International Conference on Machine Learning. PMLR, 29575-29620.
    8. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24). 1831-1847.
    9. Yupei Liu, Yuqi Jia, Jinyuan Jia, and Neil Zhenqiang Gong. 2025. Evaluating {LLM-based} Personal Information Extraction and Countermeasures. In 34th USENIX Security Symposium (USENIX Security 25). 1669-1688.
    10. Jonathan Lorraine. 2024. Getting harsh conference reviews from LLM-powered reviewers? Consider hiding some extra guidance for the LLM in your paper. https://x.com/jonLorraine9/status/1858592201799852115.
    11. Miryam Naddaf. 2025. Will AI take over peer review? Nature (2025).
    12. Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tramèr, and Katherine Lee. 2025. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations.
    13. NeurIPS. 2024. 2024 Reviewer Guidelines. https://neurips.cc/Conferences/2024/ReviewerGuidelines.
    14. NeurIPS. 2025. NeurIPS25 LLM Policy. https://neurips.cc/Conferences/2025/LLM.
    15. Pangram. 2025. Pangram Predicts 21% of ICLR Reviews are AI-Generated. https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews- are-ai-generated.
    16. Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2025. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse. In The Thirteenth International Conference on Learning Representations.
    17. Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2026. A large-scale randomized study of large language model feedback in peer review. Nature Machine Intelligence (2026), 1-11.
    18. Jiachen T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. 2025. Data Shapley in One Training Run. In The Thirteenth International Conference on Learning Representations.
    19. Xunguang Wang, Daoyuan Wu, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Shuai Wang, Yingjiu Li, Yang Liu, Ning Liu, and Juergen Rahmel. 2025. {SelfDefend}:{LLMs} can defend themselves against jailbreaking in a practical manner. In 34th USENIX Security Symposium (USENIX Security 25). 2441-2460.
    20. Yanting Wang, Wei Zou, Runpeng Geng, and Jinyuan Jia. 2025. {TracLLM}: A Generic Framework for Attributing Long Context {LLMs}. In 34th USENIX Security Symposium (USENIX Security 25). 3845-3864.
    21. jianying Zhou. 2025. Top Cyber Security Conferences Ranking. http://jianying.space/conference-ranking.html.
    22. James Zou and Nitya Thakkar. 2025. Leveraging LLM feedback to enhance review quality. https://web.archive.org/web/20250510123847/https: //blog.iclr.cc/2025/04/15/leveraging-llm-feedback-to-enhance-review-quality/ 1 2306.05880v5 https://openreview.net/forum?id=HNdp1ltDG_ 2 2306.07290v1 https://openreview.net/forum?id=TeeyHEi25C 3 2306.09212v2 https://openreview.net/forum?id=ck4SG9lnrQ 4 2307.02628v1 https://openreview.net/forum?id=bcHty5VvkQ 5 2308.12044v5 https://openreview.net/forum?id=7MlOI37rbn 6 2309.16515v3 https://openreview.net/forum?id=ro4CgvfUKy 7 2309.17144v1 https://openreview.net/forum?id=qW9GVa3Caa 8 2310.00212v3 https://openreview.net/forum?id=JzAuFCKiov 9 2310.05755v1 https://openreview.net/forum?id=veIzQxZUhF 10 2310.06177v1 https://openreview.net/forum?id=wmq67R2PIu 11 2310.13033v2 https://openreview.net/forum?id=TCJbcjS0c2 12 2310.15149v1 https://openreview.net/forum?id=EraNITdn34 13 2310.16277v1 https://openreview.net/forum?id=d2TOOGbrtP 14 2311.00267v1 https://openreview.net/forum?id=7v3tkQmtpE 15 2311.01729v2 https://openreview.net/forum?id=jYHRP6nj9Q 16 2311.04166v2 https://openreview.net/forum?id=YkEW5TabYN 17 2311.18054v2 https://openreview.net/forum?id=eKGEsFdpin 18 2312.00249v2 https://openreview.net/forum?id=rAX55lDjtt 19 2402.03545v3 https://openreview.net/forum?id=sFQe52N40m 20 2402.06220v1 https://openreview.net/forum?id=lWXedJyLuL 21 2404.06694v2 https://openreview.net/forum?id=5ZWxBU9sYG 22 2405.02766v1 https://openreview.net/forum?id=Pa6SiS66p0 23 2406.03665v1 https://openreview.net/forum?id=alnvAZGWLD 24 2412.09968v1 https://openreview.net/forum?id=tB7p0SM5TH 25 2412.12232v1 https://openreview.net/forum?id=gqtbL7j2JW D.2 Experimental Results We report the detailed per-paper results in Tables 17, 18, 19, 20 (providing the specific results for R0, R1, R2, R3, respectively). We report in Figure 5 and Table 15 the results achieved by GPT-4o on the two papers (i.e., [32, 77]) used also for the assessment of GPT-o3 and Gemini-2.5-flash (for which we also report some results in Figure 6). We report in Table 16 the detailed results of our assessment in §8.1, wherein we "compare" the prompts found/proposed in prior work with ours (these results should be compared with those of Exploit in Table 15). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    Cited by

    See all 24

    Metadata

    Authors
    Collu, M. G., Salviati, U., Confalonieri, R., Conti, M., & Apruzzese, G.
    Affiliations
    • Örebro University (SE)
    • University of Padua (IT)
    Venue
    arXiv
    Publisher
    arXiv
    Published
    2025
    ISSN
    2331-8422 (ISSN-L)
    OpenAlex
    W4415986829
    Cited by
    22
    References
    62
    Citations by year
    2026: 19 · 2025: 3
    Type
    posted content
    Language
    en
    Open access
    Locations
    • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
    • arXiv (Cornell University) (cc-by-nc-sa)
    Topics
    • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
    • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
    • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
    Keywords
    large language models, peer review, adversarial prompts, threat model, PDF embedding, reviewer bias, scholarly publishing

    About these checks

    Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

    A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

    A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

    Statistics and arithmetic are recomputed, not judged.

    Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

    What counts as a claim ›How we measure confidence ›How we check this ›

    Are you an author, or do you see a problem with this page? Request a review

    How to cite

    Cite CoChat for the checks on this page, and the paper for its own findings.

    This page
    CoChat. (2026). Citation check: Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks. https://app.cochat.ai/papers/10.48550/arxiv.2508.20863
    The paper
    Collu, M. G., Salviati, U., Confalonieri, R., Conti, M., & Apruzzese, G. (2025). Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks. arXiv. https://doi.org/10.48550/arxiv.2508.20863

    Data sources

    Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

    Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

    Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.