CoChat

Exploiting large language models in peer review: indirect prompt injection attacks and integrity probes

Open access
Torrielli, Federico, Locci, Stefano, Rapp, Amon, Di Caro, Luigi · 2026 · Scientometrics

Published version of a preprint

The DOI matches its Crossref record; 97 of the 98 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Large language models are beginning to enter peer review as tools for summarizing manuscripts, drafting evaluations, and reducing reviewer workload. Yet this use creates a security problem specific to evaluative settings: the manuscript being judged can also contain hidden instructions that shape the model’s judgment. We investigate this risk through indirect prompt injection, where hidden text embedded in a manuscript is processed like a benign prompt by a public chatbot during review generation. We introduce an Author-Reviewer-Organizer framework that distinguishes author-side manipulation from organizer-side integrity checks, and use it to evaluate both offensive payloads, which steer reviews toward positive or negative judgments, and integrity-probing payloads, which trigger refusal, watermarking, or external-site redirection. The redirection payload is introduced here as a novel integrity-check mechanism. In experiments on 100 OpenReview papers, two public chatbot systems, five payload families, two custom prompt approaches, three injection positions, and five repeated runs per condition, we generate 42000 model outputs. Hidden instructions are highly effective: positive steering, refusal, and external-site redirection exceed 98% success on both systems; watermarking remains high but less reliable, reaching 94.27% on ChatGPT and 88.17% on Gemini; and negative steering is near-ceiling on ChatGPT but lower on Gemini. Vulnerability varies with payload design, model, and document position. These findings show that current models exhibit “contextual blindness”: they do not reliably separate manuscript content from embedded control text. This weakness threatens the validity of AI-assisted review, while also creating a dual-use channel that organizers may repurpose for detection. Peer-review systems that rely on public LLMs therefore require stronger safeguards between document content and instructions before they can be trusted in evaluative workflows.

Checked 2026-11-20 · 1 reference not checked yet

Overview

CoChat Summary

TL;DR

This paper studies indirect prompt injection in AI-assisted peer review by embedding hidden instructions in manuscript PDFs and asking public ChatGPT and Gemini chatbots to review them. Across 42,000 outputs, hidden instructions frequently steered reviews, forced refusals, inserted watermarks, or redirected users, revealing what the authors call “contextual blindness” in current LLM pipelines.

Key findings

  • Both systems were highly vulnerable: pooled overall attack success rate was 98.34% (20,651/21,000) for ChatGPT and 94.02% (19,744/21,000) for Gemini. Positive steering, refusal, and external-site redirection exceeded 98% success on both systems; watermarking reached 94.27% on ChatGPT and 88.17% on Gemini; negative steering was 98.97% on ChatGPT but 86.87% on Gemini.
  • Vulnerability varied by payload, model, and injection locus. ChatGPT pooled ASR was 99.87% at L1, 95.19% at L2, and 99.96% at L1+L2; Gemini was 90.89%, 95.74%, and 95.43% across the same loci. Narrative watermarking dropped to 72.8% at L2 on ChatGPT; policy-puppetry negative steering fell to 50.0% at L1 on Gemini.
  • Leetspeak-obfuscated Refusal and Watermark payloads achieved 100% ASR on all reviewing prompts, matching plaintext performance. In a shared-benchmark comparison, the authors’ Refusal and Positive Steering payloads reached 100% ASR on both models, versus 44.6%/51.7% for Collu et al.’s Ignore and 64.2–68.3%/50.4–67.5% for Exploit-1/2.
  • Steering shifted review polarity: positive steering reached VADER mean 0.991 ± 0.001 on both models; negative steering reached -0.769 ± 0.008 on ChatGPT and -0.384 ± 0.022 on Gemini, against human reference means of 0.827 (accepted) and 0.816 (rejected). Successful outputs almost always reproduced the injected opening (99.93% ChatGPT positive, 100.00% ChatGPT negative, 99.51% Gemini positive, 97.51% Gemini negative).

Methods

  • Design: Author-Reviewer-Organizer (ARO) framework; five payload families (positive steering, negative steering, refusal, watermarking, external-site redirection), two prompt approaches (Narrative Prompting, Policy Puppetry), three injection positions (L1 first page, L2 last page, L1+L2), five repeated runs per condition; hidden text embedded via PDF render-mode 3 Tr overlay using ReportLab/pypdf.
  • Data/sample: OpenReview corpus restricted to papers published no later than 30 Nov 2022; quality filter retained reviews above 75th percentile length (avg >3279 characters), yielding 4,900 papers and >17,000 reviews; final 100 manuscripts (50 accepted, 50 rejected) selected deterministically by total review length within decision class, spanning ICLR 2019–2022 and NeurIPS 2021, with 420 individual reviews (avg 6,003 characters).
  • Procedure/analysis: automated Selenium upload-and-query workflow on public ChatGPT (GPT-5.3 Instant) and Gemini (Gemini-3-flash-preview) web interfaces, logged-out/private, no custom configuration; 42,000 outputs total. Watermark and external-site redirection scored by exact checks; refusal by primary LLM judge (Qwen3.5-27B); steering by consensus of two LLM judges (Qwen3.5-27B and Gemma-4-31B-IT), with VADER and a DistilBERT academic sentiment classifier as robustness checks; pooled success rates and Wilson intervals reported.

Limitations

  • Paper-stated: two public chatbots in a computer science peer-review setting, so rates are not fixed properties of other disciplines, proprietary systems, or future model versions; only three reviewer request templates, so attack performance may shift under other prompt formulations; carrier restricted to invisible PDF text via rendering-mode manipulation, so other carriers may produce different vulnerability profiles; the 100-paper sample is not representative of OpenReview submissions, weighted toward unusually detailed reviews and concentrated in ICLR 2019–2022 and NeurIPS 2021.
  • Paper-stated: the deceptive external-site redirection variant was not empirically evaluated; whether deceptive framing increases link-following remains untested and would require a user study. Sampling parameters (temperature, top-p, max tokens) are not exposed through the public web interfaces, so only default web-interface behavior is reported.
  • Evident from text: the sample selection by review length introduces selection bias (the paper flags this), and all NeurIPS entries in the sample are accepted papers because no rejected NeurIPS 2021 submissions entered the top-50 rejected set; the study measures model outputs, not actual reviewer behavior, and the external-site redirection signal depends on a human reviewer interacting with the model’s output.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11192-026-05695-x

Cites

97 of 98 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Taxonomy of injected payloads: ARO Attack shows the pipeline for the Indirect Prompt Injection for the Positive Steering Payload. ARO Defenses address the LLM use by reviewers via different IPI payloads employed by the Organizers: Refusal for LLM usage prevention; Watermark and External Site Redirection for LLM usage detection
Fig. 1 Taxonomy of injected payloads: ARO Attack shows the pipeline for the Indirect Prompt Injection for the Positive Steering Payload. ARO Defenses address the LLM use by reviewers via different IPI payloads employed by the Organizers: Refusal for LLM usage prevention; Watermark and External Site Redirection for LLM usage detectionCC-BY · From the PDF
Fig. 2 Example of Cyrillic homoglyphs substitution for the letters 'e' and 'o'. The original text and the watermarked version look identical to the human reader
Fig. 2 Example of Cyrillic homoglyphs substitution for the letters 'e' and 'o'. The original text and the watermarked version look identical to the human readerCC-BY · From the PDF
Fig. 3 Violin plots of pooled VADER compound scores across five runs. Positive steering is compared with human reviews of accepted papers, and negative steering with human reviews of rejected papers
Fig. 3 Violin plots of pooled VADER compound scores across five runs. Positive steering is compared with human reviews of accepted papers, and negative steering with human reviews of rejected papersCC-BY · From the PDF
Fig. 4 Steering ASR by evaluator family. VADER is included only as a coarse robustness check and is not expected to be well matched to academic review language. Judge A and Judge B are the two LLM judges, and consensus requires agreement between them
Fig. 4 Steering ASR by evaluator family. VADER is included only as a coarse robustness check and is not expected to be well matched to academic review language. Judge A and Judge B are the two LLM judges, and consensus requires agreement between themCC-BY · From the PDF
Fig. 5 Density plots of academic-classifier confidence for pooled steering outputs, separated by consensus outcome for ChatGPT (top) and Gemini (bottom). ChatGPT successful outputs are shifted toward higher confidence, while Gemini shows substantial overlap between success and failure; the text discusses the direction-level asymmetry masked by pooling
Fig. 5 Density plots of academic-classifier confidence for pooled steering outputs, separated by consensus outcome for ChatGPT (top) and Gemini (bottom). ChatGPT successful outputs are shifted toward higher confidence, while Gemini shows substantial overlap between success and failure; the text discusses the direction-level asymmetry masked by poolingCC-BY · From the PDF
Image 6
CC-BY · From the PDF
Image 7
CC-BY · From the PDF

Verification

Claims the paper makes

No claims could be anchored to a verbatim quote in the text.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 98 references

Research works printed but not linked: 3

  1. Liu, Y., Jia, Y., Geng, R., Jia, J., & Gong, N.Z. (2024). Formalizing and benchmarking prompt injection attacks and defenses. 33rd USENIX Security Symposium (USENIX Security 24) (pp. 1831–1847). Philadelphia, PAUSENIX Association. https://www.usenix.org/conference/usenixsecurity24/presentation/liu-yupei
  2. Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., & Gomez, A. N. (2017). Polosukhin, I. Attention is all you need. NIPS, 30,
  3. Wu, X., Wang, Y., Jegelka, S., & Jadbabaie, A. (2025). On the emergence of position bias in transformers. A. Singh et al. (Eds.), Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 (Vol. 267). PMLR / OpenReview.net. https://proceedings.mlr.press/v267/wu25ad.html

Cited by

See all 8

Metadata

Authors
Torrielli, F., Locci, S., Rapp, A., & Di Caro, L.
Affiliations
University of Turin (IT)
Corresponding author
  • Torrielli, Federico
  • Locci, Stefano
  • Rapp, Amon
Venue
Scientometrics
Published
2026-07
Volume
131
Issue
7
Pages
4889-4951
ISSN
  • 0138-9130 (ISSN-L)
  • 1588-2861
OpenAlex
W7168048551
Cited by
8
References
93
Citations by year
2026: 8
FWCI
7.74
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientometrics (publishedVersion, cc-by)
  • Research Square (submittedVersion, cc-by)
  • Institutional Research Information System University of Turin (University of Turin) (submittedVersion, cc-by)
Funders
  • NextGenerationEU
  • Università degli Studi di Torino
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
large language models, peer review, prompt injection, indirect prompt injection, AI-assisted review, watermarking, ChatGPT, Gemini

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-20 (model deepseek-v4.1-flash, prompt extract-v3).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Exploiting large language models in peer review: indirect prompt injection attacks and integrity probes. https://app.cochat.ai/papers/10.1007/s11192-026-05695-x
The paper
Torrielli, F., Locci, S., Rapp, A., & Di Caro, L. (2026). Exploiting large language models in peer review: indirect prompt injection attacks and integrity probes. Scientometrics. https://doi.org/10.1007/s11192-026-05695-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.