Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks
Open accessPreprint — published as 10.1145/3803804
The DOI matches its DataCite record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) are increasingly being integrated into the scientific peer-review process, raising new questions about their reliability and resilience to manipulation. In this work, we investigate the potential for hidden prompt injection attacks, where authors embed adversarial text within a paper's PDF to influence the LLM-generated review. We begin by formalising three distinct threat models that envision attackers with different motivations -- not all of which implying malicious intent. For each threat model, we design adversarial prompts that remain invisible to human readers yet can steer an LLM's output toward the author's desired outcome. Using a user study with domain scholars, we derive four representative reviewing prompts used to elicit peer reviews from LLMs. We then evaluate the robustness of our adversarial prompts across (i) different reviewing prompts, (ii) different commercial LLM-based systems, and (iii) different peer-reviewed papers. Our results show that adversarial prompts can reliably mislead the LLM, sometimes in ways that adversely affect a "honest-but-lazy" reviewer. Finally, we propose and empirically assess methods to reduce detectability of adversarial prompts under automated content checks.
Overview
CoChat Summary
TL;DR
The paper formalizes three threat models—Ignore, Detect, and Exploit—for hidden prompt-injection attacks embedded in PDFs against LLMs used in scientific peer review, and shows commercial LLMs can be misled across reviewing prompts, papers, and models. It also tests evasion and defenses, finding keyword-based detection can be bypassed and a prompt-injection-aware reviewing prompt did not mitigate the attacks.
Key findings
- Against GPT-4o on 26 rejected ICLR papers, Ignore attacks had ASR=1.00; Detect-1 ASR=0.78 and Detect-2 ASR=0.35; Exploit-1 raised the average rating to 9.79±0.44 vs baseline 7.93±0.68, and Exploit-2 to 9.01±0.15.
- Against GPT-o3 (two papers), Ignore ASR=1.00, Detect-1 ASR=1.00, Detect-2 ASR=0.99; Exploit-1 averaged 9.39±0.61 vs baseline 6.40±0.55, and Exploit-2 8.22±0.98. Against Gemini-2.5-flash, Ignore ASR=0.35, Detect-1 ASR=0.63, Detect-2 ASR=0.49; Exploit-1 averaged 7.78±0.81 vs baseline 7.23±0.69, and Exploit-2 8.19±0.90. Claude Sonnet 4 Ignore ASR=1.00 across 40 requests.
- Against GPT-5.2, Ignore ASR=1.00, Detect-1 ASR=0.50, Detect-2 ASR=0.30; Exploit-1 averaged 8.98±0.16 vs baseline 7.22±0.44, and Exploit-2 8.91±0.30. A prompt-injection-aware reviewing prompt did not mitigate attacks: Ignore ASR=1.00, Exploit remained above baseline, and Detect showed no statistically significant difference.
- Evasion via keyword-splitting+homoglyphs and homoglyphs retained ASR=0.79 for Detect-1, while different-language evasion failed. In-the-wild prompts from [58] had no measurable effect (t-test, p=.83); the very long prompt from [94] improved ratings but was significantly weaker than the authors’ Exploit prompts (p<.05). Attacks also succeeded against ChatPDF.
Methods
- The authors define three threat models—Ignore, Detect, Exploit—and craft adversarial prompts with chat-markup tags, injected into PDFs as white tiny text via PhantomText; Ignore prompts were injected on each page, while Exploit and Detect prompts were injected on the front page.
- A user study with a convenience sample of four researchers from four European universities produced four reviewing prompts (R0–R3) used to elicit reviews from LLMs.
- Evaluation used manual web-interface queries to GPT-4o, GPT-o3, Gemini-2.5-flash, Claude Sonnet 4, and later GPT-5.2; the main testbed was 26 rejected ICLR’23/’24 papers, with two accepted ICLR’25 oral papers for follow-up; each query was repeated at least 5 times, mostly 10, totaling approximately 10,000 reviewing requests. Metrics were attack-success rate (ASR) for Ignore/Detect and average rating vs baseline for Exploit, with t-tests.
Limitations
- The paper states that its considered LLMs are closed-source black boxes, that manual queries and rate limits/budget constrained coverage, and that results are not claimed to be universally generalizable; secondary experiments used only two papers.
- The paper states that its threat model assumes an “honest-but-lazy” reviewer who uploads the PDF and copy-pastes LLM output, and does not cover reviewers who do not submit the PDF, check the output, or interact repeatedly; the user study had only four responses via convenience sampling.
- The paper states that its results are “expected” given prior prompt-injection findings, with novelty in the application domain; evident limitations include limited model/paper coverage, discontinued models, restrictions on Gemini/Claude testing, and dependence on correct chat-markup tags.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig. 3 Effectiveness of Exploit prompts vs GPT-3o (for [32, 77]). This figure should be compared with Figure 5 (and Figure 6).](https://figures.cochat.ai/paper-figures/7b5c78a7dae148c4aec31644be9a012c/misleading-large-language-models-used-or-misused-in-scientific-peer-reviewing-figure-3.png)
![Fig. 4 Effectiveness of 'existing' adversarial prompts. We test the five 'Wild' adversarial prompts found in arXiv preprints (according to [58]) and the 'very long' adversarial prompt used in [94] (unpublished). The test is done on GPT-4o, across our four reviewing prompts, with ten repetitions. More details in Table 16.](https://figures.cochat.ai/paper-figures/f70e708076f547b3b241b2eb33db1b45/misleading-large-language-models-used-or-misused-in-scientific-peer-reviewing-figure-4.png)

![Fig. 6 Exploit prompts against Gemini-2.5-flash (for [32, 77]).](https://figures.cochat.ai/paper-figures/bccb2b977baa43cba60a50090c440b11/misleading-large-language-models-used-or-misused-in-scientific-peer-reviewing-figure-6.png)
Verification
Claims the paper makes
0 Valid · 0 Not valid · 0 Not checked · 3 Own results and interpretations
- Probably not a substantial claim. 3
- It may state two separate claims as one. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“We injected each of the six prompts into two different papers [32,77] and submitted them to GPT-4o under four reviewing prompts, repeating each request 10 times-resulting in 240 reviews.”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > 8.1 Were In-the-Wild Prompts Effective?
Reported 240 · recomputed 480
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > 5.2.1 Overview and Major Results.
“• Ignore attacks are always successful.”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > Reviewing Prompt
“| 23 | 2406.03665v1 | 7.40 ±0.52 | 9.50 ±0.52 | 9.00 ±0.00 | 1.00 | 0.65 | 0.40 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > GPT-4o).
“We inject the prompts only on the PDF first page (consistent with the experiment in §5.1).”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > R0 R1 R2 R3
“We report the ASR of Detect-1, Detect-2, and their simultaneous execution (Detect-1 OR Detect-2) when the adversarial prompts are inserted in each page of the paper.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > Additional information
“ChatPDF [4].ChatPDF is an LLM-based service with native support for PDF interaction”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > References
“4. 2025. Stanford Agentic Reviewer. https://paperreview.ai/.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Hence, for this experiment, we perform a total of 120 queries (40 for the baseline, and 80 for each Exploit attack).”
§ Misleading Large Language Models used (or misused) in Scientific Peer-Reviewing via Hidden Prompt-Injection Attacks > C.1 Unsuccessful Exploit vs reject reviewer
Reported 120 · recomputed 200
Citations
Cites
Comparison of convenience sampling and purposive sampling
Etikan, I. (2016) · American Journal of Theoretical and Applied Statistics · doi:10.11648/j.ajtas.20160501.11 · Cited by 11,412
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03825 · Cited by 861
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667
Bornmann, L., Haunschild, R., & Mutz, R. (2021) · Humanities and Social Sciences Communications · doi:10.1057/s41599-021-00903-w · Cited by 496
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441
SecAlign: Defending Against Prompt Injection with Preference Optimization
Chen, S., Zharmagambetov, A. S., Mahloujifar, S., Chaudhuri, K., Wagner, D., & Guo, C. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3744836 · Cited by 376
The strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2023) · arXiv · doi:10.48550/arxiv.2309.15884 · Cited by 345
The strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00327 · Cited by 345
The Political Biases of ChatGPT
Rozado, D. (2023) · Social Sciences · doi:10.3390/socsci12030148 · Cited by 281
Ganjavi, C., Eppler, M. B., Pekcan, A., Biedermann, B., Abreu, A., Collins, G. S., Gill, I. S., & Cacciamani, G. E. (2024) · BMJ · doi:10.1136/bmj-2023-077192 · Cited by 266
Poisoning Retrieval Corpora by Injecting Adversarial Passages
Zhong, Z., Huang, Z., Wettig, A., & Chen, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.849 · Cited by 258
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
AI agents under threat: A survey of key security challenges and future pathways
Deng, Z., Guo, Y., Han, C., Ma, W., Xiong, J., Wen, S., & Xiang, Y. (2025) · ACM Computing Surveys · doi:10.1145/3716628 · Cited by 210
DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
Liu, Y., Jia, Y., Jia, J., Song, D., & Gong, N. Z. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00250 · Cited by 205
Apruzzese, G., Anderson, H. S., Dambra, S., Freeman, D. M., Pierazzi, F., & Roundy, K. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00031 · Cited by 199
Self-citationBailey, M., Dittrich, D., Kenneally, E., & Maughan, D. (2012) · IEEE Security & Privacy Magazine · doi:10.1109/msp.2012.52 · Cited by 190
Phantom: General Backdoor Attacks on Retrieval Augmented Language GenerationPreprint
Chaudhari, H., Severi, G., Abascal, J., Suri, A., Jagielski, M., Choquette-Choo, C. A., Nasr, M., Nita-Rotaru, C., & Oprea, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.20485 · Cited by 157
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 149
DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios
Wu, J., Zhan, R., Wong, D., Yang, S., Yang, X., Yuan, Y., & Chao, L. (2024) · neural information processing systems · doi:10.52202/079017-3186 · Cited by 139
Poisoning Web-Scale Training Datasets is Practical
Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., & Tramèr, F. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00179 · Cited by 138
Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks
Pasquini, D., Strohmeier, M., & Troncoso, C. (2024) · AISec@CCS · doi:10.1145/3689932.3694764 · Cited by 134
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428972 · Cited by 120
Get My Drift? Catching LLM Task Drift with Activation Deltas
Abdelnabi, S., Fay, A., Cherubin, G., Salem, A., Fritz, M., & Paverd, A. (2025) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00011 · Cited by 119
Demystifying RCE Vulnerabilities in LLM-Integrated Apps
Liu, T., Deng, Z., Meng, G., Li, Y., & Chen, K. (2024) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3658644.3690338 · Cited by 111
Kendall, G., & Teixeira da Silva, J. A. (2024) · Learned Publishing · doi:10.1002/leap.1578 · Cited by 100
The crisis of peer review: Part of the evolution of science
Horta, H., & Jung, J. (2024) · Higher Education Quarterly · doi:10.1111/hequ.12511 · Cited by 98
Defending Against Prompt Injection With a Few DefensiveTokens
Chen, S., Wang, Y.-M., Carlini, N., Sitawarin, C., & Wagner, D. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762982 · Cited by 85
Quantifying large language model usage in scientific papers
Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02273-8 · Cited by 80
Azher, I. A., Seethi, V. D. R., Akella, A. P., & Alhoori, H. (2024) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3677389.3702605 · Cited by 73
Optimization-based Prompt Injection Attack to LLM-as-a-Judge
Shi, J., Yuan, Z., Liu, Y., Huang, Y., Zhou, P., Sun, L., & Gong, N. Z. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3690291 · Cited by 70
The dangers of using large language models for peer review.
Donker, T. (2023) · Lancet. Infectious Diseases (Print) · doi:10.1016/s1473-3099(23)00290-6 · Cited by 64
Cloak, Honey, Trap: Proactive Defenses Against LLM Agents
Ayzenshteyn, D., Weiss, R., & Mirsky, Y. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.15601739 · Cited by 61
Veldanda, A. K., Grob, F., Thakur, S., Pearce, H. A., Tan, B., Karri, R., & Garg, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05135 · Cited by 55
LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
Abdelnabi, S., Fay, A., Salem, A., Zverev, E., Liao, K.-C., Liu, C. C., Kuo, C. C., Weigend, J., Manlangit, D., Apostolov, A., et al. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.514 · Cited by 55
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Stelmakh, I., Shah, N. B., Singh, A., & Daumé, H. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449149 · Cited by 51
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · arXiv · doi:10.48550/arxiv.2302.12173 · Cited by 50
Friction: Deciphering Writing Feedback into Writing Revisions through LLM-Assisted Reflection
Zhang, C., Ju, K., Bidoshi, P., Yen, Y.-C. G., & Rzeszotarski, J. M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714316 · Cited by 49
ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
Jiang, F.-Q., Xu, Z., Niu, L., Lin, B. Y., & Poovendran, R. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2406.12935 · Cited by 44
ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
Jiang, F., Xu, Z., Niu, L., Lin, B. Y., & Poovendran, R. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i26.34945 · Cited by 41
LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
Abdelnabi, S., Fay, A., Salem, A., Zverev, E., Liao, K.-C., Liu, C., Kuo, C.-C., Weigend, J., Manlangit, D., Apostolov, A., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2506.09956 · Cited by 38
Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment
Shao, Z., Liu, H., Mu, J., & Gong, N. Z. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762963 · Cited by 36
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34
Evaluating and mitigating the impact of OCR errors on information retrieval
de Oliveira, L. L., Vargas, D. S., Alexandre, A. M. A., Cordeiro, F. C., Gomes, D. D. S. M., Rodrigues, M. D. C., Romeu, R. K., & Moreira, V. P. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00345-6 · Cited by 33
Labunets, A., Pandya, N. V., Hooda, A., Fu, X., & Fernandes, E. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00121 · Cited by 29
Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks
Zhou, R., Chen, L., & Yu, K. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/48d359hjdvog · Cited by 28
Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks
Pasquini, D., Strohmeier, M., & Troncoso, C. (2024) · arXiv · doi:10.48550/arxiv.2403.03792 · Cited by 27
How Not to Detect Prompt Injections with an LLM
Choudhary, S., Anshumaan, D., Palumbo, N., & Jha, S. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762980 · Cited by 25
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Hans, A., Schwarzschild, A., Cherepanova, V., Kazemi, H., Saha, A., Goldblum, M., Geiping, J., & Goldstein, T. (2024) · arXiv · doi:10.48550/arxiv.2401.12070 · Cited by 24
Kocak, B., Onur, M. R., Park, S. H., Baltzer, P., & Dietzel, M. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100018 · Cited by 18
ElZemity, A., Arief, B., & Li, S. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762968 · Cited by 18
The Hidden Threat in Plain Text: Attacking RAG Data Loaders
Castagnaro, A., Salviati, U., Conti, M., Pajola, L., & Pizzi, S. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762976 · Cited by 17
Self-citation5W1H Extraction With Large Language Models
Cao, Y., Lan, Y., Zhai, F., & Li, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651056 · Cited by 16
Kadaifçi, Ç., Işıklı, E., & Topçu, Y. İ. (2024) · Learned Publishing · doi:10.1002/leap.1637 · Cited by 16
Hiding in Plain Sight: Tweets with Hate Speech Masked by Homoglyphs
Cooper, P., Surdeanu, M., & Blanco, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.192 · Cited by 15
Defeating Prompt Injections by Design
Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2026) · 2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml68715.2026.00040 · Cited by 15
The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024) · arXiv · doi:10.48550/arxiv.2404.13208 · Cited by 14
To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty
Yadkori, Y. A., Kuzborskij, I., György, A., & Szepesvári, C. (2024) · neural information processing systems · doi:10.52202/079017-1852 · Cited by 11
Dr. Jekyll and Mr. Hyde: Two Faces of LLMsPreprint
Collu, M. G., Janssen-Groesbeek, T., Koffas, S., Conti, M., & Picek, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.03853 · Cited by 8
Self-citationDefending Against Indirect Prompt Injection Attacks With Spotlighting
Hines, K., Lopez, G., Hall, M., Zarfati, F., Zunger, Y., & Kiciman, E. (2024) · arXiv · doi:10.48550/arxiv.2403.14720 · Cited by 8
Hidden Prompts in Manuscripts Exploit AI-Assisted Peer Review
Lin, Z. (2026) · Communications of the ACM · doi:10.1145/3779116 · Cited by 7
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM InferencePreprint
Corro, L. D., Giorno, A. D., Agarwal, S., Yu, B., Awadallah, A. H., & Mukherjee, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02628 · Cited by 7
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O. (2025) · arXiv · doi:10.48550/arxiv.2502.17424 · Cited by 7
Bayesian Domain Invariant Learning via Posterior Generalization of Parameter DistributionsPreprint
Shen, S., Pan, B., Shi, T., Li, T., & Shi, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16277 · Cited by 1
Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review
Yu, S., Luo, M., Madasu, A., Lal, V., & Howard, P. (2024) · arXiv · doi:10.48550/arxiv.2410.03019 · Cited by 1
Howie, J., & Short, H. (2025) · Journal of the London Mathematical Society · doi:10.1112/jlms.70240 · Cited by 1
Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review
Ye, R., Pang, X., Chai, J., Chen, J., Yin, Z., Xiang, Z., Dong, X., Shao, J., & Chen, S. (2024) · arXiv · doi:10.48550/arxiv.2412.01708
The Hidden Threat in Plain Text: Attacking RAG Data Loaders
Castagnaro, A., Salviati, U., Conti, M., Pajola, L., & Pizzi, S. (2025) · arXiv · doi:10.48550/arxiv.2507.05093
Self-citationHowie, J., & Short, H. (2023) · arXiv · doi:10.48550/arxiv.2307.15397
Hidden prompts in manuscripts exploit ai-assisted peer review
Lin, Z. (2025) · arXiv · doi:10.48550/arxiv.2507.06185
Research works printed but not linked: 22
- AAAI. 2025. AAAI Launches AI-Powered Peer Review Assessment System. https://aaai.org/aaai-launches-ai-powered-peer-review-assessment- system/.
- Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. {StruQ}: Defending against prompt injection with structured queries. In 34th USENIX Security Symposium (USENIX Security 25). 2383-2400.
- CVPR. 2025. CVPR 2025 Changes. https://web.archive.org/web/20250418203046/https://cvpr.thecvf.com/Conferences/2025/CVPRChanges.
- ICLR. 2024. ICLR 2024 Reviewer Guide. https://iclr.cc/Conferences/2024/ReviewerGuide.
- ICML Conference. 2026. https://x.com/icmlconf/status/2022760600477073876.
- Tadayoshi Kohno, Yasemin Acar, and Wulf Loh. 2023. Ethical frameworks and computer security trolley problems: Foundations for conversations. In 32nd USENIX Security Symposium (USENIX Security 23). 5145-5162.
- Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. 2024. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. In International Conference on Machine Learning. PMLR, 29575-29620.
- Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24). 1831-1847.
- Yupei Liu, Yuqi Jia, Jinyuan Jia, and Neil Zhenqiang Gong. 2025. Evaluating {LLM-based} Personal Information Extraction and Countermeasures. In 34th USENIX Security Symposium (USENIX Security 25). 1669-1688.
- Jonathan Lorraine. 2024. Getting harsh conference reviews from LLM-powered reviewers? Consider hiding some extra guidance for the LLM in your paper. https://x.com/jonLorraine9/status/1858592201799852115.
- Miryam Naddaf. 2025. Will AI take over peer review? Nature (2025).
- Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tramèr, and Katherine Lee. 2025. Scalable extraction of training data from aligned, production language models. In The Thirteenth International Conference on Learning Representations.
- NeurIPS. 2024. 2024 Reviewer Guidelines. https://neurips.cc/Conferences/2024/ReviewerGuidelines.
- NeurIPS. 2025. NeurIPS25 LLM Policy. https://neurips.cc/Conferences/2025/LLM.
- Pangram. 2025. Pangram Predicts 21% of ICLR Reviews are AI-Generated. https://www.pangram.com/blog/pangram-predicts-21-of-iclr-reviews- are-ai-generated.
- Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2025. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse. In The Thirteenth International Conference on Learning Representations.
- Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2026. A large-scale randomized study of large language model feedback in peer review. Nature Machine Intelligence (2026), 1-11.
- Jiachen T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia. 2025. Data Shapley in One Training Run. In The Thirteenth International Conference on Learning Representations.
- Xunguang Wang, Daoyuan Wu, Zhenlan Ji, Zongjie Li, Pingchuan Ma, Shuai Wang, Yingjiu Li, Yang Liu, Ning Liu, and Juergen Rahmel. 2025. {SelfDefend}:{LLMs} can defend themselves against jailbreaking in a practical manner. In 34th USENIX Security Symposium (USENIX Security 25). 2441-2460.
- Yanting Wang, Wei Zou, Runpeng Geng, and Jinyuan Jia. 2025. {TracLLM}: A Generic Framework for Attributing Long Context {LLMs}. In 34th USENIX Security Symposium (USENIX Security 25). 3845-3864.
- jianying Zhou. 2025. Top Cyber Security Conferences Ranking. http://jianying.space/conference-ranking.html.
- James Zou and Nitya Thakkar. 2025. Leveraging LLM feedback to enhance review quality. https://web.archive.org/web/20250510123847/https: //blog.iclr.cc/2025/04/15/leveraging-llm-feedback-to-enhance-review-quality/ 1 2306.05880v5 https://openreview.net/forum?id=HNdp1ltDG_ 2 2306.07290v1 https://openreview.net/forum?id=TeeyHEi25C 3 2306.09212v2 https://openreview.net/forum?id=ck4SG9lnrQ 4 2307.02628v1 https://openreview.net/forum?id=bcHty5VvkQ 5 2308.12044v5 https://openreview.net/forum?id=7MlOI37rbn 6 2309.16515v3 https://openreview.net/forum?id=ro4CgvfUKy 7 2309.17144v1 https://openreview.net/forum?id=qW9GVa3Caa 8 2310.00212v3 https://openreview.net/forum?id=JzAuFCKiov 9 2310.05755v1 https://openreview.net/forum?id=veIzQxZUhF 10 2310.06177v1 https://openreview.net/forum?id=wmq67R2PIu 11 2310.13033v2 https://openreview.net/forum?id=TCJbcjS0c2 12 2310.15149v1 https://openreview.net/forum?id=EraNITdn34 13 2310.16277v1 https://openreview.net/forum?id=d2TOOGbrtP 14 2311.00267v1 https://openreview.net/forum?id=7v3tkQmtpE 15 2311.01729v2 https://openreview.net/forum?id=jYHRP6nj9Q 16 2311.04166v2 https://openreview.net/forum?id=YkEW5TabYN 17 2311.18054v2 https://openreview.net/forum?id=eKGEsFdpin 18 2312.00249v2 https://openreview.net/forum?id=rAX55lDjtt 19 2402.03545v3 https://openreview.net/forum?id=sFQe52N40m 20 2402.06220v1 https://openreview.net/forum?id=lWXedJyLuL 21 2404.06694v2 https://openreview.net/forum?id=5ZWxBU9sYG 22 2405.02766v1 https://openreview.net/forum?id=Pa6SiS66p0 23 2406.03665v1 https://openreview.net/forum?id=alnvAZGWLD 24 2412.09968v1 https://openreview.net/forum?id=tB7p0SM5TH 25 2412.12232v1 https://openreview.net/forum?id=gqtbL7j2JW D.2 Experimental Results We report the detailed per-paper results in Tables 17, 18, 19, 20 (providing the specific results for R0, R1, R2, R3, respectively). We report in Figure 5 and Table 15 the results achieved by GPT-4o on the two papers (i.e., [32, 77]) used also for the assessment of GPT-o3 and Gemini-2.5-flash (for which we also report some results in Figure 6). We report in Table 16 the detailed results of our assessment in §8.1, wherein we "compare" the prompts found/proposed in prior work with ours (these results should be compared with those of Exploit in Table 15). Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
Cited by
The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to AnalysisPreprint
Wang, P., Li, X., Xiang, C., Zhang, J., Li, Y., Zhang, L., Wang, X., & Tian, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10453 · Cited by 47
Autonomous Research Agents: A Survey of AI Scientists and the Verification GapPreprint
Ding, T., Nannapaneni, A., Liu, B., & Zhang, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.05179 · Cited by 23
Jailbreaking LLMs: A Survey of Attacks, Defenses and EvaluationPreprint
Hakim, S. B., Gharami, K., Ghalaty, N. F., Moni, S. S., Xu, S., & Song, H. (2026) · doi:10.36227/techrxiv.176773228.86819800/v1 · Cited by 15
When AI reviews science: Can we trust the referee?
Wang, J., Liu, Y., Xu, H., Hu, K., Di, S., Ni, W., Yue, L., Zhang, M.-L., Ren, K., & Chen, L. (2026) · The Innovation Informatics · doi:10.59717/j.xinn-inform.2026.100030 · Cited by 10
Torrielli, F., Locci, S., Rapp, A., & Di Caro, L. (2026) · Scientometrics · doi:10.1007/s11192-026-05695-x · Cited by 8
Metadata
- Authors
- Collu, M. G., Salviati, U., Confalonieri, R., Conti, M., & Apruzzese, G.
- Affiliations
- Örebro University (SE)
- University of Padua (IT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2508.20863
- OpenAlex
- W4415986829
- Cited by
- 22
- References
- 62
- Citations by year
- 2026: 19 · 2025: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (cc-by-nc-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- large language models, peer review, adversarial prompts, threat model, PDF embedding, reviewer bias, scholarly publishing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-20, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-20 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.