Many-shot Jailbreaking
Abstract
We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This is newly feasible with the larger context windows recently deployed by Anthropic, Ope-nAI and Google DeepMind. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even alignment-trained refusals are not inviolable.
“even these alignment-trained refusals are not inviolable (Hubinger et al. 2024; Anil et al. 2024)”
From Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents (2026) · cites this paper as Anil et al. 2024
Prior works have found that the negative log-likelihood (NLL) for ground-truth test outputs decreases predictably as the context length increases.
“Prior works [68, 2, 27] have found that the negative log-likelihood (NLL) for ground-truth test outputs decreases predictably as the context length increases.”
From Many-Shot In-Context Learning (2024) · cites this paper as [2]
Our results demonstrate that we can override pre-training biases themselves.
“Furthermore, while Anil et al. [2] use many-shot learning to override preferences learned during RLHF phase to elicit the biases stemming from pretraining, our results in §4.1 demonstrate that we can also override pre-training biases themselves.”
From Many-Shot In-Context Learning (2024) · cites this paper as [2]
Anil et al. used many-shot prompting up to 256 shots to jailbreak language models.
“Concurrently to our work, Anil et al. [2] used many-shot prompting (upto 256 shots) to jailbreak language models.”
From Many-Shot In-Context Learning (2024) · cites this paper as [2]
Prior works have reported smaller next-token prediction loss with longer contexts, and Jeon et al. also show this using theoretical analysis.
“Prior works [68, 2, 27, 16] have reported smaller next-token prediction loss with longer contexts, which Jeon et al. [25] also show using theoretical analysis.”
From Many-Shot In-Context Learning (2024) · cites this paper as [2]
Concurrent works and our preliminary results with GPT-4-Turbo, Gemini 1.5 Flash, and Claude-3-Opus indicate that other LLMs can also benefit from many-shot ICL.
“That said, concurrent works [2, 6] as well as our preliminary results with GPT-4-Turbo, Gemini 1.5 Flash, and Claude-3-Opus (§4.5) indicate that other LLMs can also benefit from many-shot ICL.”
From Many-Shot In-Context Learning (2024) · cites this paper as [2]
Many existing long-context language modeling benchmarks are studied in isolated settings, such as synthetic tasks, in-context learning, summarization, and retrieval-augmented settings.
“Many existing long-context language modeling benchmarks are studied in isolated settings, such as synthetic tasks (Hsieh et al., 2024; Li et al., 2024b; Levy et al., 2024; Laban et al., 2024), in-context learning (Anil et al., 2024; Bertsch et al., 2024; Li et al., 2024c; Agarwal et al., 2024), summarization (Chang et al., 2024; Kim et al., 2024), and retrieval-augmented settings (Lee et al., 2024”
From HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly (2024) · cites this paper as Anil et al., 2024
In-context learning can enable overcoming safety fine-tuning.
“or the ability to overcome safety fine-tuning (Anil et al., 2024;Zheng et al., 2024).”
From Machine Psychology (2023) · cites this paper as Anil et al., 2024
The use of a large number of examples has even greater potential to override large language models' safeguards.
“Recent work has also shown that the use of a large number (e.g. several hundred) of examples has even greater potential to override LLMs' safeguards [100].”
From Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated (2025) · cites this paper as [100]
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Generative language models exhibit social identity biases
Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., van der Linden, S., & Roozenbeek, J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00741-1 · Cited by 100 · see citing works
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3660147 · Cited by 51 · see citing works
Technological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51 · see citing works
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2025) · arXiv · doi:10.48550/arxiv.2507.19218 · Cited by 26 · see citing works
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.11018 · Cited by 35 · see citing works
“Furthermore, while Anil et al. [2] use many-shot learning to override preferences learned during RLHF phase to elicit the biases stemming from pretraining, our results in §4.1 demonstrate that we can also override pre-training biases themselves.”
InfluentialRedAgent: An Autonomous Agent for Context-Aware Red Teaming of LLM Jailbreaks
Xu, H., Zhang, W., Wang, Z., Xiao, F., Zheng, R., Ba, Z., & Ren, K. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3665230 · Cited by 3 · see citing works
Sabotage Evaluations for Frontier Models
Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., Durmus, E., Ganguli, D., Kravec, S., Shlegeris, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.21514 · Cited by 2 · see citing works
Self-citationHELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
Yen, H., Gao, T., Hou, M., Ding, K., Fleischer, D., Izsak, P., Wasserblat, M., & Chen, D. (2024) · arXiv · Cited by 2 · see citing works
“…answering (Wang et al., 2024b; Karpinska et al., 2024; Wang et al., 2024a), in-context learning (Li et al., 2024c; Bertsch et al., 2024; Xu et al., 2024; Anil et al., 2024; Agarwal et al., 2024), summarization (Chang et al., 2024; Kim et al., 2024; Shen et al., 2022), or RAG (Lee et al., 2024).”
BackgroundAgentic AI Safety: A Structured Review of Open Problems and Their Regulatory Anchoring
Valenta, T., Rozinek, O., & Horálek, J. (2026) · Applied Informatics · doi:10.3390/ai7080298 · Cited by 1 · see citing works
Position Paper: Bounded Alignment: What (Not) To Expect From AGI Agents
Minai, A. A. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228305
PRO-GO: Reference-Guided Protein Sequence Generation Using Gene Ontology Terms
Tan, D., McLoughlin, I., Ng, A. B., Wang, Z., Stern, A. C., & See, S. C. W. (2025) · IEEE Access · doi:10.1109/access.2025.3646961
“N S = 100, but in this experiment we alter the number of reference protein sequences, N R = { 1 , 2 , 3 , 4 , 5 } where N R = 5 is the practical upper limit, constrained by the model effective context window [28], [29].”
Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
Wei, W., Shang, Z., & Bai, W. (2026) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa67686.2026.11467846 · Cited by 9 · see citing works
Does Safety Molt? Evaluating LLM Safety in Multi-Agent Social Environments
Priyanshu, A., Vijay, S., & Pahwa, E. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813173
Chen, M., & Yao, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7067669
Evolva: A Multi-turn Contextual Attack for Long-Reasoning LLMs
Du, W., Yang, X., Liu, Z., Zhang, B., & Xue, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_46
SecPrompt v2: Protect LLM-Integrated Applications from Adversarial Attack
Zhang, J., Wu, J., Lu, Z., Ma, L., Xie, J., Yang, B., Zhang, W., & Huang, Q. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3541-4_4
Kalinich, M., Luccarelli, J., Maria, J. P. S., Flathers, M., Nguyen, A., Song, S. H., Makhoul, K. G., Criado, M. J. R., Ginapp, C. M., Hill, B., et al. (2026) · medRxiv · doi:10.64898/2026.07.10.26357132
Ding, F., Liao, Z., & Li, Q. (2026) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai69603.2026.11642016
“Techniques like Many-shot Jailbreaking [31], persona modulation [32], and persuasive paraphrasing [33] induce harmful responses through context manipulation.”
A C-SSRS-Based Risk-Aware Evaluation Framework for Self-Harm Safety in LLMs
Zhang, L., Zhao, F., & Yang, X. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00035
“Beyond AITest, research on automated red teaming has mainly developed along two directions: benchmark-driven safety evaluation and automated jailbreak generation [1], [3]–[9], [11], [21]–[30].”
The Grammar of the Metric: A Linguistic Typology of Performance-Measurement Blind SpotsPreprint
Magni, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7368499
Yuxuan, Z., & Hussain, W. M. H. W. (2026) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2026.101629
Shining Shadows: Latent Semantics in Human and Artificial IntelligencePreprint
Magni, L. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7422138
Cross-layer manifold attack and projection for robust safety alignment of large language models
Peng, Z., Du, J., Zhou, Q., et al. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.135082
Evaluating the Robustness of Safety-Aligned LLM Behavior to Short Contextual PrefixesPreprint
Cholewa, M., & Głomb, P. (2026) · Research Square · doi:10.21203/rs.3.rs-10590385/v1
Chiang, E., Calloway, L. R., Elliott, J., & Grant, T. (2026) · AI & Society · doi:10.1007/s00146-026-03334-w
Measuring Reproducibility in LLM Research: A Rubric and Cross-Topic Corpus Study
Chong, C. J., Hou, C., Liu, S., & Yao, Z. (2026) · ACM Conference on Reproducibility and Replicability (ACM REP) · doi:10.1145/3820002.3828584
Datura: Progressive Red Teaming Testing for Tool Invocation Chain in LLM Agents
Shao, Y., Bao, Z., Huang, Y., Shi, Y., Weng, M., Sun, Y., Yang, L., Ma, L., Su, T., & Wan, C. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832101
The Fabricated Diary: A New Form of Jailbreaking
Gilly, T. (2026) · doi:10.68481/djpem7
Yi, Z., Yang, Z., Liu, Q., Ni, R., Han, S., Li, C., Zhang, Z., Xu, P., & Ruan, W. (2026) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-026-00252-0
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
- Authors
- Anil, C., Durmus, E., Panickssery, N., Sharma, M., Benton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D., Mosconi, F., Agrawal, R., Schaeffer, R., Bashkansky, N., Svenningsen, S., Lambert, M., Radhakrishnan, A., Denison, C., Hubinger, E., Bai, Y., Bricken, T., Maxwell, T., Schiefer, N., Sully, J., Tamkin, A., Lanhan, T., Nguyen, K., Korbak, T., Kaplan, J., Ganguli, D., Bowman, S., Perez, E., Grosse, R., & Duvenaud, D.
- ORCID
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 129696-129742
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W4415800797
- Semantic Scholar
- af5633c94ed2beb282f6a53c595eb437e8e7b630
- Cited by
- 25
- References
- 0
- Citations by year
- 2026: 23 · 2025: 2
- FWCI
- 7.38
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
- Physical Sciences › Computer Science › Signal Processing › Advanced Malware Detection Techniques
- Keywords
- jailbreaking, few-shot learning, adversarial attacks, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
