CoChat
Sign Up
PreprintCited by 20Privacy-Preserving Technologies in DataAdversarial Robustness in Machine LearningDomain Adaptation and Few-Shot Learning

In-Context Unlearning: Language Models as Few Shot Unlearners

Open access
Pawelczyk, Martin, Neel, Seth, Lakkaraju, Himabindu·2023·arXiv
DOI 10.48550/arxiv.2310.07579arXiv 2310.07579
Open this paper in CoChat — free accountOpen PDF

Abstract

Machine unlearning, the study of efficiently removing the impact of specific training instances on a model, has garnered increased attention in recent years due to regulatory guidelines such as the \emph{Right to be Forgotten}. Achieving precise unlearning typically involves fully retraining the model and is computationally infeasible in case of very large models such as Large Language Models (LLMs). To this end, recent work has proposed several algorithms which approximate the removal of training data without retraining the model. These algorithms crucially rely on access to the model parameters in order to update them, an assumption that may not hold in practice due to computational constraints or having only query access to the LLMs. In this work, we propose a new class of unlearning methods for LLMs called ``In-Context Unlearning.'' This method unlearns instances from the model by simply providing specific kinds of inputs in context, without the need to update model parameters. To unlearn specific training instances, we present these instances to the LLMs at inference time along with labels that differ from their ground truth. Our experimental results demonstrate that in-context unlearning performs on par with, or in some cases outperforms other state-of-the-art methods that require access to model parameters, effectively removing the influence of specific instances on the model while preserving test accuracy.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.07579

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ICUL can effectively eliminate the influence of training points on the model's output, occasionally outperforming white-box methods that necessitate direct access to LLM parameters and are more computationally demanding.

    “The results demonstrate that ICUL can effectively eliminate the influence of training points on the model's output, occasionally outperforming white-box methods that necessitate direct access to the LLM parameters and are more computationally demanding.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Pawelczyk et al 2023

  • A few authors use the success of a Membership Inference Attack (MIA) as a metric.

    “Membership Inference Attack (MIA) : A few authors (Wang et al 2023; Chen and Yang 2023; Pawelczyk et al 2023) use the success of a MIA as a metric.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Pawelczyk et al 2023

  • ICUL was empirically evaluated on three classification datasets using a test called LiRAForget.

    “The ICUL method was empirically evaluated on three classification datasets using a test called LiRAForget, which was also introduced to empirically measure unlearning effectiveness.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Pawelczyk et al 2023

  • ICUL constructs a specific context at inference time that makes the model classify a training point as if it had never seen it during training.

    “To remove the impact of a specific training point on the model's output, ICUL (Pawelc -zyk et al 2023 ) constructs a specific context at inference time that makes the model classify it as if it had never seen the data point during training.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Pawelczyk et al 2023

  • Jang et al (2022) and Pawelczyk et al (2023) show that larger models tend to exhibit better utility retention.

    “Jang et al (2022) and Pawelczyk et al (2023) show that larger models tend to exhibit better utility retention.”

    From Digital forgetting in large language models: a survey of unlearning methods (2025) · cites this paper as Pawelczyk et al (2023)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 53

  1. Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, S. Leace: Perfect lin- ear concept erasure in closed form. arXiv preprint arXiv:2306.03819, 2023. · arXiv
  2. Biega, A. J., Potash, P., Daumé, H., Diaz, F., and Finck, M. Operationalizing the legal principle of data minimization for personalization. In ACM(43) SIGIR '20, pp. 399-408, 2020.
  3. Borisov, V., Seßler, K., Leemann, T., Pawelczyk, M., and Kasneci, G. Language models are realistic tabular data generators. In International Conference on Learning Representations (ICLR), 2023.
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
  5. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lund- berg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712, 2023. · arXiv
  6. Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first prin- ciples. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
  7. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. In International Conference on Learn- ing Representations (ICLR), 2023.
  8. Cook, R. D. and Weisberg, S. Characterizations of an em- pirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-508, 1980.
  9. Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486, 2023. · arXiv
  10. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey for in-context learning. arXiv:2301.00234, 2023. · arXiv
  11. Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  12. Ginart, A., Guan, M. Y., Valiant, G., and Zou, J. Making ai forget you: Data deletion in machine learning. In Pro- ceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019.
  13. Goel, S., Prabhu, A., Sanyal, A., Lim, S.-N., Torr, P., and Ku- maraguru, P. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint arXiv:2201.06640, 2022. · arXiv
  14. Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
  15. Golatkar, A., Achille, A., and Soatto, S. Forgetting outside the box: Scrubbing deep networks of information acces- sible from input-output observations. arXiv:2003.02960, 2020b. · arXiv
  16. Golatkar, A., Achille, A., Ravichandran, A., Polito, M., and Soatto, S. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 792-801, 2021.
  17. Goldsteen, A., Ezov, G., Shmelkin, R., Moffie, M., and Farkash, A. Data minimization for gdpr compliance in machine learning models. AI and Ethics, pp. 1-15, 2021.
  18. Graves, L., Nagisetty, V., and Ganesh, V. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  19. Guo, C., Goldstein, T., Hannun, A., and Van Der Maaten, L. Certified data removal from machine learning models. In International Conference on Machine Learing (ICML), 2019.
  20. Henderson, P., Li, X., Jurafsky, D., Hashimoto, T., Lemley, M. A., and Liang, P. Foundation models and fair use. arXiv:2303.15715, 2023. · arXiv
  21. Huang, Y. and Canonne, C. L. Tight bounds for machine unlearning via differential privacy. arXiv:2309.00886, 2023. · arXiv
  22. Izzo, Z., Anne Smart, M., Chaudhuri, K., and Zou, J. Ap- proximate data deletion from machine learning models. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
  23. Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al. Measuring forgetting of memorized training examples. In International Conference on Learning Rep- resentations (ICLR), 2023.
  24. Jang, J., Yoon, D., Yang, S., Cha, S., Lee, M., Logeswaran, L., and Seo, M. Knowledge unlearning for mitigating pri- vacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023.
  25. Kurmanji, M., Triantafillou, P., and Triantafillou, E. To- wards unbounded machine unlearning. arXiv preprint arXiv:2302.09880, 2023. · arXiv
  26. Leemann, T., Pawelczyk, M., and Kasneci, G. Gaussian membership inference privacy. In Advances in neural information processing systems (NeurIPS), 2023.
  27. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2022.
  28. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55, 2023.
  29. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048- 11064, 2022.
  30. Morris, J. X., Zhao, W., Chiu, J. T., Shmatikov, V., and Rush, A. M. Language model inversion. arXiv preprint arXiv:2311.13647, 2023. · arXiv
  31. Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023. · arXiv
  32. Neel, S., Roth, A., and Sharifi-Malvajerdi, S. Descent-to- delete: Gradient-based methods for machine unlearning. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory (ALT), 2021.
  33. Neyman, J. and Pearson, E. S. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophi- cal Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Char- acter, 231(694-706):289-337, 1933.
  34. Pang, B. and Lee, L. Seeing stars: Exploiting class relation- ships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005. · arXiv
  35. Pawelczyk, M., Leemann, T., Biega, A., and Kasneci, G. On the trade-off between actionable explanations and the right to be forgotten. In International Conference on Learning Representations (ICLR), 2023.
  36. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  37. In Su, J., Duh, K., and Carreras, X. (eds.), Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https: //aclanthology.org/D16-1264. · DOI · Paper page
  38. Ravfogel, S., Twiton, M., Goldberg, Y., and Cotterell, R. D. Linear adversarial concept erasure. In International Con- ference on Machine Learning, pp. 18400-18421.
  39. PMLR, 2022a.
  40. Ravfogel, S., Vargas, F., Goldberg, Y., and Cotterell, R. Ad- versarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6034-6055, 2022b.
  41. Sekhari, A., Acharya, J., Kamath, G., and Suresh, A. T. Remember what you want to forget: Algorithms for ma- chine unlearning. In Advances in Neural Information Processing Systems, 2021.
  42. Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
  43. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013.
  44. Tirumala, K., Markosyan, A., Zettlemoyer, L., and Agha- janyan, A. Memorization without overfitting: Analyzing the training dynamics of large language models. In Ad- vances in Neural Information Processing Systems, 2022.
  45. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023. · arXiv
  46. Tsybakov, A. B. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.
  47. Union, E. Regulation (eu) 2016/679 of the european parlia- ment and of the council. Official Journal of the European Union, 2016.
  48. Voigt, P. and Von dem Bussche, A. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10:3152676, 2017.
  49. Wang, L., Chen, T., Yuan, W., Zeng, X., Wong, K.-F., and Yin, H. Kga: A general machine unlearning frame- work based on knowledge gap alignment. arXiv preprint arXiv:2305.06535, 2023. · arXiv
  50. Wei, J., Wei, J., Tay, Y., Tran, D., Webson, A., Lu, Y., Chen, X., Liu, H., Huang, D., Zhou, D., et al. Larger language models do in-context learning differently. arXiv:2303.03846, 2023. · arXiv
  51. Wu, Y., Dobriban, E., and Davidson, S. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning (ICML), 2020.
  52. Zhang, R. and Zhang, S. Rethinking influence functions of neural networks in the over-parameterized regime. In Pro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
  53. Zhang, X., Zhao, J., and LeCun, Y. Character-level con- volutional networks for text classification. In Advances in neural information processing systems (NeurIPS), vol- ume 28, 2015.

Cited by

Sorted by citations

Authors
Pawelczyk, M., Neel, S., & Lakkaraju, H.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.07579
arXiv
2310.07579
OpenAlex
W4387634446
Cited by
20
References
0
Citations by year
2026: 2 · 2025: 11 · 2024: 6 · 2023: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2008461, 2238714, 2040989)
  • Harvard Data Science Initiative, Harvard University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
machine unlearning, large language models, right to be forgotten

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review