Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Open accessAbstract
Watermarking generative models consists of planting a statistical signal (watermark) in a model's output so that it can be later verified that the output was generated by the given model. A strong watermarking scheme satisfies the property that a computationally bounded attacker cannot erase the watermark without causing significant quality degradation. In this paper, we study the (im)possibility of strong watermarking schemes. We prove that, under well-specified and natural assumptions, strong watermarking is impossible to achieve. This holds even in the private detection algorithm setting, where the watermark insertion and detection algorithms share a secret key, unknown to the attacker. To prove this result, we introduce a generic efficient watermark attack; the attacker is not required to know the private key of the scheme or even which scheme is used. Our attack is based on two assumptions: (1) The attacker has access to a "quality oracle" that can evaluate whether a candidate output is a high-quality response to a prompt, and (2) The attacker has access to a "perturbation oracle" which can modify an output with a nontrivial probability of maintaining quality, and which induces an efficiently mixing random walk on high-quality outputs. We argue that both assumptions can be satisfied in practice by an attacker with weaker computational capabilities than the watermarked model itself, to which the attacker has only black-box access. Furthermore, our assumptions will likely only be easier to satisfy over time as models grow in capabilities and modalities. We demonstrate the feasibility of our attack by instantiating it to attack three existing watermarking schemes for large language models: Kirchenbauer et al. (2023), Kuditipudi et al. (2023), and Zhao et al. (2023). The same attack successfully removes the watermarks planted by all three schemes, with only minor quality degradation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
No watermark is immune to obfuscation through paraphrasing.
“Nevertheless, authors of a recent paper claim they have proved that no watermark is immune to obfuscation through paraphrasing (Zhang et al., 2023).”
From Generative AI detection in higher education assessments (2025) · cites this paper as Zhang et al., 2023
The “Limitations of generative model watermarks” section says paraphrasing attacks have had mixed success and explicitly distinguishes the authors’ attack because it does not preserve meaning. The appendix section on quality-preserving perturbations adds that removing a watermark may require changing semantic details. The paper therefore does not claim to have proved that paraphrasing can obfuscate every watermark. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Self-citationHiDDeN: Hiding Data With Deep Networks
Zhu, J., Kaplan, R., Johnson, J., & Fei-Fei, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01267-0_40 · Cited by 1,186 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
Embedding Watermarks into Deep Neural Networks
Uchida, Y., Nagai, Y., Sakazawa, S., & Satoh, S. (2017) · International Conference on Multimedia Retrieval · doi:10.1145/3078971.3078974 · Cited by 709 · see citing works
Protecting Intellectual Property of Deep Neural Networks with Watermarking
Zhang, J., Gu, Z., Jang, J., Wu, H., Stoecklin, M. P., Huang, H., & Molloy, I. (2018) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3196494.3196550 · Cited by 556 · see citing works
StegaStamp: Invisible Hyperlinks in Physical Photographs
Tancik, M., Mildenhall, B., & Ng, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00219 · Cited by 491 · see citing works
Digital watermarks for audio signals
Boney, L., Tewfik, A. H., & Hamdy, K. N. (1996) · IEEE International Conference on Multimedia Computing and Systems (ICMCS) · doi:10.1109/mmcs.1996.535015 · Cited by 472 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · see citing works
Rotation, scale and translation invariant digital image watermarking
O'Ruanaidh, J. J., & Pun, T. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.1997.647968 · Cited by 363 · see citing works
ELI5: Long Form Question Answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · arXiv · doi:10.48550/arxiv.1907.09190 · Cited by 325 · see citing works
RNN-Stega: Linguistic Steganography Based on Recurrent Neural Networks
Yang, Z., Guo, X., Chen, Z.-M., Huang, Y., & Zhang, Y. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2871746 · Cited by 323 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
On the Reliability of Watermarks for Large Language ModelsPreprint
Kirchenbauer, J., Geiping, J., Wen, Y., Shu, M., Saifullah, K., Kong, K., Fernando, K., Saha, A., Goldblum, M., & Goldstein, T. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2306.04634 · Cited by 257 · see citing works
Natural Language Watermarking and Tamperproofing
Atallah, M., Raskin, V., Hempelmann, C. F., et al. (2002) · Information Hiding · doi:10.1007/3-540-36415-3_13 · Cited by 222
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07327 · Cited by 218 · see citing works
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Wong, D. F., & Chao, L. S. (2023) · arXiv · doi:10.48550/arxiv.2310.14724 · Cited by 212 · see citing works
A Watermark for Large Language Models
Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023) · arXiv · doi:10.48550/arxiv.2301.10226 · Cited by 194 · see citing works
CLIPScore: A Reference-free Evaluation Metric for Image CaptioningPreprint
Hessel, J., Holtzman, A., Forbes, M., Bras, R. J. L., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08718 · Cited by 174 · see citing works
Can AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 161 · see citing works
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2301.11305 · Cited by 151 · see citing works
The Curse of Recursion: Training on Generated Data Makes Models Forget
Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., & Anderson, R. (2023) · arXiv · doi:10.48550/arxiv.2305.17493 · Cited by 151 · see citing works
Watermarking digital images for copyright protection
Boland, F. (1995) · IEE International Conference on Image Processing and its Applications (IPA) · doi:10.1049/cp:19950674 · Cited by 151 · see citing works
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2023) · arXiv · doi:10.48550/arxiv.2303.13408 · Cited by 150 · cited 1× · see citing works
“Empirically, paraphrasing attacks work well against post-hoc detection algorithms for LLMs [28, 30].”
MethodShow Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144 · see citing works
DeepSigns: A Generic Watermarking Framework for IP Protection of Deep Learning ModelsPreprint
Rouhani, B. D., Chen, H., & Koushanfar, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.00750 · Cited by 80 · see citing works
Supervised GAN Watermarking for Intellectual Property Protection
Fei, J., Xia, Z., Tondi, B., & Barni, M. (2022) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs55849.2022.9975409 · Cited by 63 · see citing works
Invisible Image Watermarks Are Provably Removable Using Generative AI
Zhao, X., Zhang, K., Su, Z., Vasan, S., Grishchenko, I., Kruegel, C., Vigna, G., Wang, Y.-X., & Li, L. (2024) · neural information processing systems · doi:10.52202/079017-0276 · Cited by 61 · see citing works
On the Possibilities of AI-Generated Text Detection
Chakraborty, S., Bedi, A. S., Zhu, S., An, B., Manocha, D., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.04736 · Cited by 59 · see citing works
GPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2304.02819 · Cited by 40 · see citing works
Embarrassingly Simple Text Watermarks
Sato, R., Takezawa, Y., Bao, H., Niwa, K., & Yamada, M. (2023) · arXiv.org · doi:10.48550/arxiv.2310.08920 · Cited by 35
Provable Robust Watermarking for AI-Generated Text
Zhao, X., Ananth, P., Li, L., & Wang, Y.-X. (2023) · arXiv · doi:10.48550/arxiv.2306.17439 · Cited by 30 · see citing works
A Recipe for Watermarking Diffusion ModelsPreprint
Zhao, Y., Pang, T., Du, C., Yang, X. P., Cheung, N. M., & Lin, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.10137 · Cited by 29 · see citing works
Spear Phishing With Large Language Models
Hazell, J. (2023) · arXiv · doi:10.48550/arxiv.2305.06972 · Cited by 28 · see citing works
Undetectable Watermarks for Language Models
Christ, M., Gunn, S., & Zamir, O. (2023) · arXiv · doi:10.48550/arxiv.2306.09194 · Cited by 27 · see citing works
Robust distortion-free watermarks for language models
Kuditipudi, R., Thickstun, J., Hashimoto, T., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.15593 · Cited by 23 · see citing works
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., & Li, H. (2023) · arXiv · doi:10.48550/arxiv.2306.09341 · Cited by 22 · see citing works
Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and RobustPreprint
Wen, Y., Kirchenbauer, J., Geiping, J., & Goldstein, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.20030 · Cited by 22 · see citing works
Ghostbuster: Detecting Text Ghostwritten by Large Language Models
Verma, V., Fleisig, E., Tomlin, N., & Klein, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15047 · Cited by 16 · see citing works
Bot or Human? Detecting ChatGPT Imposters with A Single QuestionPreprint
Wang, H., Luo, X., Wang, W., & Yan, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06424 · Cited by 14 · see citing works
Robustness of AI-Image Detectors: Fundamental Limits and Practical AttacksPreprint
Saberi, M., Sadasivan, V. S., Rezaei, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.00076 · Cited by 4 · see citing works
Neural Linguistic SteganographyPreprint
Ziegler, Z. M., Deng, Y., & Rush, A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01496 · Cited by 4 · see citing works
Securing Deep Generative Models with Universal Adversarial SignaturePreprint
Zeng, Y., Zhou, M., Xue, Y., & Patel, V. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16310 · Cited by 3 · see citing works
Leveraging Optimization for Adaptive Attacks on Image WatermarksPreprint
Lukas, N., Diaa, A., Fenaux, L., & Kerschbaum, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.16952 · Cited by 2 · see citing works
NP is as easy as detecting unique solutions
Valiant, L. G., & Vazirani, V. V. (1985) · Proceedings of the seventeenth annual ACM symposium on Theory of computing - STOC '85 · doi:10.1145/22145.22196
Generating Steganographic Text with LSTMs
Fang, T., Jaggi, M., & Argyraki, K. (2017) · arXiv · doi:10.48550/arxiv.1705.10742
The Stable Signature: Rooting Watermarks in Latent Diffusion Models
Fernandez, P., Couairon, G., Jégou, H., Douze, M., & Furon, T. (2023) · arXiv · doi:10.48550/arxiv.2303.15435
Adversarial Diffusion Distillation
Sauer, A., Lorenz, D., Blattmann, A., & Rombach, R. (2023) · arXiv · doi:10.48550/arxiv.2311.17042
Flagged first, then by citations
Research works printed but not linked: 15
- Michael Arnold. Audio watermarking: Features, applications and algorithms. In 2000 IEEE International conference on multimedia and expo. ICME2000. Proceedings. Latest advances in the fast changing world of multimedia (cat. no. 00TH8532), volume 2, pages 1013-1016. IEEE, 2000.
- Mikhail J Atallah, Victor Raskin, Michael Crogan, Christian Hempelmann, Florian Kerschbaum, Dina Mohamed, and Sanket Naik. Natural language watermarking: Design, analysis, and a proof- of-concept implementation. In Information Hiding: 4th International Workshop, IH 2001 Pittsburgh, PA, USA, April 25-27, 2001 Proceedings 4, pages 185-200. Springer, 2001.
- Arpit Bansal, Ping-yeh Chiang, Michael J Curry, Rajiv Jain, Curtis Wigington, Varun Manjunatha, John P Dickerson, and Tom Goldstein. Certified neural network watermarks with randomized smoothing. In International Conference on Machine Learning, pages 1450-1465. PMLR, 2022.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- Jamie Hayes and George Danezis. Generating steganographic images via adversarial training. Advances in neural information processing systems, 30, 2017.
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
- Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. Transactions on Machine Learning Research, 2022.
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
- Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
- Mika Westerlund. The emergence of deepfake technology: A review. Technology innovation manage- ment review, 9(11), 2019.
- Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
- Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision, pages 14448-14457, 2021.
- Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
Cited by
Sorted by citations
Can AI-Generated Text be Reliably Detected?
Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11156 · Cited by 161 · see citing works
Generative AI detection in higher education assessments
Ardito, C. G. (2025) · New Directions for Teaching and Learning · doi:10.1002/tl.20624 · Cited by 86 · see citing works
Chan, A., Ezell, C., Kaufmann, M., Wei, K., Hammond, L., Bradley, H., Bluemke, E., Rajkumar, N., Krueger, D., Kolt, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658948 · Cited by 84 · see citing works
National Institute of Standards and Technology (US) (2024) · doi:10.6028/nist.ai.100-4 · Cited by 9 · see citing works
Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text
Cheng, Y., Sadasivan, V. S., Saberi, M., Saha, S., & Feizi, S. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-1588 · Cited by 3 · see citing works
- Authors
- Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B.
- Affiliations
- Harvard University (US)
- George Mason University (US)
- Sapienza University of Rome (IT)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2311.04378
- OpenAlex
- W4388555465
- Cited by
- 12
- References
- 50
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134157)
- U.S. Department of Energy (DESC0022199)
- Accenture
- European Commission (PE00000014)
- Defense Advanced Research Projects Agency (W911NF2010021, DE-SC0022199)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Steganography and Watermarking Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Cryptography and Data Security
- Keywords
- watermarking, generative models, large language models, watermark removal, impossibility results, black-box attacks, random walks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
