Knockoff Nets: Stealing Functionality of Black-Box Models
Open accessAbstract
Machine Learning (ML) models are increasingly deployed in the wild to perform a wide range of tasks. In this work, we ask to what extent can an adversary steal functionality of such ``victim'' models based solely on blackbox interactions: image in, predictions out. In contrast to prior work, we study complex victim blackbox models, and an adversary lacking knowledge of train/test data used by the model, its internals, and semantics over model outputs. We formulate model functionality stealing as a two-step approach: (i) querying a set of input images to the blackbox model to obtain predictions; and (ii) training a ``knockoff'' with queried image-prediction pairs. We make multiple remarkable observations: (a) querying random images from a different distribution than that of the blackbox training data results in a well-performing knockoff; (b) this is possible even when the knockoff is represented using a different architecture; and (c) our reinforcement learning approach additionally improves query sample efficiency in certain settings and provides performance gains. We validate model functionality stealing on a range of datasets and tasks, as well as show that a reasonable knockoff of an image analysis API could be created for as little as $30.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Loss of privacy has been compromised by DL methods even without data fusion; a few images can threaten privacy despite obfuscation, and DNN model parameters can be exposed via input queries.
“Notwithstanding this explicit concern from regulatory bodies, loss of privacy has been compromised by DL methods in scenarios where no data fusion is performed. For instance, a few images are enough to threaten users' privacy even in the presence of image obfuscation [420], and the model parameters of a DNN can be exposed by simply performing input queries on the model [356, 357].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [356]
The Introduction describes using input queries and the resulting predictions to train a knockoff model. But the Related Work section distinguishes this paper’s functionality stealing from extraction of model parameters: its approach works independently of the victim model’s internals. The source therefore supports the query-based attack, not the claim that queries expose the DNN’s parameters. (LLM, conf=0.87)
Experimental — accuracy not yet measuredUnder minimal assumptions, data model functionality stealing is possible.
“The latter shows that, under minimal assumptions, data model functionality stealing is possible.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [356]
Capability stealing is typically performed under black-box access, where the attacker observes only the model's outputs and has no access to its internal states, parameters, or training data.
“This capability stealing (Tramèr et al., 2016;Orekondy et al., 2019) is typically performed under black-box access, where the attacker observes only the model's outputs and has no access to its internal states, parameters, or training data.”
From How to Steal Reasoning Without Reasoning Traces (2026) · cites this paper as Orekondy et al., 2019
Privacy loss and intent loss subjective scores can explain loss of privacy, and such explanations have motivated secure matching cryptographic protocols.
“An approach to explain loss of privacy is by using privacy loss and intent loss subjective scores.”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [356]
The Related Work section discusses privacy-sensitive visual content, while the Rewards section defines measures for selecting images in a model-stealing attack. Neither addresses subjective privacy-loss or intent-loss scores as explanations of privacy loss. Searches for those scores and for secure matching cryptographic protocols found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · see citing works
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680 · see citing works
The Power of Ensembles for Active Learning in Image Classification
Beluch, W. H., Genewein, T., Nurnberger, A., & Kohler, J. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00976 · Cited by 655 · see citing works
PRADA: Protecting Against DNN Model Stealing Attacks
Juuti, M., Szyller, S., Marchal, S., & Asokan, N. (2019) · 2019 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2019.00044 · Cited by 440 · cited 2× · see citing works
“[19, 25] extend the line of work by stealing models (slightly more complex than [35] e.”
BackgroundMethod
Flagged first, then by citations
Research works printed but not linked: 38
- N. Akhtar, J. Liu, and A. Mian. Defense against universal adversarial perturbations. In CVPR, 2018. 2
- A. N. Bhagoji, W. He, B. Li, and D. Song. Exploring the space of black-box attacks on deep neural networks. arXiv preprint arXiv:1712.09491, 2017. 8 · arXiv
- C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil. Model compression. In KDD, 2006. 1, 2, 4
- G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker. Learn- ing efficient object detection models with knowledge distil- lation. In NIPS, 2017. 2
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 3, 4, 5, 12
- S. Ebert, M. Fritz, and B. Schiele. Ralf: A reinforced active learning formulation for object class recognition. In CVPR, 2012. 3, 7
- T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar. Born again neural networks. In ICML, 2018. 2, 4
- S. Han, H. Mao, and W. J. Dally. Deep compression: Com- pressing deep neural networks with pruning, trained quanti- zation and huffman coding. In ICLR, 2016. 4
- G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015. 1, 2, 3, 4, 5, 12 · arXiv
- G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 8
- F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accu- racy with 50x fewer parameters and¡ 0.5 mb model size. arXiv:1602.07360, 2016. 4 · arXiv
- A. Ilyas, L. Engstrom, A. Athalye, and J. Lin. Black-box adversarial attacks with limited queries and information. In ICML, 2018. 8
- A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Multi-class active learning for image classification. In CVPR, 2009. 4
- V. Khrulkov and I. Oseledets. Art of singular vectors and universal adversarial perturbations. In CVPR, 2018. 2
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 8
- A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, et al. The open images dataset v4: Unified image classifi- cation, object detection, and visual relationship detection at scale. arXiv:1811.00982, 2018. 5, 8, 11 · arXiv
- Y. Liu, X. Chen, C. Liu, and D. Song. Delving into transfer- able adversarial examples and black-box attacks. In ICLR, 2017. 8
- Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In ICCV, 2015. 8
- S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deep- fool: a simple and accurate method to fool deep neural net- works. In CVPR, 2016. 2
- S. J. Oh, M. Augustin, B. Schiele, and M. Fritz. Towards reverse-engineering black-box neural networks. In ICLR, 2018. 1, 2, 4
- S. J. Oh, R. Benenson, M. Fritz, and B. Schiele. Faceless person recognition; privacy implications in social media. In ECCV, 2016. 2
- S. J. Oh, M. Fritz, and B. Schiele. Adversarial image pertur- bation for privacy protection -a game theory perspective. In ICCV, 2017. 2
- T. Orekondy, M. Fritz, and B. Schiele. Connecting pixels to privacy and utility: Automatic redaction of private informa- tion in images. In CVPR, 2018. 2
- T. Orekondy, B. Schiele, and M. Fritz. Towards a visual privacy advisor: Understanding and predicting privacy risks in images. In ICCV, 2017. 2
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Ma- chine learning in Python. Journal of Machine Learning Re- search, 12:2825-2830, 2011. 12
- O. Poursaeed, I. Katsman, B. Gao, and S. Belongie. Genera- tive adversarial perturbations. In CVPR, 2018. 2
- A. Quattoni and A. Torralba. Recognizing indoor scenes. In CVPR, 2009. 4, 11
- A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 3
- A. Salem, Y. Zhang, M. Humbert, M. Fritz, and M. Backes. Ml-leaks: Model and data independent membership infer- ence attacks and defenses on machine learning models. In NDSS, 2019. 1
- B. Settles and M. Craven. An analysis of active learning strategies for sequence labeling tasks. In EMNLP, 2008. 2, 4, 7
- K. Simonyan and A. Zisserman. Very deep con- volutional networks for large-scale image recognition. arXiv:1409.1556, 2014. 4, 8 · arXiv
- Q. Sun, L. Ma, S. J. Oh, L. van Gool, B. Schiele, and M. Fritz. Natural and effective obfuscation by head inpaint- ing. In CVPR, 2018. 2
- R. S. Sutton and A. G. Barto. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998. 4
- S. Tong and D. Koller. Support vector machine active learn- ing with applications to text classification. JMLR, 2001. 2
- F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Risten- part. Stealing machine learning models via prediction apis. In USENIX Security, 2016. 1, 2
- Z. Wu, Z. Wang, Z. Wang, and H. Jin. Towards privacy- preserving visual recognition via adversarial training: A pilot study. In ECCV, 2018. 2
- R. Yonetani, V. N. Boddeti, K. M. Kitani, and Y. Sato. Privacy-preserving visual learning using doubly permuted homomorphic encryption. In ICCV, 2017. 2
- Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu. Deep mutual learning. In CVPR, 2018. 2 Test Set Buddha: 0.65 Boxing glove: 0.07 Jesus Christ: 0.034 Buddha: 0.49 Cake: 0.06 Minotaur: 0.05 Buddha: 0.28 Minotaur: 0.09 Dog: 0.09 Buddha: 0.68 Tombstone: 0.07 Elephant: 0.04 Buddha: 0.5 Minaret: 0.08 Tower Pisa: 0.05 People: 0.26 Buddha: 0.25 T-shirt: 0.14 Floppy-disk: 0.77 Socks: 0.04 Mattress: 0.03 Floppy-disk: 0.46 iPod: 0.18 CD: 0.06 Floppy-disk: 0.18 Flashlight: 0.08 Pez-dispenser: 0.07 Floppy-disk: 0.74 Necktie: 0.04 Video Projec.: 0.02 Floppy-disk: 0.61 Socks: 0.02 Teddy bear: 0.02 iPod: 0.49 Floppy-disk: 0.08 CD: 0.04 Tomato: 0.96 Grapes: 0.01 Boxing glove: 0.00 Tomato: 0.43 Welding mask: 0.12 Bowling ball: 0.01 Tomato: 0.27 Dice: 0.18 Stained glass: 0.01 Tomato: 0.94 Boxing glove: 0.04 Bowling ball: 0.00 Tomato: 0.41 Cactus: 0.07 Iguana: 0.6 Mushroom: 0.17 Tomato: 0.10 Birdbath: 0.07 Figure 13: Qualitative results: Caltech256. Extends Figure 6 in the main paper. GT labels are underlined, correct knockoff top-1 predictions in green and incorrect in red. Transfer Set Test Set Gadwall: 0.65 Nighthawk: 0.06 Horned Lark: 0.05 Gadwall: 0.31 B. Swallow: 0.14 N. Flicker: 0.12 Gadwall: 0.14 Chuck. Widow: 0.13 Swain. Warbler: 0.1 Gadwall: 0.95 Mallard: 0.01 Rb. Merganser: 0.00 Gadwall: 0.44 Mallard: 0.15 Rb. Merganser: 0.11 Pom. Jaeger: 0.16 Black Tern: 0.11 Herring Gull: 0.07 Lin. Sparrow: 0.82 Ovenbird: 0.07 House Sparrow: 0.03 Lin. Sparrow: 0.49 Mockingbird: 0.18 N. Waterthrush: 0.07 Lin. Sparrow: 0.32 Song Sparrow: 0.06 Tree Sparrow: 0.05 Lin. Sparrow: 0.64 Hen. Sparrow: 0.05 Clay c. Sparrow: 0.03 Lin. Sparrow: 0.50 Song Sparrow: 0.24 Hen. Sparrow: 0.04 Hen. Sparrow: 0.28 Ovenbird: 0.11 Lin. Sparrow: 0.10 Cactus Wren: 0.95 W. Meadowlark: 0.02 Lin. Sparrow: 0.00 Cactus Wren: 0.88 Geococcyx: 0.04
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
PRADA: Protecting Against DNN Model Stealing Attacks
Juuti, M., Szyller, S., Marchal, S., & Asokan, N. (2019) · 2019 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2019.00044 · Cited by 440 · see citing works
“These attacks benefit from correlations between different [40] or same [37] pre-trained models.”
BackgroundMethodArtificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data
Yu, N., Skripniuk, V., Abdelnabi, S., & Fritz, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01418 · Cited by 195 · see citing works
Self-citationUpdates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
Salem, A., Bhattacharya, A., Backes, M., Fritz, M., & Zhang, Y. (2020) · Universität des Saarlandes · doi:10.22028/d291-47370 · Cited by 157 · see citing works
Self-citationLocal and Central Differential Privacy for Robustness and Privacy in Federated Learning
Naseri, M., Hayes, J., & De Cristofaro, E. (2022) · Proceedings 2022 Network and Distributed System Security Symposium · doi:10.14722/ndss.2022.23054 · Cited by 188 · see citing works
Thieves on Sesame Street! Model Extraction of BERT-based APIs
Krishna, K., Tomar, G. S., Parikh, A. P., Papernot, N., & Iyyer, M. (2019) · arXiv · doi:10.48550/arxiv.1910.12366 · Cited by 111 · see citing works
ARCANE: An Efficient Architecture for Exact Machine Unlearning
Yan, H., Li, X., Guo, Z., Li, H., Li, F., & Lin, X. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/556 · Cited by 97 · see citing works
On the Robustness of Backdoor-based Watermarking in Deep Neural Networks
Shafieinejad, M., Lukas, N., Wang, J., Li, X., & Kerschbaum, F. (2021) · Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3437880.3460401 · Cited by 98 · see citing works
Imitation Attacks and Defenses for Black-box Machine Translation Systems
Wallace, E., Stern, M., & Song, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.446 · Cited by 75 · see citing works
Survey: Leakage and Privacy at Inference Time
Jegorova, M., Kaul, C., Mayor, C., O'Neil, A. Q., Weir, A., Murray-Smith, R., & Tsaftaris, S. A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229593 · Cited by 77 · see citing works
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48 · see citing works
“In this work, we study one possible resolution to this question: model imitation (Wallace et al., 2020; Orekondy et al., 2019).”
BackgroundMethodBlack-Box Dissector: Towards Erasing-Based Hard-Label Model Stealing Attack
Wang, Y., Li, J., Liu, H., Wang, Y., Wu, Y., Huang, F., & Ji, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20065-6_12 · Cited by 40 · see citing works
Kabir, S., Hossain, M. S., & Andersson, K. (2025) · Algorithms · doi:10.3390/a18090556 · Cited by 44 · see citing works
Graves, L., Nagisetty, V., & Ganesh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17371 · Cited by 5 · see citing works
How to Steal Reasoning Without Reasoning Traces
Zhang, T., Morris, J. X., & Shmatikov, V. (2026) · arXiv · doi:10.48550/arxiv.2603.07267
“This capability stealing (Tramèr et al., 2016; Orekondy et al., 2019) is typically performed under black-box access, where the attacker observes only the model’s outputs and has no access to its internal states, parameters, or training data.”
The Limits of Query Monitoring against Distributed Model Extraction
Khilar, S. C. (2026) · arXiv
“That asymmetry makes distillation through the API attractive: collect the model’s answers to many well-chosen prompts and train a smaller model to imitate them (Hinton et al., 2015; Tramèr et al., 2016; Orekondy et al., 2019).”
- Authors
- Orekondy, T., Schiele, B., & Fritz, M.
- Affiliations
- Max Planck Institute for Informatics (DE)
- MPI Informatics
- Helmholtz Center for Information Security (DE)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 4949-4958
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2905209730
- Cited by
- 549
- References
- 6
- Citations by year
- 2026: 62 · 2025: 76 · 2024: 113 · 2023: 91 · 2022: 78 · 2021: 73 · 2020: 38 · 2019: 16 · 2018: 1 · 2012: 1
- FWCI
- 19.61
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- MPG.PuRe (Max Planck Society) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- CISPA Helmholtz Center
- Funders
- Deutsche Forschungsgemeinschaft (DFG CRC 1223, CRC 1223)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Keywords
- black-box attacks, adversarial learning, image classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
