The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations
Open accessAbstract
With the growing adoption of machine learning, sharing of learned models is becoming popular. However, in addition to the prediction properties the model producer aims to share, there is also a risk that the model consumer can infer other properties of the training data the model producer did not intend to share. In this paper, we focus on the inference of global properties of the training data, such as the environment in which the data was produced, or the fraction of the data that comes from a certain class, as applied to white-box Fully Connected Neural Networks (FCNNs). Because of their complexity and inscrutability, FCNNs have a particularly high risk of leaking unexpected information about their training sets; at the same time, this complexity makes extracting this information challenging. We develop techniques that reduce this complexity by noting that FCNNs are invariant under permutation of nodes in each layer. We develop our techniques using representations that capture this invariance and simplify the information extraction task. We evaluate our techniques on several synthetic and standard benchmark datasets and show that they are very effective at inferring various data properties. We also perform two case studies to demonstrate the impact of our attack. In the first case study we show that a classifier that recognizes smiling faces also leaks information about the relative attractiveness of the individuals in its training set. In the second case study we show that a classifier that recognizes Bitcoin mining from performance counters also leaks information about whether the classifier was trained on logs from machines that were patched for the Meltdown and Spectre attacks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In order to better capture the level of confidence of the model, one may also sort the posteriors before the difference or concatenation operations [20].”
From When Machine Unlearning Jeopardizes Privacy (2021) · cites this paper as [20]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Deep Learning with Differential Privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016) · Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2976749.2978318 · Cited by 6,578
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754
(2018) · Cited by 3,510
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
Practical Black-Box Attacks against Machine Learning
Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., & Swami, A. (2017) · Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security · doi:10.1145/3052973.3053009 · Cited by 3,495
Rodinia: A benchmark suite for heterogeneous computing
Che, S., Boyer, M. J., Meng, J., Tarjan, D., Sheaffer, J. W., Lee, S.-H., & Kevin Skadron, Y. (2009) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc.2009.5306797 · Cited by 3,100
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Universal Adversarial Perturbations
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., & Frossard, P. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.17 · Cited by 2,680
(2011) · IEEE Software · doi:10.1109/ms.2011.67 · Cited by 2,038
Spectre Attacks: Exploiting Speculative Execution
Kocher, P. C., Horn, J., Fogh, A., Genkin, D., Gruss, D., Haas, W., Hamburg, M., Lipp, M., Mangard, S., Prescher, T., et al. (2019) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2019.00002 · Cited by 1,847
Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2641190.2641198 · Cited by 1,452
Trojaning Attack on Neural Networks
Liu, Y., Ma, S., Aafer, Y., Lee, W.-C., Zhai, J., Wang, W., & Zhang, X. (2018) · Proceedings 2018 Network and Distributed System Security Symposium · doi:10.14722/ndss.2018.23291 · Cited by 1,270
Meltdown: reading kernel memory from user space
Lipp, M., Schwarz, M., Gruss, D., Prescher, T., Haas, W., Fogh, A., Horn, J., Mangard, S., Kocher, P. C., Genkin, D., et al. (2018) · USENIX Security Symposium · Cited by 821
Stealing Machine Learning Models via Prediction APIs
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016) · arXiv · doi:10.48550/arxiv.1609.02943 · Cited by 796
Parboil: A Revised Benchmark Suite for Scientific and Commercial Throughput Computing
Stratton, J. A., Rodrigues, C. I., Sung, I.-J., Obeid, N., Chang, L., Anssari, N., Liu, G. D., & Hwu, W. (2012) · Cited by 707
Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing.
Fredrikson, M., Lantz, E., Jha, S., Lin, S. M., Page, D., & Ristenpart, T. (2014) · PubMed · Cited by 658
Machine Learning Models that Remember Too Much
Song, C., Ristenpart, T., & Shmatikov, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134077 · Cited by 526
Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., & Felici, G. (2015) · International Journal of Security and Networks · doi:10.1504/ijsn.2015.071829 · Cited by 518
Stealing Hyperparameters in Machine Learning
Wang, B., & Gong, N. Z. (2018) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp.2018.00038 · Cited by 505
Robust Physical-World Attacks on Deep Learning ModelsPreprint
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. X. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08945 · Cited by 443
On the feasibility of online malware detection with performance counters
Demme, J., Maycock, M., Schmitz, J., Tang, A., Waksman, A., Sethumadhavan, S., & Stolfo, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2485922.2485970 · Cited by 341
Zaheer, M., Kottur, S., Ravanbakhsh, S., Póczos, B., Salakhutdinov, R. R., & Smola, A. J. (2017) · neural information processing systems · Cited by 279
Real time detection of cache-based side-channel attacks using hardware performance counters
Chiappetta, M., Savaş, E., & Yılmaz, C. (2016) · Applied Soft Computing · doi:10.1016/j.asoc.2016.09.014 · Cited by 262
On the feasibility of online malware detection with performance counters
Demme, J., Maycock, M., Schmitz, J., Tang, A., Waksman, A., Sethumadhavan, S., & Stolfo, S. J. (2013) · ACM SIGARCH Computer Architecture News · doi:10.1145/2508148.2485970 · Cited by 232
Robust Physical-World Attacks on Machine Learning Models.Preprint
Evtimov, I., Eykholt, K., Fernandes, E., Kohno, T., Li, B., Prakash, A., Rahmati, A., & Song, D. X. (2017) · arXiv (Cornell University) · Cited by 195
Adversarial Examples that Fool DetectorsPreprint
Lu, J., Sibai, H., & Fabry, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.02494 · Cited by 159
Wang, X., & Karri, R. (2013) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/2463209.2488831 · Cited by 100
Analysis of universal adversarial perturbationsPreprint
Moosavi-Dezfooli, S.-M., Fawzi, A., Fawzi, O., Pascal Frossard, B., & Soatto, S. (2017) · arXiv (Cornell University) · Cited by 86
Mining on Someone Else’s Dime: Mitigating Covert Mining Operations in Clouds and Enterprises
Tahir, R., Huzaifa, M., Das, A., Ahmad, M. A., Gunter, C. A., Zaffar, F., Caesar, M., & Borisov, N. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66332-6_13 · Cited by 55
Self-citationOpenML: networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., et al. (2014) · doi:10.48550/arxiv.1407.7722 · Cited by 53
Stealing Hyperparameters in Machine LearningPreprint
Wang, B., & Gong, N. Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.05351 · Cited by 52
Towards Reverse-Engineering Black-Box Neural NetworksPreprint
Oh, S. J., Augustin, M., Schiele, B., & Fritz, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.01768 · Cited by 42
Klein, S. (2017) · Apress eBooks · doi:10.1007/978-1-4842-2143-3_14 · Cited by 39
The Unintended Consequences of Overfitting: Training Data Inference Attacks.Preprint
Yeom, S., Fredrikson, M., & Jha, S. (2017) · arXiv (Cornell University) · Cited by 20
Flagged first, then by citations
Research works printed but not linked: 9
- Amazon. 2018. Machine Learning at AWS. https://aws.amazon.com/machine-learning/ Retrieved August 15, 2018 from Amazon. 2018. Machine Learning at AWS. https://aws.amazon.com/machine-learning/ Retrieved August 15, 2018 from
- BigML Inc. 2018. BigML. https://bigml.com/ Retrieved August 15, 2018 from BigML Inc. 2018. BigML. https://bigml.com/ Retrieved August 15, 2018 from
- Eric Florenzano . 2016. Gadientzoo. https://www.gradientzoo.com/ Retrieved August 15, 2018 from Eric Florenzano. 2016. Gadientzoo. https://www.gradientzoo.com/ Retrieved August 15, 2018 from
- Google Cloud . 2018. Cloud Machine Learning Engine. https://cloud.google.com/ml-engine/ Retrieved August 15, 2018 from Google Cloud. 2018. Cloud Machine Learning Engine. https://cloud.google.com/ml-engine/ Retrieved August 15, 2018 from
- Jamie Hayes , Luca Melis , George Danezis , and Emiliano De Cristofaro . 2017. LOGAN: evaluating privacy leakage of generative models using generative adversarial networks. arXiv preprint arXiv:1705.07663 ( 2017 ). Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. 2017. LOGAN: evaluating privacy leakage of generative models using generative adversarial networks. arXiv preprint arXiv:1705.07663 (2017). · arXiv
- Zhiqiang Lin , Xiangyu Zhang , and Dongyan Xu . 2010 . Automatic reverse engineering of data structures from binary execution . In Proceedings of the 11th Annual Information Security Symposium. CERIAS-Purdue University, 5. Zhiqiang Lin, Xiangyu Zhang, and Dongyan Xu. 2010. Automatic reverse engineering of data structures from binary execution. In Proceedings of the 11th Annual Information Security Symposium. CERIAS-Purdue University, 5.
- PyTorch core team. 2018. Pytorch . http://pytorch.org/ Retrieved August 15, 2018 from PyTorch core team. 2018. Pytorch. http://pytorch.org/ Retrieved August 15, 2018 from
- Perf Wiki . 2018. Linux profiling with performance counters. https://perf.wiki.kernel.org/index.php/Main_Page Retrieved August 15, 2018 from Perf Wiki. 2018. Linux profiling with performance counters. https://perf.wiki.kernel.org/index.php/Main_Page Retrieved August 15, 2018 from
- Manzil Zaheer Satwik Kottur Siamak Ravanbakhsh Barnabas Poczos Ruslan R Salakhutdinov and Alexander J Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems. 3394--3404. Manzil Zaheer Satwik Kottur Siamak Ravanbakhsh Barnabas Poczos Ruslan R Salakhutdinov and Alexander J Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems. 3394--3404.
Cited by
Sorted by citations
Exploiting Unintended Feature Leakage in Collaborative Learning
Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00029 · Cited by 1,734
Nasr, M., Shokri, R., & Houmansadr, A. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00065 · Cited by 1,760
“class. This attack works on datasets where all of the members in the same classes are similar. No metric for leakage is used to measure privacy, except illustrating the generated images. Ganju et al. [26] designed a new property inference attack on fully connected networks. In this attack, the attacker goal is to extr…”
BackgroundInverting Gradients -- How easy is it to break privacy in federated learning?Preprint
Geiping, J., Bauermeister, H., Dröge, H., & Moeller, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.14053 · Cited by 848
“A related, but easier problem, compared to the full reconstruction of input images, is the retrieval of input attributes [23, 10] from local updates, e.”
BackgroundMembership Inference Attacks on Machine Learning: A Survey
Hu, H., Salcic, Z. A., Sun, L., Dobbie, G., Yu, P. S., & Zhang, X. (2022) · ACM Computing Surveys · doi:10.1145/3523273 · Cited by 662
“However, one must be aware that DP cannot only be used to mitigate MIAs, but also other forms of privacy attacks such as attribute inference attacks [47, 48] and property inference attacks [4, 49].”
BackgroundMembership Inference Attacks From First Principles
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., & Tramèr, F. (2022) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46214.2022.9833649 · Cited by 728
“For example, a clas-sifier trained on bitcoin logs can reveal whether or not the machines that generated the logs were patched for Meltdown and Spectre [15].”
Background
Metadata
- Authors
- Ganju, K., Wang, Q., Yang, W., Gunter, C. A., & Borisov, N.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- The University of Texas at Dallas (US)
- Venue
- Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security
- Publisher
- ACM
- Published
- 2018-10-15
- Pages
- 619-633
- ISSN
- 1543-7221 (ISSN-L)
- OpenAlex
- W2897830718
- Semantic Scholar
- 490c30b1d6b680be3c5a13552073e4fc10a850dc
- Cited by
- 475
- References
- 37
- Citations by year
- 2026: 27 · 2025: 58 · 2024: 89 · 2023: 98 · 2022: 74 · 2021: 69 · 2020: 39 · 2019: 19 · 2018: 2
- FWCI
- 13.11
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM SIGSAC Conference on Computer and Communications Security (CCS) (publishedVersion)
- Funders
- U.S. Department of Energy (DE-OE0000780)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
- Physical Sciences › Computer Science › Hardware and Architecture › Security and Verification in Computing
- Keywords
- property inference attacks, fully connected neural network, permutation invariance, training data leakage, white-box attacks, model sharing, performance counters
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
