Preserving data privacy in machine learning systems
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
The wide adoption of Machine Learning to solve a large set of real-life problems came with the need to collect and process large volumes of data, some of which are considered personal and sensitive, raising serious concerns about data protection. Privacy-enhancing technologies (PETs) are often indicated as a solution to protect personal data and to achieve a general trustworthiness as required by current EU regulations on data protection and AI. However, an off-the-shelf application of PETs is insufficient to ensure a high-quality of data protection, which one needs to understand. This work systematically discusses the risks against data protection in modern Machine Learning systems taking the original perspective of the data owners, who are those who hold the various data sets, data models, or both, throughout the machine learning life cycle and considering the different Machine Learning architectures. It argues that the origin of the threats, the risks against the data, and the level of protection offered by PETs depend on the data processing phase, the role of the parties involved, and the architecture where the machine learning systems are deployed. By offering a framework in which to discuss privacy and confidentiality risks for data owners and by identifying and assessing privacy-preserving countermeasures for machine learning, this work could facilitate the discussion about compliance with EU regulations and directives. We discuss current challenges and research questions that are still unsolved in the field. In this respect, this paper provides researchers and developers working on machine learning with a comprehensive body of knowledge to let them advance in the science of data protection in machine learning field as well as in closely related fields such as Artificial Intelligence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Shamir, A. (1979) · Communications of the ACM · doi:10.1145/359168.359176 · Cited by 13,756 · cited 2×
“…SPDZ (Damgård et al., 2012) (Keller, 2020) TinyOT (Nielsen et al., 2012) SPDZ (Damgård et al., 2012) MASCOT (Keller et al., 2016) SPDZ, Overdrive (Keller et al., 2018) Yao grained circuits (Yao, 1982) Generalised secret sharing (Benaloh and Leichter, 1988) Shamir (Shamir, 1979) other MPC protocols;”
BackgroundRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“Approaches to address transparency by using explainability tools focus on explaining the already used black box models; while the paradigm of transparency by interpretability mainly focusses on building easy-to-trace and less opaque models (Rudin, 2019).”
Backgroundk-ANONYMITY: A MODEL FOR PROTECTING PRIVACY
SWEENEY, L. (2002) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488502001648 · Cited by 8,571 · cited 4×
“…privacy (Brickell and Shmatikov, 2008) 𝛽 -likeness (Cao and Karras, 2012) 𝛿 -presence (Nergiz et al., 2007) JAVA AnonyPy (Fujita, 2021) k-anonymity (Sweeney, 2002) l-diversity (Machanavajjhala et al., 2006) t-closeness (Li et al., 2007) Python Hybrid toolkits HaGrid/PySyft (Hall et al.,…”
MethodInfluentialFully homomorphic encryption using ideal lattices
Gentry, C. (2009) · Proceedings of the forty-first annual ACM symposium on Theory of computing · doi:10.1145/1536414.1536440 · Cited by 6,806 · cited 3×
“Crypto-based training uses recently proposed advanced cryptographic schemes that mainly include homomorphic encryption (HE) (Gentry, 2009; van Dijk et al., 2010; Brakerski et al., 2014; Martins et al., 2017; Cheon et al., 2017) and functional encryption (FE) (Boneh et al., 2011; Goldwasser et al.,…”
BackgroundFederated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3298981 · Cited by 6,322 · cited 2×
“…goal is the usage of surrogate datasets, which are formed by grouping anonymised datasets and abstracting the dataset using sketching techniques (Sabay et al., 2018; Yang et al., 2019a) or even using generative models to generate synthetic datasets (Nik Aznan et al., 2019) (Assefa et al., 2021).”
MethodThe Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2014) · Foundations and Trends® in Theoretical Computer Science · doi:10.1561/0400000042 · Cited by 5,557
Membership Inference Attacks Against Machine Learning Models
Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.41 · Cited by 4,754 · cited 7×
“There are various privacy attacks against machine learning systems such as membership inference attacks (Shokri et al., 2017; Bernau et al., 2019; Jia et al., 2019a; Li et al., 2020) and model inversion attacks (Fredrikson et al., 2015; He et al., 2019; Wu et al., 2016).”
BackgroundMethodInfluentialThe Algorithmic Foundations of Differential Privacy
Dwork, C., & Roth, A. (2013) · doi:10.1561/9781601988195 · Cited by 3,856
t-Closeness: Privacy Beyond k-Anonymity and l-Diversity
Li, N., Li, T., & Venkatasubramanian, S. (2007) · 2007 IEEE 23rd International Conference on Data Engineering · doi:10.1109/icde.2007.367856 · Cited by 3,451 · cited 3×
“…and Karras, 2012) 𝛿 -presence (Nergiz et al., 2007) JAVA AnonyPy (Fujita, 2021) k-anonymity (Sweeney, 2002) l-diversity (Machanavajjhala et al., 2006) t-closeness (Li et al., 2007) Python Hybrid toolkits HaGrid/PySyft (Hall et al., 2021) SMPC protocols Differential privacy Federated learning Python”
BackgroundMethodProtocols for secure computations
Yao, A. C. (1982) · 23rd Annual Symposium on Foundations of Computer Science (sfcs 1982) · doi:10.1109/sfcs.1982.38 · Cited by 3,306
Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
Fredrikson, M., Jha, S., & Ristenpart, T. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813677 · Cited by 2,961
Homomorphic Encryption for Arithmetic of Approximate Numbers
Cheon, J. H., Kim, A., Kim, M., & Song, Y. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70694-8_15 · Cited by 2,522 · cited 13×
“An underestimation is useless from a privacy perspective, whereas the overestimation of noise can cause a huge utility loss, especially since CKKS is a lossy encryption scheme where the decryption outputs a noisy result.”
BackgroundMethodInfluentialPrivacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 1×
“In differentially private stochastic gradient descent proposed by Abadi et al. (2016) and also (Shokri and Shmatikov, 2015) privacy comes at the cost of utility.”
BackgroundZhu, L., & Han, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-63076-8_2 · Cited by 1,906
Nasr, M., Shokri, R., & Houmansadr, A. (2019) · 2019 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2019.00065 · Cited by 1,760
Privacy-Preserving Deep Learning via Additively Homomorphic Encryption
Phong, L. T., Aono, Y., Hayashi, T., Wang, L., & Moriai, S. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2017.2787987 · Cited by 1,662
Unique in the Crowd: The privacy bounds of human mobility
de Montjoye, Y.-A., Hidalgo, C. A., Verleysen, M., & Blondel, V. D. (2013) · Scientific Reports · doi:10.1038/srep01376 · Cited by 1,645 · cited 1×
“Furthermore, sensitive attributes concealing techniques are susceptible to deanonymisation attacks (Narayanan and Shmatikov, 2019; De Montjoye et al., 2013).”
BackgroundDeep Models Under the GAN: Information Leakage from Collaborative Deep Learning
Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134012 · Cited by 1,377 · cited 1×
“During a horizontal federated learning setting, 20 reconstruction attacks exploit the gradients shared as intermediate results between the central aggregator and local clients (Zhu et al., 2019; Yin et al., 2021; Hitaj et al., 2017; Yang et al., 2023).”
BackgroundMultiparty Computation from Somewhat Homomorphic Encryption
Damgård, I., Pastro, V., Smart, N., & Zakarias, S. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-32009-5_38 · Cited by 1,331 · cited 6×
“…2012; Evans et al., 2018) Python ABY /ABY3 (Demmler et al., 2015) SPDZ (Damgård et al., 2012) (Keller, 2020) TinyOT (Nielsen et al., 2012) SPDZ (Damgård et al., 2012) MASCOT (Keller et al., 2016) SPDZ, Overdrive (Keller et al., 2018) Yao grained circuits (Yao, 1982) Generalised secret sharing…”
BackgroundMethodInfluentialManipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning
Jagielski, M., Oprea, A., Biggio, B., Liu, C., Nita-Rotaru, C., & Li, B. (2018) · 2018 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2018.00057 · Cited by 861
Stealing Machine Learning Models via Prediction APIs
Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016) · arXiv · doi:10.48550/arxiv.1609.02943 · Cited by 796
Hardware Trojan attacks: Threat analysis and countermeasures
Bhunia, S., Hsiao, M. S., Banga, M., & Narasimhan, S. (2014) · Proceedings of the IEEE · doi:10.1109/jproc.2014.2334493 · Cited by 766
Deep learning in computer vision: A critical review of emerging techniques and application scenarios
Chai, J., Zeng, H., Li, A., & Ngai, E. W. T. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100134 · Cited by 754 · cited 1×
“…high performance and accuracy, which have led them to become drivers of innovation in several disciplines and sectors such as computer vision (Chai et al., 2021), autonomous transportation, health care (Ghassemi et al., 2020), biomedicine (Mamoshina et al., 2016) and law (Surden, 2014).”
BackgroundBarbedo, J. G. A. (2018) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2018.08.013 · Cited by 750
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · cited 1×
“Carlini et al. (2021) designed a simple attack to study the extent of training data memorisation in GPT2.”
BackgroundLepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · cited 1×
“As a technical property in machine learning models, transparency addresses the understandability of how the model functions internally: at the model level, the individual components (model parameters), and the training algorithm (Lepri et al., 2018).”
BackgroundApplications of Deep Learning in Biomedicine
Mamoshina, P., Vieira, A., Lane, E., & Zhavoronkov, A. (2016) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.5b00982 · Cited by 701
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629
Federated Machine Learning: Concept and Applications
Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019) · arXiv · doi:10.48550/arxiv.1902.04885 · Cited by 600
Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization
Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E. C., & Roli, F. (2017) · Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3128572.3140451 · Cited by 549
SoK: Security and Privacy in Machine Learning
Papernot, N., McDaniel, P., Sinha, A., & Wellman, M. P. (2018) · 2018 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2018.00035 · Cited by 502 · cited 14×
“G-PATE (Long et al., 2021) addresses this limitation by proving that it is not necessary to ensure that the discriminator is differentially private to train a differentially private generator.”
BackgroundResultInfluentialWhen Machine Learning Meets Privacy: A Survey and Outlook
Liu, B., Ding, M., Shaham, S., Rahayu, W., Farokhi, F., & Lin, Z. (2022) · ACM Computing Surveys · doi:10.1145/3436755 · Cited by 455
Privacy-Preserving Machine Learning: Threats and Solutions
Al-Rubaie, M., & Chang, J. M. (2019) · IEEE Security & Privacy · doi:10.1109/msec.2018.2888775 · Cited by 444
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 1×
“Among those works, we find the Microsoft fairness checklist (Madaio et al., 2020) that incorporates different stages of the AI system development and deployment life cycle.”
MethodModel inversion attacks against collaborative inference
He, Z., Zhang, T., & Lee, R. B. (2019) · Proceedings of the 35th Annual Computer Security Applications Conference · doi:10.1145/3359789.3359824 · Cited by 313 · cited 2×
“There are various privacy attacks against machine learning systems such as membership inference attacks (Shokri et al., 2017; Bernau et al., 2019; Jia et al., 2019a; Li et al., 2020) and model inversion attacks (Fredrikson et al., 2015; He et al., 2019; Wu et al., 2016).”
BackgroundCrypTen: Secure Multi-Party Computation Meets Machine Learning
Knott, B., Venkataraman, S., Hannun, A., Sengupta, S., Ibrahim, M., & van der Maaten, L. (2021) · arXiv · doi:10.48550/arxiv.2109.00984 · Cited by 257 · cited 1×
“…PALISADE (PALISADE, 2020) BGV (Brakerski et al., 2014) BFV (Brakerski, 2012) FHEW (Ducas and Micciancio, 2015) CKKS (Cheon et al., Crypten toolkit (Knott et al., 2021) Additive secret sharing (Damgård et al., 2012; Evans et al., 2018) Python ABY /ABY3 (Demmler et al., 2015) SPDZ (Damgård et al.,…”
BackgroundDistributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
Chen, Y., Su, L., & Xu, J. (2018) · Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems · doi:10.1145/3219617.3219655 · Cited by 228
Quantifying Memorization Across Neural Language Models
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022) · arXiv · doi:10.48550/arxiv.2202.07646 · Cited by 204 · cited 1×
“Moreover, the risk of memorisation by language models can be increased when there are duplicate sequences within the training data (Carlini et al., 2023).”
BackgroundUpdates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning
Salem, A., Bhattacharya, A., Backes, M., Fritz, M., & Zhang, Y. (2020) · Universität des Saarlandes · doi:10.22028/d291-47370 · Cited by 157
This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology
Mulligan, D. K., Kroll, J. A., Kohli, N., & Wong, R. Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359221 · Cited by 111
Residual unfairness in fair machine learning from prejudiced data
Kallus, N., & Zhou, A. (2018) · arXiv · doi:10.48550/arxiv.1806.02887 · Cited by 83
Approximate Homomorphic Encryption with Reduced Approximation Error
Kim, A., Papadimitriou, A., & Polyakov, Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-95312-6_6 · Cited by 71 · cited 2×
“The usage of approximate number schemes such as CKKS result in an additional noise 𝐷𝑒𝑐 ( 𝑐𝑡 ) = ̃𝑚 = 𝑚 + 𝑓 , where f is a small error (Kim et al., 2022).”
MethodUnderstanding Unintended Memorization in Language Models Under Federated Learning
Thakkar, O. D., Ramaswamy, S., Mathews, R., & Beaufays, F. (2021) · Proceedings of the Third Workshop on Privacy in Natural Language Processing · doi:10.18653/v1/2021.privatenlp-1.1 · Cited by 39
Cited by
Artificial Intelligence in Religious Education: Ethical, Pedagogical, and Theological Perspectives
Papakostas, C. (2025) · Religions · doi:10.3390/rel16050563 · Cited by 78
Artificial intelligence across the cancer care continuum
Riaz, I. B., Khan, M. A., & Osterman, T. J. (2025) · Cancer · doi:10.1002/cncr.70050 · Cited by 40
Metadata
- Authors
- El Mestari, S. Z., Lenzini, G., & Demirci, H.
- Venue
- Computers & Security
- Publisher
- Elsevier BV
- Published
- 2024-02
- Volume
- 137
- Pages
- 103605
- ISSN
- 0167-4048 (ISSN-L)
- OpenAlex
- W4389118479
- Cited by
- 162
- References
- 44
- FWCI
- 14.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.