Quantifying and Localizing Usable Information Leakage from Neural Network Gradients

“LeNet [59] C6(5)-P(2)-C16(5)-P(2)-F120-F84-O CIFAR-100, LFW, CelebA, & PubFig AlexNet [60] C8(5)-C16(3)-P(2)-C32(3)×3-P(2)-F120-F84-O VGG9 [61] C8(3)-C16(3)×2-P(2)-C32(3)×3-P(2)-F256-F128-O”

“LeNet AlexNet VGG9 TextClf 1 2 3 4 5 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 9 1 2 3 4 0.0 0.2 0.4 0.6 0.8 1.0 Layer No.”

“LeNet [59] C6(5)-P(2)-C16(5)-P(2)-F120-F84-O CIFAR-100, LFW, CelebA, & PubFig AlexNet [60] C8(5)-C16(3)-P(2)-C32(3)×3-P(2)-F120-F84-O VGG9 [61] C8(3)-C16(3)×2-P(2)-C32(3)×3-P(2)-F256-F128-O”

“As one example, both peak signalto-noise ratio (PSNR) and SSIM can be used for images; where the latter is defined based on the human visual system and thus is more suitable for measuring human perception [48], [49], while the former is a general statistical measure in signal processing applications.”

“, face identity); as also shown in [42], [43].”

“TextClf [62] C16(3)-P(2)-C16(8)-P(2)-F50-O IMDB & CSI”

“This could be the local aggregation (e.g., batch size in FedSGD or multiple steps in FedAvg) and also the global aggregation over updates of all clients.”

“Information leakages in other widely used networks such as Recurrent [74], [75] or Graph neural networks need investigating; ii) More defense mechanisms can be analyzed using our proposed metrics.”

“The gradient sharing can be among clients or through a central server; depending on the specific settings [34], [7], [33].”

“LeNet, AlexNet, and VGG9 models are trained on CIFAR-100 [63] and three other image datasets with attributes, including Labeled Faces in the Wild (LFW) [64], Large-scale CelebFaces Attributes (CelebA) [65], and Public Figures Face Database (PubFig) [66].”

“Collaborative learning [1], [2] allows private data owners (holding personal, medical, financial, or governmental datasets [3], [4], [5]) to participate in training a deep neural network (DNN) model for some target task.”

“As one example, both peak signalto-noise ratio (PSNR) and SSIM can be used for images; where the latter is defined based on the human visual system and thus is more suitable for measuring human perception [48], [49], while the former is a general statistical measure in signal processing applications.”

“As a final remark, membership inference attack, another well-known attack that infers the participation of a particular client in the training dataset [11], [46], can also be formulated as latent attribute when defining P = 1 as membership and P = 0 as non-membership.”

“Collaborative learning [1], [2] allows private data owners (holding personal, medical, financial, or governmental datasets [3], [4], [5]) to participate in training a deep neural network (DNN) model for some target task.”

“However, it is not straightforward to apply DPI for analyzing the information flow in backward propagation, because the computation of gradients Gl in backward propagation form a more complicated graphical model (Figure 1- 2©).”

“Here we follow [70], [71], i) to clip gradients using l2-norm and then ii) to add Gaussian noises.”

“For a batch size B > 1, the optimizer can have B! different permutations, so that choosing the right gradient descent direction is complicated - as a solution a single training sample can be updated in each optimization step [7], [15].”

“, face identity); as also shown in [42], [43].”

“As a final remark, membership inference attack, another well-known attack that infers the participation of a particular client in the training dataset [11], [46], can also be formulated as latent attribute when defining P = 1 as membership and P = 0 as non-membership.”

“However, as noted by existing attacks to collaborative learning [7], [8], [9], [10], [11], the shared gradients might be enough for either the central server or other clients to infer private attributes of a data owner or even to reconstruct their private data.”

“Nevertheless, it is well-known that the estimation of Shannon’s entropy and MI for high-dimensional data can have a high bias or variance [22], [23].”

“LeNet, AlexNet, and VGG9 models are trained on CIFAR-100 [63] and three other image datasets with attributes, including Labeled Faces in the Wild (LFW) [64], Large-scale CelebFaces Attributes (CelebA) [65], and Public Figures Face Database (PubFig) [66].”

“These attacks are usually based on training some generative adversarial networks and optimizing an attack objective function [37], [7].”

“Collaborative learning [1], [2] allows private data owners (holding personal, medical, financial, or governmental datasets [3], [4], [5]) to participate in training a deep neural network (DNN) model for some target task.”

“Privacy attacks on collaborative learning are usually conducted on the exposed elements such as the gradients [9], [7] and intermediate representations [35].”

“, k} are principal angles between the two subspaces which can be computed using numerical methods [58].”

“Using different norms will help in capturing attackers with different capabilities, similarly to [55], [53], where p-norm reflects how the attacker measures distance between two data samples (e.”

“For example, [7] and [8] use ‖ĝ − g‖p with p-norm, while [10] uses cosine similarity between ĝ and g.”

“, split learning [30], differential privacy (DP) [31], or trusted execution environments [32].”

“Our measures could also provide insights for layer-wise, federated training of models [34], [72], which could further reduce the privacy risk by exposing only specific layers instead of the complete model.”

“Nevertheless, it is well-known that the estimation of Shannon’s entropy and MI for high-dimensional data can have a high bias or variance [22], [23].”

“For example, [7] and [8] use ‖ĝ − g‖p with p-norm, while [10] uses cosine similarity between ĝ and g.”

“In a separate line of work, information theory is used for measuring the amount of original or latent information that is captured in the outputs of different DNN layers and how this information evolves during the training procedure [17], [18], [19], [20], [21].”

“These gradient-based attacks are categorized into: i) the original information, to reconstruct the input data [15], [10], [38], namely data reconstruction attack (DRA), and ii) the latent information, to infer attributes of the input data [9], namely attribute inference attack (AIA).”

“Bayesian-based generalization analysis [76] may also help to theoretically characterize information leakages.”

“In the following, we explain how gradient sensitivity, which previously has been studied for examining model robustness and generalization [28], [50], can be expanded into metrics to be used for studying both original and latent information.”

“Using different norms will help in capturing attackers with different capabilities, similarly to [55], [53], where p-norm reflects how the attacker measures distance between two data samples (e.”

“The attacker can even actively participate in the training process [36].”

“The gradient sharing can be among clients or through a central server; depending on the specific settings [34], [7], [33].”

“In addition, a low sensitivity has been linked to increased robustness of trained models [54].”

“In this paper we use the Grassmann distance; other common distances defined on Grassmannians (e.g., Asimov) [57] can be derived similarly.”

“Collaborative learning [1], [2] allows private data owners (holding personal, medical, financial, or governmental datasets [3], [4], [5]) to participate in training a deep neural network (DNN) model for some target task.”

“, the last Conv layer and the first FC layer) inside the TEE using model partitioned execution techniques across trusted and untrusted environments similar to [32].”

“More specifically, the DRA tries to minimize the following objective by updating both dummy input x′ and dummy label y′, x̂, ŷ = arg min x′,y′ dist(ĝ, g), (5) where ĝ and g denote the dummy and observed gradients, respectively, and dist(·, ·) can be any suitably chosen distance metric for them.”

“We also use a cropped version (100×100 RGB) of PubFig which contains 8300 facial images made up of 100 images for each of 83 persons [69], marked with 73 attributes (e.”

“As one example, clipping or adding noises differently for layers in DP [31], [73] and directly hiding layers in TEEs [32] could be one promising protection without compromising the utility of the other layers and our quantification helps in understanding and designing these layer-wise protection mechanisms.”

“As one example, clipping or adding noises differently for layers in DP [31], [73] and directly hiding layers in TEEs [32] could be one promising protection without compromising the utility of the other layers and our quantification helps in understanding and designing these layer-wise protection mechanisms.”

“Definition 1 (Usable Attribute Information [27]): Let G and P be random variables denoting the observed gradients and private latent attribute, and g and p be their realisations, respectively.”

“Few recent works [15], [16] explain why data reconstruction attacks can be successful in certain applications, but we still lack a unifying theory for measuring different types of information contained in gradients.”

“In the following, we explain how gradient sensitivity, which previously has been studied for examining model robustness and generalization [28], [50], can be expanded into metrics to be used for studying both original and latent information.”

“In IMDB reviews [67] dataset, each review is labeled with sentiment and length, and in CSI corpus [68], each corpus review is labeled with sentiment, veracity, length, etc.”

“To measure the difference between the computed gradients with and without target latent attribute, we can compute the Grassmann geodesic distance [56], [57] between these two subspaces as the leakage risk of this latent information p in layer l’s gradients:”

“Few recent works [15], [16] explain why data reconstruction attacks can be successful in certain applications, but we still lack a unifying theory for measuring different types of information contained in gradients.”

“In a separate line of work, information theory is used for measuring the amount of original or latent information that is captured in the outputs of different DNN layers and how this information evolves during the training procedure [17], [18], [19], [20], [21].”

“To quantify the information flow in forward propagation through DNN layers, previous works analyzed intermediate representations of the input data in a layer-wise manner to understand the flow of information [39], [40], [41].”

“In addition to the measurement of (semi-standard) defense techniques such as DP and dropout, [14] suggests applying simple model adaptation such as increasing the models’ depth and the number of clients, which drop the attack success rates in general.”

“input/output can reflect model robustness [28], and robustness, in turn, is related to privacy risks [29].”

“However, it needs to be considered that also obfuscated gradients may become subject to reconstruction attacks and leak information 71,72 .”