Research on the Deep Learning Method Based on Data Feature Relevance and Adaptive Differential Privacy

KANG Hai-yan; WANG Xiao-shi

doi:10.12263/DZXB.20220892

您当前的位置：

首页 >

文章列表页 >

Research on the Deep Learning Method Based on Data Feature Relevance and Adaptive Differential Privacy

PAPERS | 更新时间：2025-12-11

- Research on the Deep Learning Method Based on Data Feature Relevance and Adaptive Differential Privacy
- ACTA ELECTRONICA SINICA Vol. 52, Issue 6, Pages: 1963-1976(2024)
- 作者机构：
  
  北京信息科技大学信息安全系，北京 100192
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20220892
  CLC： TP309.2
- Received：26 July 2022，
  
  Revised：2023-04-01，
  
  Published：25 June 2024
- 稿件说明：
移动端阅览
康海燕, 王骁识. 基于数据特征相关性和自适应差分隐私的深度学习方法研究[J]. 电子学报, 2024, 52(06): 1963-1976.

KANG Hai-yan, WANG Xiao-shi. Research on the Deep Learning Method Based on Data Feature Relevance and Adaptive Differential Privacy[J]. Acta Electronica Sinica, 2024, 52(06): 1963-1976.
康海燕, 王骁识. 基于数据特征相关性和自适应差分隐私的深度学习方法研究[J]. 电子学报, 2024, 52(06): 1963-1976. DOI：10.12263/DZXB.20220892

KANG Hai-yan, WANG Xiao-shi. Research on the Deep Learning Method Based on Data Feature Relevance and Adaptive Differential Privacy[J]. Acta Electronica Sinica, 2024, 52(06): 1963-1976. DOI：10.12263/DZXB.20220892

摘要

基于差分隐私的深度学习隐私保护方法中，训练周期的长度以及隐私预算的分配方式直接制约着深度学习模型的效用.针对现有深度学习结合差分隐私的方法中模型训练周期有限、隐私预算分配不合理导致模型安全性与可用性差的问题，提出一种基于数据特征相关性和自适应差分隐私的深度学习方法（deep learning methods based on data feature Relevance and Adaptive Differential Privacy，RADP）.首先，该方法利用逐层相关性传播算法在预训练模型上计算出原始数据集上每个特征的平均相关性；然后，使用基于信息熵的方法计算每个特征平均相关性的隐私度量，根据隐私度量对特征平均相关性自适应地添加拉普拉斯噪声；在此基础上，根据加噪保护后的每个特征平均相关性，合理分配隐私预算，自适应地对特征添加拉普拉斯噪声；最后，理论分析该方法（RADP）满足

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=64211773&type=

2.28600001

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=64211775&type=

1.26999998

-差分隐私，并且兼顾安全性与可用性.同时，在三个真实数据集（MNIST，Fashion-MNIST，CIFAR-10）上的实验结果表明，RADP方法的准确率以及平均损失均优于AdLM（Adaptive Laplace Mechanism）方法、DPSGD（Differential Privacy with Stochastic Gradient Descent）方法和DPDLIGDO（Differentially Private Deep Learning with Iterative Gradient Descent Optimization）方法，并且RADP方法的稳定性仍能保持良好.

Abstract

In the deep learning privacy protection based on differential privacy

the length of the training period and the allocation of the privacy budget directly restrict the utility of the deep learning model. In the existing methods of deep learning combined with differential privacy

the model training cycle is limited and the budget allocation of a large number of feature privacy is unreasonable

which leads to poor security and availability of the model. We propose a method of deep learning methods based on data feature relevance and adaptive differential privacy (RADP). First

the method uses the layer-by-layer correlation propagation algorithm to calculate the average correlation of each feature parameter and the output result on the original data set on the pre-trained model and uses the information entropy-based method to calculate the average correlation of each feature parameter. According to the privacy metric

the Laplace noise is adaptively added to the average correlation; on this basis

according to the average correlation of each feature parameter

the privacy budget is allocated reasonably

Laplace noise is added to the feature parameters; finally

theoretical analysis shows that the method proposed in this paper satisfies

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=64211776&type=

2.28600001

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=64211788&type=

1.26999998

-differential privacy and take into account security and availability. Based on the experimental results on 3 real datasets MNIST

Fashion-MNIST

and CIFAR-10

the accuracy and average loss of RADP are better than those of the AdLM (Adaptive Laplace Mechanism) method

the DPSGD (Differential Privacy with Stochastic Gradient Descent) method and the DPDLIGDO (Differentially Private Deep Learning with Iterative Gradient Descent Optimization) method. Moreover

the stability of RADP method can still be maintained well.

关键词

Keywords

references

康海燕 , 冀源蕊 . 基于本地化差分隐私的时序位置发布方案研究 [J ] . 电子学报 , 2022 , 50 ( 9 ): 2 222 - 2232 .

KANG H Y , JI Y R . Research on time-serial location data publication based on local differential privacy [J ] . Acta Electronica Sinica , 2022 , 50 ( 9 ): 2222 - 2232 . (in Chinese)

周纯毅 , 陈大卫 , 王尚 , 等 . 分布式深度学习隐私与安全攻击研究进展与挑战 [J ] . 计算机研究与发展 , 2021 , 58 ( 5 ): 927 - 943 .

ZHOU C Y , CHEN D W , WANG S , et al . Research and challenge of distributed deep learning privacy and security attack [J ] . Journal of Computer Research and Development , 2021 , 58 ( 5 ): 927 - 943 . (in Chinese)

ROCHER L , HENDRICKX J M , DE MONTJOYE Y A . Estimating the success of re-identifications in incomplete datasets using generative models [J ] . Nature Communications , 2019 , 10 : 3069 .

刘艺璇 , 陈红 , 刘宇涵 , 等 . 联邦学习中的隐私保护技术‍ [J ] . 软件学报 , 2022 , 33 ( 3 ): 1057 - 1092 .

LIU Y X , CHEN H , LIU Y H , et al . Privacy-preserving techniques in federated learning [J ] . Journal of Software , 2022 , 33 ( 3 ): 1057 - 1092 . (in Chinese)

刘睿瑄 , 陈红 , 郭若杨 , 等 . 机器学习中的隐私攻击与防御 [J ] . 软件学报 , 2020 , 31 ( 3 ): 866 - 892 .

LIU R X , CHEN H , GUO R Y , et al . Survey on privacy attacks and defenses in machine learning [J ] . Journal of Software , 2020 , 31 ( 3 ): 866 - 892 . (in Chinese)

SHOKRI R , STRONATI M , SONG C Z , et al . Membership inference attacks against machine learning models [C ] // 2017 IEEE Symposium on Security and Privacy . Piscataway : IEEE , 2017 : 3 - 18 .

CARLINI N , LIU C , ERLINGSSON Ú , et al . The secret sharer: Evaluating and testing unintended memorization in neural networks [C ] // Proceedings of the 28th USENIX Conference on Security Symposium . New York : ACM , 2019 : 267 - 284 .

CARLINI N , TRAMER F , WALLACE E , et al . Extracting training data from large language models [EB/OL ] . ( 2021-06-15 )[ 2022-05-20 ] . https://arxiv.org/abs/2012.07805 https://arxiv.org/abs/2012.07805 .

RADFORD A , WU J , CHILD R , et al . Language models are unsupervised multitask learners [EB/OL ] . ( 2019-01-01 )[ 2022-06-09 ] . https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

DWORK C , MCSHERRY F , NISSIM K , et al . Calibrating noise to sensitivity in private data analysis [C ] // Proceedings of the Third Conference on Theory of Cryptography . New York : ACM , 2006 : 265 - 284 .

DWORK C . Differential privacy: A survey of results [C ] // Proceedings of the 5th International Conference on Theory and Ａpplications of Ｍodels of Computation . New York : ACM , 2008 : 1 - 19 .

DWORK C , ROTH A . The algorithmic foundations of differential privacy [J ] . Foundations and Trends® in Theoretical Computer Science , 2014 , 9 ( 3-4 ): 211 - 407 .

MIRONOV I . Rényi differential privacy [C ] // 2017 IEEE 30th Computer Security Foundations Symposium (CSF) . Piscataway : IEEE , 2017 : 263 - 275 .

陈思 , 付安民 , 柯海峰 , 等 . MCDP: 基于神经网络的多集群分布式差分隐私数据发布方法 [J ] . 电子学报 , 2020 , 48 ( 12 ): 2297 - 2303 .

CHEN S , FU A M , KE H F , et al . MCDP: Multi-cluster differential privacy data publishing method based on neural network [J ] . Acta Electronica Sinica , 2020 , 48 ( 12 ): 2297 - 2303 . (in Chinese)

DONG J S , ROTH A , SU W J . Gaussian differential privacy [J ] . Journal of the Royal Statistical Society Series B: Statistical Methodology , 2022 , 84 ( 1 ): 3 - 37 .

ABADI M , CHU A , GOODFELLOW I , et al . Deep learning with differential privacy [C ] // Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security . New York : ACM , 2016 : 308 - 318 .

MCSHERRY F , TALWAR K . Mechanism design via differential privacy [C ] // 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) . Piscataway : IEEE , 2007 : 94 - 103 .

BRENDAN MCMAHAN H , RAMAGE D , TALWAR K , et al . Learning differentially private recurrent language models [EB/OL ] . ( 2017-10-17 )[ 2022-05-20 ] . http://export.‍arxiv.org/abs/1710.06963 http://export.‍arxiv.org/abs/1710.06963 .

GEYER R C , KLEIN T , NABI M . Differentially private federated learning: A client level perspective [EB/OL ] . ( 2017-12-20 )[ 2022-05-20 ] . http://export.arxiv.org/abs/1712.07557 http://export.arxiv.org/abs/1712.07557 .

BASSILY R , SMITH A , THAKURTA A . Private empirical risk minimization: Efficient algorithms and tight error bounds [C ] // 2014 IEEE 55th Annual Symposium on Foundations of Computer Science . Piscataway : IEEE , 2014 : 464 - 473 .

BU Z Q , DONG J S , LONG Q , et al . Deep learning with Gaussian differential privacy [EB/OL ] . ( 2019-11-26 )[ 2022-05-20 ] . http://export.arxiv.org/abs/1911.11607 http://export.arxiv.org/abs/1911.11607 .

WANG Y X , FIENBERG S E , SMOLA A J . Privacy for free: Posterior sampling and stochastic gradient Monte Carlo [C ] // Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 . New York : ACM , 2015 : 2493 - 2502 .

LI B , CHEN C Y , LIU H , et al . On connecting stochastic gradient MCMC and differential privacy [EB/OL ] . ( 2017-12-25 )[ 2022-05-20 ] . https://arxiv.org/pdf/1712.09097.pdf https://arxiv.org/pdf/1712.09097.pdf .

KAIROUZ P , MCMAHAN B , SONG S , et al . Practical and private (deep) learning without sampling or shuffling‍ [EB/OL ] . ( 2021-12-10 )[ 2022-05-20 ] . https://arxiv.org/abs/2103.00039 https://arxiv.org/abs/2103.00039 .

PHAN N , WU X T , HU H , et al . Adaptive Laplace mechanism: Differential privacy preservation in deep learning‍ [C ] // 2017 IEEE International Conference on Data Mining (ICDM) . Piscataway : IEEE , 2017 : 385 - 394 .

XU C G , REN J , ZHANG D Y , et al . GANobfuscator: Mitigating information leakage under GAN via differential privacy [J ] . IEEE Transactions on Information Forensics and Security , 2019 , 14 ( 9 ): 2358 - 2371 .

PAPERNOT N , THAKURTA A , SONG S , et al . Tempered sigmoid activations for deep learning with differential privacy [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 10 ): 9312 - 9321 .

ZILLER A , USYNIN D , BRAREN R , et al . Medical imaging deep learning with differential privacy [J ] . Scientific Reports , 2021 , 11 : 13524 .

LIU X , ZHOU P , QIU T , et al . Blockchain-enabled contextual online learning under local differential privacy for coronary heart disease diagnosis in mobile edge computing [J ] . IEEE Journal of Biomedical and Health Informatics , 2020 , 24 ( 8 ): 2177 - 2188 .

HAN R , LI D , OUYANG J , et al . Accurate differentially private deep learning on the edge [J ] . IEEE Transactions on Parallel and Distributed Systems , 2021 , 32 ( 9 ): 2231 - 2247 .

WENG J S , WENG J , ZHANG J L , et al . DeepChain: Auditable and privacy-preserving deep learning with blockchain-based incentive [J ] . IEEE Transactions on Dependable and Secure Computing , 2021 , 18 ( 5 ): 2438 - 2455 .

IWAHANA K , YANAI N , CRUZ J P , et al . SPGC: An integrated framework of secure computation and differential privacy for collaborative learning [C ] // Data Privacy Management, Cryptocurrencies and Blockchain Technology . New York : ACM , 2021 : 89 - 105 .

TORFI A , FOX E A , REDDY C K . Differentially private synthetic medical data generation using convolutional GANs [J ] . Information Sciences , 2022 , 586 : 485 - 500 .

Papernot N , Song S , Mironov I , et al . Scalable private learning with PATE [EB/OL ] . ( 2018-02-24 )[ 2022-05-20 ] . http://export.arxiv.org/abs/1802.08908 http://export.arxiv.org/abs/1802.08908 .

谭作文 , 张连福 . 机器学习隐私保护研究综述 [J ] . 软件学报 , 2020 , 31 ( 7 ): 2127 - 2156 .

TAN Z W , ZHANG L F . Survey on privacy preserving techniques for machine learning [J ] . Journal of Software , 2020 , 31 ( 7 ): 2127 - 2156 . (in Chinese)

JAYARAMAN B , WANG L X , EVANS D , et al . Distributed learning without distress: Privacy-preserving empirical risk minimization [C ] // Proceedings of the 32nd International Conference on Neural Information Processing Systems . New York : ACM , 2018 : 6346 - 6357 .

LU Z G , ASGHAR H J , KAAFAR M ALI , et al . A differentially private framework for deep learning with convexified loss functions [J ] . IEEE Transactions on Information Forensics and Security , 2022 , 17 : 2151 - 2165 .

CHENG A D , WANG J X , ZHANG X S , et al . DPNAS: Neural architecture search for deep learning with differential privacy [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 6 ): 6358 - 6366 .

DING X F , CHEN L , ZHOU P , et al . Differentially private deep learning with iterative gradient descent optimization [J ] . ACM/IMS Transactions on Data Science , 2022 , 2 ( 4 ): 1 - 27 .

DWORK C , ROTHBLUM G N , VADHAN S . Boosting and differential privacy [C ] // 2010 IEEE 51st Annual Symposium on Foundations of Computer Science . Piscataway : IEEE , 2010 : 51 - 60 .

KIFER D , LIN B R . Towards an axiomatization of statistical privacy and utility [C ] // Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems . New York : ACM , 2010 : 147 - 158 .

BACH S , BINDER A , MONTAVON G , et al . On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation [J ] . PLoS One , 2015 , 10 ( 7 ): e0130140 .

SHANNON C E . A mathematical theory of communication [J ] . Bell System Technical Journal , 1948 , 27 ( 4 ): 623 - 656 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Differentially Private with Sparse and Smooth Self-Distillation

Encrypted Traffic Pre-Trained Model with Long-Range Dependency and Local Dynamic Perception

Lightweight Intrusion Detection in the Edge Internet of Things with Dual-Teacher Distillation

Demodulation Method for Starlink Signals Based on Correlation and Clustering

Remote Traffic Fingerprinting Attack Based on Cell Sequence Reconstruction

Related Author

ZHAO Deng-feng

XUE Da-xuan

ZHAO Su-yun

CHEN Hong

ZHANG Lei

LIU Xiao

ZHANG Longgang

HE Qing

Related Institution

School of Information, Renmin University of China

School of Microelectronics and Communication Engineering, Chongqing University

Chongqing Key Laboratory of Bio-Perception & Multimodal Intelligent Information Processing

Xi’an University of Posts and Telecommunications

Hangzhou Dianzi University

⁰