Differentially Private with Sparse and Smooth Self-Distillation

ZHAO Deng-feng; XUE Da-xuan; ZHAO Su-yun; CHEN Hong

doi:10.12263/DZXB.20250133

您当前的位置：

首页 >

文章列表页 >

Differentially Private with Sparse and Smooth Self-Distillation

PAPERS | 更新时间：2025-12-27

- Differentially Private with Sparse and Smooth Self-Distillation
- ACTA ELECTRONICA SINICA Vol. 53, Issue 9, Pages: 3310-3318(2025)
- 作者机构：
  
  中国人民大学信息学院，北京 100081
- 作者简介：
- 基金信息：
  
  National Key Research and Development Program of China(2023YFB4503600);National Natural Science Foundation of China(U23A20299;U24B20144;62172424;62276270;62322214)
- DOI：10.12263/DZXB.20250133
  CLC： TP39;
- Received：21 February 2025，
  
  Accepted：24 June 2025，
  
  Published：25 September 2025
- 稿件说明：
移动端阅览
赵登峰, 薛大暄, 赵素云, 等. 基于稀疏平滑自蒸馏的差分隐私深度学习方法[J]. 电子学报, 2025, 53(09): 3310-3318.

ZHAO Deng-feng, XUE Da-xuan, ZHAO Su-yun, et al. Differentially Private with Sparse and Smooth Self-Distillation[J]. Acta Electronica Sinica, 2025, 53(09): 3310-3318.
赵登峰, 薛大暄, 赵素云, 等. 基于稀疏平滑自蒸馏的差分隐私深度学习方法[J]. 电子学报, 2025, 53(09): 3310-3318. DOI：10.12263/DZXB.20250133

ZHAO Deng-feng, XUE Da-xuan, ZHAO Su-yun, et al. Differentially Private with Sparse and Smooth Self-Distillation[J]. Acta Electronica Sinica, 2025, 53(09): 3310-3318. DOI：10.12263/DZXB.20250133

摘要

为了减少深度学习中隐私泄露的风险，许多研究利用差分隐私技术来训练神经网络.然而，这些隐私保护方法通常会导致模型性能显著下降.为了在隐私保护与模型效用之间实现平衡，本文提出了一种基于稀疏平滑自蒸馏的差分隐私深度学习（Differentially Private learning with sparse and smooth Self-Distillation，DP3SD）方法，通过双温度缩放机制来增强隐私保护深度学习的效用.具体而言，该方法设计了一种由稀疏分类损失和光滑蒸馏损失组成的双温度缩放损失函数.通过将较低温度应用于分类损失，能够使学生模型的类别预测分布更加锐化，从而减少低概率类别的影响，这些类别通常可能是由噪声引起的.相反，较高温度应用于蒸馏损失，能够平滑教师模型和学生模型的预测分布，从而在差分隐私约束下实现稳定和高效的知识迁移.在差分隐私随机梯度下降的严格隐私保障下，本文提出的双重缩放机制能够减轻噪声带来的扰动，提升学生模型的泛化能力.在三个公开数据集上的大量实验表明：本文提出的方法能够在确保严格数据隐私的同时，增强模型的可用性.

Abstract

To mitigate privacy leakage risks in deep learning

numerous studies utilize differential privacy techniques to train neural networks. However

substantial performance degradation is often unavoidable. To address the privacy-utility trade-off

we propose the differentially private learning with sparse and smooth self-distillation (DP3SD) method

which leverages dual temperature scaling to enhance the utility of privacy-preserving deep learning. Specifically

DP3SD proposes a dual scaling loss function composed of a sparse classification loss and a smooth distillation loss. By incorporating a lower temperature into the classification loss

the class prediction distribution of student model is sharpened

thereby reducing the influence of low-probability classes that are likely noise-induced. Conversely

applying a higher temperature to the distillation loss

the prediction distributions of both the teacher and student models are smoothed

thus promoting stable and efficient knowledge transfer under differential privacy constraints. This dual scaling mechanism

under strict privacy guarantees through differential privacy stochastic gradient descent

facilitates the student model in progressively enhancing its learning from the teacher model while simultaneously alleviating the perturbations caused by privacy constraints. By extensive experiments on three public datasets

we find that DP3SD can effectively improve model performance while ensuring rigorous data privacy.

关键词

Keywords

references

SONG C Z , RISTENPART T , SHMATIKOV V . Machine learning models that remember too much [C ] // Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . New York : ACM , 2017 : 587 - 601 .

WANG L , THAKKAR O , MATHEWS R . Unintended memorization in large ASR models, and how to mitigate it [C ] // ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing . Piscataway : IEEE , 2024 : 4655 - 4659 .

CARLINI N , TRAMER F , WALLACE E , et al . Extracting training data from large language models [C ] // Proceedings of the 30rd USENIX Security Symposium . Berkeley : USENIX Association , 2021 : 2633 - 2650

WANG L J , WANG J J , WAN J , et al . Property existence inference against generative models [C ] // Proceedings of the 33rd USENIX Security Symposium . Berkeley : USENIX Association , 2024 : 2423 - 2440 .

DWORK C , MCSHERRY F , NISSIM K , et al . Calibrating noise to sensitivity in private data analysis [C ] // Theory of Cryptography . Berlin : Springer , 2006 : 265 - 284 .

LI Z , WU Y T , CHEN Y H , et al . Membership inference attacks against large vision-language models [C ] // Proceedings of the 38th International Conference on Neural Information Processing Systems . New York : ACM , 2025 : 98645 - 98674 .

ABADI M , CHU A , GOODFELLOW I , et al . Deep learning with differential privacy [C ] // Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security . New York : ACM , 2016 : 308 - 318 .

TANG X Y , PANDA A , SEHWAG V , et al . Differentially private image classification by learning priors from random processes [C ] // Proceedings of the 37th International Conference on Neural Information Processing Systems . New York : ACM , 2023 : 35855 - 35877 .

NASRSAEED MAHLOUJIFAR M , MAHLOUJIFAR S , TANG X Y , et al . Effectively using public data in privacy preserving machine learning [C ] // Proceedings of the 40th International Conference on Machine Learning . Cambridge : PMLR , 2023 : 25718 - 25732 .

FURLANELLO T , LIPTON Z , TSCHANNEN M , et al . Born again neural networks [C ] // Proceedings of Machine Learning Research . Cambridge : PMLR , 2018 : 1607 - 1616 .

DWORK C , ROTH A . The algorithmic foundations of differential privacy [J ] . Foundations and Trends in Theoretical Computer Science , 2014 , 9 ( 3/4 ): 211 - 407 .

GENTRY C , HALEVI S . Implementing gentry’s fully-homomorphic encryption scheme [C ] // Advances in Cryptology-EUROCRYPT 2011 . Berlin : Springer , 2011 : 129 - 148 .

刘艺璇 , 陈红 , 刘宇涵 , 等 . 联邦学习中的隐私保护技术 [J ] . 软件学报 , 2022 , 33 ( 3 ): 1057 - 1092 .

LIU Y X , CHEN H , LIU Y H , et al . Privacy-preserving techniques in federated learning [J ] . Journal of Software , 2022 , 33 ( 3 ): 1057 - 1092 . (in Chinese)

MIRONOV I . Rényi differential privacy [C ] // 2017 IEEE 30th Computer Security Foundations Symposium . Piscataway : IEEE , 2017 : 263 - 275 .

GOPI S , LEE Y T , WUTSCHITZ L . Numerical composition of differential privacy [C ] // Proceedings of the 35th International Conference on Neural Information Processing Systems . New York : ACM , 2021 : 11631 - 11642 .

方晨 , 郭渊博 , 王娜 , 等 . 基于生成对抗网络的差分隐私数据发布方法 [J ] . 电子学报 , 2020 , 48 ( 10 ): 1983 - 1992 .

FANG C , GUO Y B , WANG N , et al . Differential private data publishing method based on generative adversarial network [J ] . Acta Electronica Sinica , 2020 , 48 ( 10 ): 1983 - 1992 . (in Chinese)

康海燕 , 王骁识 . 基于数据特征相关性和自适应差分隐私的深度学习方法研究 [J ] . 电子学报 , 2024 , 52 ( 6 ): 1963 - 1976 .

KANG H Y , WANG X S . Research on the deep learning method based on data feature relevance and adaptive differential privacy [J ] . Acta Electronica Sinica , 2024 , 52 ( 6 ): 1963 - 1976 . (in Chinese)

HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network [EB/OL ] . ( 2015-03-09 )[ 2025-02-20 ] . https://arXiv.org/abs/1503.02531 https://arXiv.org/abs/1503.02531 .

刘松 , 罗杨宇 , 许佳培 , 等 . 基于轻量自蒸馏的低成本联邦学习 [J ] . 电子学报 , 2025 , 53 ( 1 ): 259 - 269 .

LIU S , LUO Y Y , XU J P , et al . Low-cost federated learning based on lightweight self-distillation [J ] . Acta Electronica Sinica , 2025 , 53 ( 1 ): 259 - 269 . (in Chinese)

KIM K , JI B , YOON D , et al . Self-knowledge distillation with progressive refinement of targets [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2022 : 6547 - 6556 .

SHEJWALKAR V , GANESH A , MATHEWS R , et al . Recycling scraps: Improving private learning by leveraging intermediate checkpoints [EB/OL ] . ( 2024-09-17 )[ 2025-02-20 ] . https://arxiv.org/abs/2210.01864 https://arxiv.org/abs/2210.01864 .

DE S , BERRADA L , HAYES J , et al . Unlocking high-accuracy differentially private image classification through scale [EB/OL ] . ( 2022-06-16 )[ 2025-02-20 ] . https://arXiv.org/abs/2204.13650 https://arXiv.org/abs/2204.13650 .

ALTSCHULER J M , TALWAR K . Privacy of noisy stochastic gradient descent: More iterations without more privacy loss [C ] // Proceedings of the 36th Conference on Neural Information Processing Systems . San Diego : NeurIPS , 2022 : 3788 - 3800 .

BOENISCH F , MÜHL C , DZIEDZIC A , et al . Have it your way: Individualized privacy assignment for DP-SGD [C ] // Proceedings of the 37th Conference on Neural Information Processing Systems . San Diego : NeurIPS , 2024 : 8 - 12 .

DENG L . The MNIST database of handwritten digit images for machine learning research [best of the web] [J ] . IEEE Signal Processing Magazine , 2012 , 29 ( 6 ): 141 - 142 .

XIAO H , RASUL K , VOLLGRAF R . Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms [EB/OL ] . ( 2017-08-25 )[ 2025-02-20 ] . https://arXiv.org/abs/1708.07747 https://arXiv.org/abs/1708.07747 .

KRIZHEVSKY A . Convolutional deep belief networks on CIFAR-10 [EB/OL ] . ( 2010-12-21 )[ 2025-02-20 ] . https://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf https://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf .

FLEMINGS J , ANNAVARAM M . Differentially private knowledge distillation via synthetic text generation [C ] // Findings of the Association for Computational Linguistics ACL 2024 . Stroudsburg : ACL , 2024 : 12957 - 12968 .

TRAMÈR F , BONEH D . Differentially private learning needs better features (or much more data) [EB/OL ] . ( 2021-02-18 )[ 2025-02-20 ] . https://arXiv.org/abs/2011.11660 https://arXiv.org/abs/2011.11660 .

PAPERNOT N , THAKURTA A , SONG S , et al . Tempered sigmoid activations for deep learning with differential privacy [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 10 ): 9312 - 9321 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Low-Rank Adaptation Based Flexibility-Aware Distillation Method

Research on the Deep Learning Method Based on Data Feature Relevance and Adaptive Differential Privacy

Trajectory Privacy Protection Method Based on Differential Privacy

Related Author

LI Jia-ming

BAO Zhi-qiang

HUANG Zhen-hua

SUN Sheng-li

CHEN Yun-wen

KANG Hai-yan

WANG Xiao-shi

YUAN Shui-lian

Related Institution

School of Computer Science, South China Normal University

Research Institute of Hundsun Technologies Co., Ltd.

School of Software & Microelectronics, Peking University

DataGrand Co., Ltd.

Information Security Department， Beijing Information Science and Technology University

⁰