Building Privacy Shield in Online Generative AI Services

QI Tao; WANG Huili; YANG Peiru; WANG Wendan; TAN Zhipeng; HUANG Yongfeng; WANG Shangguang; XU Hongyan; LUO Chuanwen

doi:10.12263/DZXB.20250793

您当前的位置：

首页 >

文章列表页 >

Building Privacy Shield in Online Generative AI Services

PAPERS | 更新时间：2026-06-22

- Building Privacy Shield in Online Generative AI Services
- ACTA ELECTRONICA SINICA Vol. 54, Issue 1, Pages: 50-67(2026)
- 作者机构：
  
  1.北京邮电大学计算机学院网络与交换技术全国重点实验室，北京 100876
  2.清华大学电子工程系，北京 100084
  3.华中科技大学武汉光电国家研究中心，湖北武汉 430074
  4.北京林业大学信息学院，北京 100083
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62425203;62502044)
- DOI：10.12263/DZXB.20250793
  CLC： TP18;TP393.08;TP311.13
- Received：10 September 2025，
  
  Accepted：07 January 2026，
  
  Published：25 January 2026
- 稿件说明：
移动端阅览
齐涛, 王慧丽, 杨珮茹, 等. 面向在线生成式人工智能服务的隐私保护方法[J]. 电子学报, 2026, 54(01): 50-67.

QI Tao, WANG Huili, YANG Peiru, et al. Building Privacy Shield in Online Generative AI Services[J]. Acta Electronica Sinica, 2026, 54(01): 50-67.
齐涛, 王慧丽, 杨珮茹, 等. 面向在线生成式人工智能服务的隐私保护方法[J]. 电子学报, 2026, 54(01): 50-67. DOI：10.12263/DZXB.20250793

QI Tao, WANG Huili, YANG Peiru, et al. Building Privacy Shield in Online Generative AI Services[J]. Acta Electronica Sinica, 2026, 54(01): 50-67. DOI：10.12263/DZXB.20250793

摘要

近年来，在线人工智能系统在众多领域展现出强大的推理能力，对社会产生了广泛的影响。在使用此类模型服务时，用户通常需将相关查询数据上传至云端平台以提供明确的任务指令。然而，这些查询数据可能包含隐私敏感或者机密信息，直接与云端平台共享会存在隐私泄露风险。此外，人工智能平台通常也会收集并利用用户数据进一步训练模型，可能导致用户的私有信息被生成式大模型记忆，并在后续公共服务中被生成并传播，从而加剧隐私泄露的可能性。现有生成式人工智能应用的隐私保护机制普遍依赖于针对提示词的脱敏技术，其安全性高度依赖敏感信息识别的准确性，通常需依赖大量标注数据进行隐私识别模型训练，不仅在实施成本上存在挑战，在训练过程中还极有可能引入新的隐私漏洞。为应对这一问题，本文提出一种新型隐私保护协同学习框架PrivateAI，该框架的核心思想是在严格保障隐私安全的前提下，充分利用分散在不同终端设备中的敏感数据，以训练本地隐私识别模型。同时，PrivateAI通过提取云端大模型推理过程中隐含的知识，并将其压缩为轻量级知识蒸馏数据集，实现对本地模型的高效性能增强。此外，针对标注数据和大模型蒸馏数据的异构性挑战，本框架引入了异构知识融合机制，用于对齐并整合来自基础模型与分布式标注数据的多源知识，从而显著提升隐私识别模型的泛化能力与隐私风险预警性能。为验证PrivateAI的实际效果，本文在两个真实医疗数据集上进行了系统评估。该框架能够在满足隐私约束的前提下，有效训练隐私识别模型，并对潜在隐私风险进行预警。在两个公开医疗数据集上的实验结果表明，PrivateAI训练得到的模型可最高提升53.7个百分点的隐私保护成功率。上述验证展现出PrivateAI在缓解隐私泄露风险方面的潜力，可作为在线智能应用中预防隐私泄露的有效工具。

Abstract

In recent years

state-of-the-art online artificial intelligence systems demonstrate remarkable capabilities in various fields

exerting broad social impacts. In order to access these model services

users are typically required to upload their personal data to the cloud platform. However

these queries may contain sensitive or confidential information

and directly sharing them with cloud platforms introduces potential privacy leakage risks. Moreover

platforms may exploit user data for further model training

causing private information to be memorized by the model and later regenerated in public services

thereby aggravating the risk of privacy breaches. Existing privacy-preserving mechanisms in generative AI applications predominantly rely on prompt sanitization techniques

whose security critically depends on the accuracy of sensitive information identification. These approaches usually require large amounts of annotated data for model training

which not only raises implementation costs but may also introduce new privacy vulnerabilities in specific scenarios. To address this issue

this paper proposes a novel privacy-preserving collaborative learning framework named PrivateAI. The core idea of this framework is to fully exploit sensitive data distributed across different devices to train local privacy identification models

while strictly ensuring data privacy. Meanwhile

PrivateAI extracts the implicit knowledge embedded in the large foundation models and compresses it into a lightweight distilled dataset

thereby achieving effective privacy detection performance enhancement of local models. In addition

to tackle the heterogeneity challenge between the knowledge extracted from labeled data and foundation models

the framework introduces a heterogeneous knowledge fusion mechanism that aligns and integrates multi-source knowledge from both the foundational models and distributed labeled datasets. We evaluate PrivateAI on two datasets

and the results demonstrate that models learned by PrivateAI can maximally improve the privacy protection success rate by 53.7 percentage points. PrivateAI holds significant potential in mitigating privacy breaches

acting as a sentinel against severe privacy leakage incidents within online AI applications.

关键词

Keywords

references

Kim M , Chen Chen , Wang Peng , et al . Detection of ovarian cancer via the spectral fingerprinting of quantum-defect-modified carbon nanotubes in serum by machine learning [J ] . Nature Biomedical Engineering , 2022 , 6 ( 3 ): 267 - 275 . DOI: 10.1038/s41551-022-00860-y http://dx.doi.org/10.1038/s41551-022-00860-y

Jumper J , Evans R , Pritzel A , et al . Highly accurate protein structure prediction with AlphaFold [J ] . Nature , 2021 , 596 ( 7873 ): 583 - 589 . DOI: 10.1038/s41586-021-03819-2 http://dx.doi.org/10.1038/s41586-021-03819-2

Reuters . ChatGPT sets record for fastest-growing user base (Analyst note) [EB/OL ] . ( 2023-02-01 ) [ 2025-08-21 ] . https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/ https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/ .

Crompton H , Burke D . Artificial intelligence in higher education: The state of the field [J ] . International Journal of Educational Technology in Higher Education , 2023 , 20 ( 1 ): 22 . DOI: 10.1186/s41239-023-00392-8 http://dx.doi.org/10.1186/s41239-023-00392-8

Gibney E . The scant science behind Cambridge Analytica’s controversial marketing techniques [J ] . Nature , 2018 : 555 : 286 - 287 . DOI: 10.1038/d41586-018-03880-4 http://dx.doi.org/10.1038/d41586-018-03880-4

Martin K D , Zimmermann J . Artificial intelligence and its implications for data privacy [J ] . Current Opinion in Psychology , 2024 , 58 : 101829 . DOI: 10.1016/j.copsyc.2024.101829 http://dx.doi.org/10.1016/j.copsyc.2024.101829

Hartley J , Sanchez P P , Haider F , et al . Neural networks memorise personal information from one sample [J ] . Scientific Reports , 2023 , 13 : 21366 . DOI: 10.1038/s41598-023-48034-3 http://dx.doi.org/10.1038/s41598-023-48034-3

Bloomberg . Samsung bans ChatGPT and other generative AI use by staff after leak [EB/OL ] . ( 2023-05-02 ) [ 2025-08-21 ] . https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak . DOI: 10.1007/978-1-4842-9367-6 http://dx.doi.org/10.1007/978-1-4842-9367-6

Ren Xuebin , Yu Chia-Mu , Yu Weiren , et al . LoPub: High-dimensional crowdsourced data publication with local differential privacy [J ] . IEEE Transactions on Information Forensics and Security , 2018 , 13 ( 9 ): 2151 - 2166 . DOI: 10.1109/tifs.2018.2812146 http://dx.doi.org/10.1109/tifs.2018.2812146

Norgeot B , Muenzen K , Peterson T A , et al . Protected Health Information filter (Philter): Accurately and securely de-identifying free-text clinical notes [J ] . npj Digital Medicine , 2020 , 3 : 57 . DOI: 10.1038/s41746-020-0258-y http://dx.doi.org/10.1038/s41746-020-0258-y

Warnat-Herresthal S , Schultze H , Shastry K L , et al . Swarm Learning for decentralized and confidential clinical machine learning [J ] . Nature , 2021 , 594 ( 7862 ): 265 - 270 .

Kaissis G A , Makowski M R , Rückert D , et al . Secure, privacy-preserving and federated machine learning in medical imaging [J ] . Nature Machine Intelligence , 2020 , 2 ( 6 ): 305 - 311 . DOI: 10.1038/s42256-020-0186-1 http://dx.doi.org/10.1038/s42256-020-0186-1

Qi Tao , Wang Huili , Huang Yongfeng . Towards the robustness of differentially private federated learning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2024 , 38 ( 18 ): 19911 - 19919 . DOI: 10.1609/aaai.v38i18.29967 http://dx.doi.org/10.1609/aaai.v38i18.29967

Mcmahan H B , Xu Zheng , Zhang Yanxiang . A hassle-free algorithm for strong differential privacy in federated learning systems [C ] // Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track . Stroudsburg : ACL , 2024 : 842 - 865 . DOI: 10.18653/v1/2024.emnlp-industry.64 http://dx.doi.org/10.18653/v1/2024.emnlp-industry.64

康海燕 , 王骁识 . 基于数据特征相关性和自适应差分隐私的深度学习方法研究 [J ] . 电子学报 , 2024 , 52 ( 6 ): 1963 - 1976 . DOI: 10.12263/DZXB.20220892 http://dx.doi.org/10.12263/DZXB.20220892

Kang Haiyan , Wang Xiaoshi . Research on the deep learning method based on data feature relevance and adaptive differential privacy [J ] . Acta Electronica Sinica , 2024 , 52 ( 6 ): 1963 - 1976 . (in Chinese) . DOI: 10.12263/DZXB.20220892 http://dx.doi.org/10.12263/DZXB.20220892

Deußer T , Sparrenberg L , Berger A , et al . A survey on current trends and recent advances in text anonymization [C ] // 2025 IEEE 12th International Conference on Data Science and Advanced Analytics . Piscataway : IEEE , 2025 : 11247969 . DOI: 10.1109/dsaa65442.2025.11247969 http://dx.doi.org/10.1109/dsaa65442.2025.11247969

Kovačević A , Bašaragin B , Milošević N , et al . De-identification of clinical free text using natural language processing: A systematic review of current approaches [J ] . Artificial Intelligence in Medicine , 2024 , 151 : 102845 . DOI: 10.1016/j.artmed.2024.102845 http://dx.doi.org/10.1016/j.artmed.2024.102845

Sweeney L . Replacing personally-identifying information in medical records, the Scrub system [J ] . Proceedings , 1996 : 333 - 337 .

Robertson S , Zaragoza H . The probabilistic relevance framework: BM25 and beyond [J ] . Foundations and Trends in Information Retrieval , 2009 , 3 ( 4 ): 333 - 389 .

Lample G , Ballesteros M , Subramanian S , et al . Neural architectures for named entity recognition [C ] // Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Stroudsburg : ACL , 2016 : 260 - 270 . DOI: 10.18653/v1/n16-1030 http://dx.doi.org/10.18653/v1/n16-1030

Kenton J D , Chang Mingwei , Toutanova L K . BERT: Pre-training of deep bidirectional transformers for language understanding [C ] // Proceedings of the NAACL , 2019 : 4171 - 4186 . DOI: 10.18653/v1/n19-1423 http://dx.doi.org/10.18653/v1/n19-1423

Houman P , Montani I , Van Landschoot R , et al . spaCy: Industrial-strength natural language processing in Python [EB/OL ] . ( 2024 ) [ 2025-09-09 ] . https://spacy.io/ https://spacy.io/ .

Akbik A , Bergmann T , Blythe D , et al . FLAIR: An easy-to-use framework for state-of-the-art NLP [C ] // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) . Kerrville : Association for Computational Linguistics 2019 : 54 - 59 . DOI: 10.18653/v1/n18-5 http://dx.doi.org/10.18653/v1/n18-5

Johnson A E W , Bulgarelli L , Pollard T J . Deidentification of free-text medical records using pre-trained bidirectional transformers [C ] // Proceedings of the ACM Conference on Health, Inference, and Learning . New York : ACM , 2020 : 214 - 221 . DOI: 10.1145/3368555.3384455 http://dx.doi.org/10.1145/3368555.3384455

Dou Yao , Krsek I , Naous T , et al . Reducing privacy risks in online self-disclosures with language models [C ] // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2024 : 13732 - 13754 . DOI: 10.18653/v1/2024.acl-long.741 http://dx.doi.org/10.18653/v1/2024.acl-long.741

Mishra K , Pagare H , Sharma K . A hybrid rule-based NLP and machine learning approach for PII detection and anonymization in financial documents [J ] . Scientific Reports , 2025 , 15 : 22729 . DOI: 10.1038/s41598-025-04971-9 http://dx.doi.org/10.1038/s41598-025-04971-9

Frikha A , Walha N , Nakka K K , et al . IncogniText: Privacy-enhancing conditional text anonymization via LLM-based private attribute randomization [C ] // Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics . The Asian Federation of Natural Language Processing and The Association for Computational Linguistics , 2025 : 2490 - 2501 . DOI: 10.18653/v1/2025.ijcnlp-long.134 http://dx.doi.org/10.18653/v1/2025.ijcnlp-long.134

Yang Tianyu , Zhu Xiaodan , Gurevych I . Robust utility-preserving text anonymization based on large language models [C ] // Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2025 : 28922 - 28941 . DOI: 10.18653/v1/2025.acl-long.1404 http://dx.doi.org/10.18653/v1/2025.acl-long.1404

Staab R , Vero M , Balunovic M , et al . Language models are advanced anonymizers [C/OL ] // Proceedings of the ICLR , 2025 , https://openreview.net/forum?id=82p8VHRsaK https://openreview.net/forum?id=82p8VHRsaK .

McMahan B , Moore E , Ramage D , et al . Communication-efficient learning of deep networks from decentralized data [C ] // Proceedings of the AISTATS , 2017 : 1273 - 1282 .

Reddi S J , Charles Z , Zaheer M , et al . Adaptive federated optimization [C/OL ] // Proceedings of the ICLR , 2021 , https://openreview.net/forum?id=LkFG3lB13U5 https://openreview.net/forum?id=LkFG3lB13U5 .

Li Tian , Sahu A K , Zaheer M , et al . Federated optimization in heterogeneous networks [C ] // Proceedings of the Third Conference on Machine Learning and Systems . Austin : MLSys , 2020 , 2 : 429 - 450 . DOI: 10.1109/ieeeconf44664.2019.9049023 http://dx.doi.org/10.1109/ieeeconf44664.2019.9049023

Dwork C , McSherry F , Nissim K , et al . Calibrating noise to sensitivity in private data analysis [M ] // Theory of Cryptography . Berlin : Springer , 2006 : 265 - 284 . DOI: 10.1007/11681878_14 http://dx.doi.org/10.1007/11681878_14

McMahan H B , Ramage D , Talwar K , et al . Learning differentially private recurrent language models [C/OL ] // Proceedings of the ICLR , 2018 , https://openreview.net/forum?id=BJ0hF1Z0b https://openreview.net/forum?id=BJ0hF1Z0b .

Wei Kang , Li Jun , Ding Ming , et al . Federated learning with differential privacy: Algorithms and performance analysis [J ] . IEEE Transactions on Information Forensics and Security , 2020 , 15 : 3454 - 3469 . DOI: 10.1109/tifs.2020.2988575 http://dx.doi.org/10.1109/tifs.2020.2988575

Girgis A , Data D , Diggavi S , et al . Shuffled model of differential privacy in federated learning [C ] // Proceedings of the AISTAS , 2021 : 2521 - 2529 . DOI: 10.1109/jsait.2021.3056102 http://dx.doi.org/10.1109/jsait.2021.3056102

赵登峰 , 薛大暄 , 赵素云 , 等 . 基于稀疏平滑自蒸馏的差分隐私深度学习方法 [J ] . 电子学报 , 2025 , 53 ( 9 ): 3310 - 3318 .

Zhao Dengfeng , Xue Daxuan , Zhao Suyun , et al . Differentially private with sparse and smooth self-distillation [J ] . Acta Electronica Sinica , 2025 , 53 ( 9 ): 3310 - 3318 . (in Chinese)

Domingo-Ferrer J , Sánchez D , Blanco-Justicia A . The limits of differential privacy (and its misuse in data release and machine learning) [J ] . Communications of the ACM , 2021 , 64 ( 7 ): 33 - 35 . DOI: 10.1145/3433638 http://dx.doi.org/10.1145/3433638

Kingma D P , Ba J . Adam: A method for stochastic optimization [C ] // Proceedings of the ICLR , 2015 . DOI: 10.1007/978-3-662-46214-0_1 http://dx.doi.org/10.1007/978-3-662-46214-0_1

Uzuner O , Luo Yuan , Szolovits P . Evaluating the state-of-the-art in automatic de-identification [J ] . Journal of the American Medical Informatics Association , 2007 , 14 ( 5 ): 550 - 563 . DOI: 10.1197/jamia.m2444 http://dx.doi.org/10.1197/jamia.m2444

Liu V , Musen M A , Chou T . Data breaches of protected health information in the United States [J ] . Jama , 2015 , 313 ( 14 ): 1471 . DOI: 10.1001/jama.2015.2252 http://dx.doi.org/10.1001/jama.2015.2252

Stubbs A , Uzuner Ö . Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus [J ] . Journal of Biomedical Informatics , 2015 , 58 : S20 - S29 . DOI: 10.1016/j.jbi.2015.07.020 http://dx.doi.org/10.1016/j.jbi.2015.07.020

Wu Xidong , Huang Feihu , Hu Zhengmian , et al . Faster adaptive federated learning [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2023 , 37 ( 9 ): 10379 - 10387 . DOI: 10.1609/aaai.v37i9.26235 http://dx.doi.org/10.1609/aaai.v37i9.26235

Sun Jingwei , Li Ang , Wang Binghui , et al . Soteria: Provable defense against privacy leakage in federated learning from representation perspective [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 9307 - 9315 . DOI: 10.1109/cvpr46437.2021.00919 http://dx.doi.org/10.1109/cvpr46437.2021.00919

Vaswani A , Shazeer N , Parmar N , et al . Attention is all you need [C ] // Proceedings of the NeurIPS . Long Beach : Curran Associates Inc. , 2017 : 5998 - 6008 .

Liu Yinhan , Ott M , Goyal N , et al . RoBERTa: A robustly optimized BERT pretraining approach [PP/OL ] . V1.arXiv ( 2019-07-26 ). https://doi.org/10.48550/arXiv.1907.11692 https://doi.org/10.48550/arXiv.1907.11692 .

Carlini N , Chien S , Nasr M , et al . Membership inference attacks from first principles [C ] // 2022 IEEE Symposium on Security and Privacy . Piscataway : IEEE , 2022 : 1897 - 1914 . DOI: 10.1109/sp46214.2022.9833649 http://dx.doi.org/10.1109/sp46214.2022.9833649

Diao E M , Ding Jie , Tarokh V . SemiFL: Semi-supervised federated learning for unlabeled clients with alternate training [C ] // Proceedings of the 36th International Conference on Neural Information Processing Systems . New Orleans : Curran Associates Inc. , 2022 : 17871 - 17884 . DOI: 10.52202/068431-1299 http://dx.doi.org/10.52202/068431-1299

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Differentially Private with Sparse and Smooth Self-Distillation

Blockchain-Based Hierarchical Federated Learning System

Trajectory Privacy Protection Method Based on Differential Privacy

Selective Learning for Incomplete Medical Data

OOD Generalization of Vehicle Trajectory Prediction Based on Causal Model of Social Interaction

Related Author

QI Tao

WANG Wendan

WANG Shangguang

ZHAO Deng-feng

XUE Da-xuan

ZHAO Su-yun

CHEN Hong

HU Rong-lei

Related Institution

Lab of Networking and Switching Technology, School of Computer Science, Beijing University of Posts and Telecommunications

School of Information, Renmin University of China

Department of Electronics and Information Engineering, Beijing Electronic Science and Technology Institute

College of Computer Science and Technology， Nanjing University of Aeronautics and Astronautics

School of Cryptography Engineering, Information Engineering University

⁰