多要素协同的文生图扩散模型反定制对抗样本

叶登攀; 唐龙; 陈思润; 刘梓毅; 吕芸娜; 石绣文

doi:10.12263/DZXB.20250596

您当前的位置：

首页 >

文章列表页 >

多要素协同的文生图扩散模型反定制对抗样本

学术论文 | 更新时间：2026-02-05

- 多要素协同的文生图扩散模型反定制对抗样本
- Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration
- 电子学报 2025年53卷第10期页码：3730-3743
- 作者机构：
  
  1.广州大学网络空间安全学院，广东广州 510799
  2.武汉大学国家网络安全学院，湖北武汉 430072
- 作者简介：
  
  [ "叶登攀男，1975年9月出生于湖北省黄冈市.现为广州大学网络空间安全学院教授、博士生导师.主要研究方向为多媒体安全、人工智能与机器学习等.中国电子学会会员编号：E190161643M.E-mail: yedp2001@163.com" ]
  [ "唐龙男，1996年8月出生于四川省攀枝花市.2025年博士毕业于武汉大学国家网络安全学院.现就职于华为技术有限公司.主要研究方向为对抗样本攻防、多模态大模型安全、人脸隐私保护等.E-mail: l_tang@whu.edu.cn" ]
  [ "陈思润男，2001年9月出生于河南省许昌市.2023年本科毕业于华南理工大学计算机科学与工程学院，现为武汉大学国家网络安全学院硕士研究生.主要研究方向为多媒体安全、对抗攻防等.中国电子学会会员编号：E190160925A.E-mail: chensirun@whu.edu.cn" ]
  [ "刘梓毅男，1995年7月出生于新疆维吾尔自治区乌鲁木齐市.2017年本科毕业于重庆交通大学信息科学与工程学院，2022年硕士毕业于桂林电子科技大学计算机与信息安全学院，现为武汉大学国家网络安全学院博士研究生.主要研究方向为网络流量分析，对抗样本攻击与防御.中国电子学会会员编号：E190184468A.E-mail: ziyi_liu@whu.edu.cn" ]
  [ "吕芸娜女，2001年3月出生于福建省泉州市.2023年本科毕业于湖南大学信息科学与工程学院，现为武汉大学国家网络安全学院硕士研究生.主要研究方向为对抗样本攻击与防御.中国电子学会会员编号：E190160924A.E-mail: lvyunna@whu.edu.cn" ]
  [ "石绣文女，2002年7月出生于湖北省黄冈市.2024年本科毕业于中国地质大学（武汉），现为武汉大学硕士研究生.主要研究方向为多媒体信息安全.E-mail: shixiuwen@whu.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(62472325)
- DOI：10.12263/DZXB.20250596
  中图分类号： TP391;
- 收稿：2025-07-07，
  
  录用：2025-10-24，
  
  纸质出版：2025-10-25
- 稿件说明：
移动端阅览
叶登攀, 唐龙, 陈思润, 等. 多要素协同的文生图扩散模型反定制对抗样本[J]. 电子学报, 2025, 53(10): 3730-3743.

YE Deng-pan, TANG Long, CHEN Si-run, et al. Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration[J]. Acta Electronica Sinica, 2025, 53(10): 3730-3743.
叶登攀, 唐龙, 陈思润, 等. 多要素协同的文生图扩散模型反定制对抗样本[J]. 电子学报, 2025, 53(10): 3730-3743. DOI：10.12263/DZXB.20250596

YE Deng-pan, TANG Long, CHEN Si-run, et al. Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration[J]. Acta Electronica Sinica, 2025, 53(10): 3730-3743. DOI：10.12263/DZXB.20250596

摘要

基于文生图扩散模型的微调技术有助于实现高质量的图像定制化生成效果，但也存在隐私泄露和被用于操纵舆论的风险.当前研究主要聚焦于构造基于提示词级别或图像级别的对抗样本来实现对生成特定人物或风格定制化图像的反制，然而却忽略了这两个不同模态层面对抗样本之间的关联性，以及模型内部功能模块之间对抗性的关联.这些不足导致现有方法生成的对抗样本在实际场景中的反定制化性能受到限制.为此，本文提出了双重反扩散对抗样本生成方法（Dual Anti-Diffusion，DADiff），这是一种反制扩散模型定制化的两阶段对抗样本生成框架，将提示词级别的对抗攻击融入图像级别对抗样本的生成过程中.在第一阶段，DADiff生成提示词级别的对抗向量，以文本层面的对抗扰动信息引导后续的图像层面对抗扰动生成；第二阶段，除了对扩散UNet模型进行端到端对抗攻击外，DADiff还对其自注意力和交叉注意力模块进行干扰，旨在打破图像像素之间的相关性，并使图像利用实例提示词向量和对抗提示词向量计算得到的交叉注意力结果保持一致.此外，DADiff还引入了局部随机时间步长梯度集成策略，通过整合多个分段时间步长的随机梯度来更新对抗扰动.在主流人脸图像数据集和艺术风格图像数据集上的实验结果表明，与现有方法相比，DADiff在跨提示词，关键词不匹配和跨模型的反定制化任务上的平均性能提升了20%.

Abstract

Fine-tuning text-to-image diffusion models enables high-quality customized image generation

yet it also introduces risks of privacy leakage and potential misuse for opinion manipulation. Current research primarily focuses on prompt- or image-level adversarial attacks to counter model customization; however

it overlooks the inter-modal correlation between prompt- and image-level adversarial perturbations

as well as the adversarial interplay among the model’s internal functional modules. This limitation restricts the practical effectiveness of existing anti-customization methods. To address this

we propose dual anti-diffusion (DADiff)

a two-stage framework that integrates prompt-level adversarial attacks into the generation of image-level adversarial examples. In the first stage

DADiff generates adversarial prompt vectors to guide the subsequent image-level perturbation. In the second stage

beyond performing an end-to-end attack on the diffusion UNet

DADiff further perturbs its self-attention and cross-attention modules—aiming to break pixel-wise correlations and enforce consistency by aligning the cross-attention maps derived from the original instance prompt and those from the adversarial prompt vector. Additionally

DADiff introduces a local-random timestep gradient ensemble strategy

which updates adversarial perturbations by aggregating stochastic gradients sampled from multiple segmented timestep intervals. Experimental results on mainstream facial and artistic style datasets show that DADiff achieves an average performance improvement of 20% over existing methods across cross-prompt

keyword-mismatch

and cross-model anti-customization scenarios.

关键词

Keywords

references

HUANG Y , HUANG J C , LIU Y F , et al . Diffusion model-based image editing: A survey [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2025 , 47 ( 6 ): 4409 - 4437 .

RAMESH A , PAVLOV M , GOH G , et al . Zero-shot text-to-image generation [EB/OL ] . ( 2021-02-26 )[ 2025-09-30 ] . https://arXiv.org/abs/2102.12092 https://arXiv.org/abs/2102.12092 .

SAHARIA C , CHAN W , SAXENA S , et al . Photorealistic text-to-image diffusion models with deep language understanding [EB/OL ] . ( 2022-05-23 )[ 2025-09-30 ] . https://arXiv.org/abs/2205.11487 https://arXiv.org/abs/2205.11487 .

NICHOL A , DHARIWAL P . Improved denoising diffusion probabilistic models [EB/OL ] . ( 2021-02-18 )[ 2025-09-30 ] . https://arXiv.org/abs/2102.09672 https://arXiv.org/abs/2102.09672 .

RUIZ N , LI Y Z , JAMPANI V , et al . DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 22500 - 22510 .

HU E J , SHEN Y L , WALLIS P , et al . LoRA: Low-rank adaptation of large language models [EB/OL ] . ( 2021-10-16 )[ 2025-09-30 ] . https://arXiv.org/abs/2106.09685 https://arXiv.org/abs/2106.09685 .

GAL R , ALALUF Y , ATZMON Y , et al . An image is worth one word: Personalizing text-to-image generation using textual inversion [EB/OL ] . ( 2022-08-02 )[ 2025-09-20 ] . https://arXiv.org/abs/2208.01618 https://arXiv.org/abs/2208.01618 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // The 31st Annual Conference on Neural Information Processing Systems . New York : Curran Associates Inc , 2017 : 30 .

HO J , JAIN A , ABBEEL P . Denoising diffusion probabilistic models [EB/OL ] . ( 2020-12-16 )[ 2025-09-30 ] . https://arXiv.org/abs/2006.11239 https://arXiv.org/abs/2006.11239 .

ROMBACH R , BLATTMANN A , LORENZ D , et al . High-resolution image synthesis with latent diffusion models [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 10674 - 10685 .

ZHUANG H M , ZHANG Y H , LIU S J . A pilot study of query-free adversarial attack against stable diffusion [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2023 : 2385 - 2392 .

YANG Y J , GAO R Y , WANG X S , et al . MMA-diffusion: Multimodal attack on diffusion models [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2024 : 7737 - 7746 .

SALMAN H , KHADDAJ A , LECLERC G , et al . Raising the cost of malicious AI-powered image editing [EB/OL ] . ( 2023-02-13 )[ 2025-09-30 ] . https://arxiv.org/abs/2302.06588 https://arxiv.org/abs/2302.06588 .

LIANG C M , WU X Y , HUA Y , et al . Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples [EB/OL ] . ( 2023-06-06 )[ 2025-09-30 ] . https://arXiv.org/abs/2302.04578 https://arXiv.org/abs/2302.04578 .

MADRY A , MAKELOV A , SCHMIDT L , et al . Towards deep learning models resistant to adversarial attacks [EB/OL ] . ( 2019-09-04 )[ 2025-09-30 ] . https://arXiv.org/abs/1706.06083 https://arXiv.org/abs/1706.06083 .

VAN LE T , PHUNG H , NGUYEN T H , et al . Anti-DreamBooth: Protecting users from personalized text-to-image synthesis [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 2116 - 2127 .

WANG F F , TAN Z T , WEI T Y , et al . SimAC: A simple anti-customization method for protecting face privacy against text-to-image synthesis of diffusion models [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2024 : 12047 - 12056 .

LIU Y S , AN J Y , ZHANG W Q , et al . Disrupting diffusion: Token-level attention erasure attack against diffusion-based customization [C ] // Proceedings of the 32nd ACM International Conference on Multimedia . New York : ACM , 2024 : 3587 - 3596 .

MOKADY R , HERTZ A , ABERMAN K , et al . Null-text inversion for editing real images using guided diffusion models [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 6038 - 6047 .

DONG Y P , LIAO F Z , PANG T Y , et al . Boosting adversarial attacks with momentum [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 9185 - 9193 .

SONG J M , MENG C L , ERMON S . Denoising diffusion implicit models [EB/OL ] . ( 2022-10-05 )[ 2025-09-30 ] . https://arXiv.org/abs/2010.02502 https://arXiv.org/abs/2010.02502 .

DHARIWAL P , NICHOL A . Diffusion models beat GANs on image synthesis [C ] // Proceedings of the 35th International Conference on Neural Information Processing Systems . New York : ACM , 2021 : 8780 - 8794 .

YANG L , ZHANG Z L , SONG Y , et al . Diffusion models: A comprehensive survey of methods and applications [J ] . ACM Computing Surveys , 2023 , 56 ( 4 ): 1 - 39 .

CAO Z Y , LI J H , XU X R . DiffusionAAE: Enhancing hyperspectral image classification with conditional diffusion model and Adversarial Autoencoder [J ] . Ecological Informatics , 2025 , 87 : 103118 .

WANG L Z , YANG S , LIU S , et al . Not all steps are created equal: Selective diffusion distillation for image manipulation [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 7438 - 7447 .

LIU Z W , LUO P , WANG X G , et al . Deep learning face attributes in the wild [C ] // 2015 IEEE International Conference on Computer Vision . Piscataway : IEEE , 2016 : 3730 - 3738 .

CAO Q , SHEN L , XIE W D , et al . VGGFace2: A dataset for recognising faces across pose and age [C ] // 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition . Piscataway : IEEE , 2018 : 67 - 74 .

DENG J K , GUO J , VERVERAS E , et al . RetinaFace: Single-shot multi-level face localisation in the wild [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2020 : 5202 - 5211 .

DENG J K , GUO J , XUE N N , et al . ArcFace: Additive angular margin loss for deep face recognition [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2020 : 4685 - 4694 .

TERHÖRST P , KOLF J N , DAMER N , et al . SER-FIQ: Unsupervised estimation of face image quality based on stochastic embedding robustness [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2020 : 5650 - 5659 .

MITTAL A , MOORTHY A K , BOVIK A C . No-reference image quality assessment in the spatial domain [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 12 ): 4695 - 4708 .

HEUSEL M , RAMSAUER H , UNTERTHINER T , et al . GANs trained by a two time-scale update rule converge to a local Nash equilibrium [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems . New York : ACM , 2017 : 6629 - 6640 .

VON P , PATIL S , LOZHKOV A , et al . Diffusers: State-of-the-art diffusion models [EB/OL ] . ( 2025-08-20 )[ 2025-09-30 ] . https://github.com/huggingface/diffusers https://github.com/huggingface/diffusers .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于类别相似性驱动的动态伪目标对抗攻击方法研究

结合自适应步长策略和数据增强机制提升对抗攻击迁移性

基于显著图的电磁信号对抗样本生成方法

基于噪声初始化、Adam-Nesterov方法和准双曲动量方法的对抗样本生成方法

气相外延GaN的拉曼散射