Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration

YE Deng-pan; TANG Long; CHEN Si-run; LIU Zi-yi; LÜ Yun-na; SHI Xiu-wen

doi:10.12263/DZXB.20250596

您当前的位置：

首页 >

文章列表页 >

Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration

PAPERS | 更新时间：2026-02-05

- Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration
- ACTA ELECTRONICA SINICA Vol. 53, Issue 10, Pages: 3730-3743(2025)
- 作者机构：
  
  1.广州大学网络空间安全学院，广东广州 510799
  2.武汉大学国家网络安全学院，湖北武汉 430072
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62472325)
- DOI：10.12263/DZXB.20250596
  CLC： TP391;
- Received：07 July 2025，
  
  Accepted：24 October 2025，
  
  Published：25 October 2025
- 稿件说明：
移动端阅览
叶登攀, 唐龙, 陈思润, 等. 多要素协同的文生图扩散模型反定制对抗样本[J]. 电子学报, 2025, 53(10): 3730-3743.

YE Deng-pan, TANG Long, CHEN Si-run, et al. Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration[J]. Acta Electronica Sinica, 2025, 53(10): 3730-3743.
叶登攀, 唐龙, 陈思润, 等. 多要素协同的文生图扩散模型反定制对抗样本[J]. 电子学报, 2025, 53(10): 3730-3743. DOI：10.12263/DZXB.20250596

YE Deng-pan, TANG Long, CHEN Si-run, et al. Anti-Customization Adversarial Examples Against Text-to-Image Diffusion Models with Multi-Element Collaboration[J]. Acta Electronica Sinica, 2025, 53(10): 3730-3743. DOI：10.12263/DZXB.20250596

摘要

基于文生图扩散模型的微调技术有助于实现高质量的图像定制化生成效果，但也存在隐私泄露和被用于操纵舆论的风险.当前研究主要聚焦于构造基于提示词级别或图像级别的对抗样本来实现对生成特定人物或风格定制化图像的反制，然而却忽略了这两个不同模态层面对抗样本之间的关联性，以及模型内部功能模块之间对抗性的关联.这些不足导致现有方法生成的对抗样本在实际场景中的反定制化性能受到限制.为此，本文提出了双重反扩散对抗样本生成方法（Dual Anti-Diffusion，DADiff），这是一种反制扩散模型定制化的两阶段对抗样本生成框架，将提示词级别的对抗攻击融入图像级别对抗样本的生成过程中.在第一阶段，DADiff生成提示词级别的对抗向量，以文本层面的对抗扰动信息引导后续的图像层面对抗扰动生成；第二阶段，除了对扩散UNet模型进行端到端对抗攻击外，DADiff还对其自注意力和交叉注意力模块进行干扰，旨在打破图像像素之间的相关性，并使图像利用实例提示词向量和对抗提示词向量计算得到的交叉注意力结果保持一致.此外，DADiff还引入了局部随机时间步长梯度集成策略，通过整合多个分段时间步长的随机梯度来更新对抗扰动.在主流人脸图像数据集和艺术风格图像数据集上的实验结果表明，与现有方法相比，DADiff在跨提示词，关键词不匹配和跨模型的反定制化任务上的平均性能提升了20%.

Abstract

Fine-tuning text-to-image diffusion models enables high-quality customized image generation

yet it also introduces risks of privacy leakage and potential misuse for opinion manipulation. Current research primarily focuses on prompt- or image-level adversarial attacks to counter model customization; however

it overlooks the inter-modal correlation between prompt- and image-level adversarial perturbations

as well as the adversarial interplay among the model’s internal functional modules. This limitation restricts the practical effectiveness of existing anti-customization methods. To address this

we propose dual anti-diffusion (DADiff)

a two-stage framework that integrates prompt-level adversarial attacks into the generation of image-level adversarial examples. In the first stage

DADiff generates adversarial prompt vectors to guide the subsequent image-level perturbation. In the second stage

beyond performing an end-to-end attack on the diffusion UNet

DADiff further perturbs its self-attention and cross-attention modules—aiming to break pixel-wise correlations and enforce consistency by aligning the cross-attention maps derived from the original instance prompt and those from the adversarial prompt vector. Additionally

DADiff introduces a local-random timestep gradient ensemble strategy

which updates adversarial perturbations by aggregating stochastic gradients sampled from multiple segmented timestep intervals. Experimental results on mainstream facial and artistic style datasets show that DADiff achieves an average performance improvement of 20% over existing methods across cross-prompt

keyword-mismatch

and cross-model anti-customization scenarios.

关键词

Keywords

references

HUANG Y , HUANG J C , LIU Y F , et al . Diffusion model-based image editing: A survey [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2025 , 47 ( 6 ): 4409 - 4437 .

RAMESH A , PAVLOV M , GOH G , et al . Zero-shot text-to-image generation [EB/OL ] . ( 2021-02-26 )[ 2025-09-30 ] . https://arXiv.org/abs/2102.12092 https://arXiv.org/abs/2102.12092 .

SAHARIA C , CHAN W , SAXENA S , et al . Photorealistic text-to-image diffusion models with deep language understanding [EB/OL ] . ( 2022-05-23 )[ 2025-09-30 ] . https://arXiv.org/abs/2205.11487 https://arXiv.org/abs/2205.11487 .

NICHOL A , DHARIWAL P . Improved denoising diffusion probabilistic models [EB/OL ] . ( 2021-02-18 )[ 2025-09-30 ] . https://arXiv.org/abs/2102.09672 https://arXiv.org/abs/2102.09672 .

RUIZ N , LI Y Z , JAMPANI V , et al . DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 22500 - 22510 .

HU E J , SHEN Y L , WALLIS P , et al . LoRA: Low-rank adaptation of large language models [EB/OL ] . ( 2021-10-16 )[ 2025-09-30 ] . https://arXiv.org/abs/2106.09685 https://arXiv.org/abs/2106.09685 .

GAL R , ALALUF Y , ATZMON Y , et al . An image is worth one word: Personalizing text-to-image generation using textual inversion [EB/OL ] . ( 2022-08-02 )[ 2025-09-20 ] . https://arXiv.org/abs/2208.01618 https://arXiv.org/abs/2208.01618 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // The 31st Annual Conference on Neural Information Processing Systems . New York : Curran Associates Inc , 2017 : 30 .

HO J , JAIN A , ABBEEL P . Denoising diffusion probabilistic models [EB/OL ] . ( 2020-12-16 )[ 2025-09-30 ] . https://arXiv.org/abs/2006.11239 https://arXiv.org/abs/2006.11239 .

ROMBACH R , BLATTMANN A , LORENZ D , et al . High-resolution image synthesis with latent diffusion models [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 10674 - 10685 .

ZHUANG H M , ZHANG Y H , LIU S J . A pilot study of query-free adversarial attack against stable diffusion [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2023 : 2385 - 2392 .

YANG Y J , GAO R Y , WANG X S , et al . MMA-diffusion: Multimodal attack on diffusion models [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2024 : 7737 - 7746 .

SALMAN H , KHADDAJ A , LECLERC G , et al . Raising the cost of malicious AI-powered image editing [EB/OL ] . ( 2023-02-13 )[ 2025-09-30 ] . https://arxiv.org/abs/2302.06588 https://arxiv.org/abs/2302.06588 .

LIANG C M , WU X Y , HUA Y , et al . Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples [EB/OL ] . ( 2023-06-06 )[ 2025-09-30 ] . https://arXiv.org/abs/2302.04578 https://arXiv.org/abs/2302.04578 .

MADRY A , MAKELOV A , SCHMIDT L , et al . Towards deep learning models resistant to adversarial attacks [EB/OL ] . ( 2019-09-04 )[ 2025-09-30 ] . https://arXiv.org/abs/1706.06083 https://arXiv.org/abs/1706.06083 .

VAN LE T , PHUNG H , NGUYEN T H , et al . Anti-DreamBooth: Protecting users from personalized text-to-image synthesis [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 2116 - 2127 .

WANG F F , TAN Z T , WEI T Y , et al . SimAC: A simple anti-customization method for protecting face privacy against text-to-image synthesis of diffusion models [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2024 : 12047 - 12056 .

LIU Y S , AN J Y , ZHANG W Q , et al . Disrupting diffusion: Token-level attention erasure attack against diffusion-based customization [C ] // Proceedings of the 32nd ACM International Conference on Multimedia . New York : ACM , 2024 : 3587 - 3596 .

MOKADY R , HERTZ A , ABERMAN K , et al . Null-text inversion for editing real images using guided diffusion models [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 6038 - 6047 .

DONG Y P , LIAO F Z , PANG T Y , et al . Boosting adversarial attacks with momentum [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 9185 - 9193 .

SONG J M , MENG C L , ERMON S . Denoising diffusion implicit models [EB/OL ] . ( 2022-10-05 )[ 2025-09-30 ] . https://arXiv.org/abs/2010.02502 https://arXiv.org/abs/2010.02502 .

DHARIWAL P , NICHOL A . Diffusion models beat GANs on image synthesis [C ] // Proceedings of the 35th International Conference on Neural Information Processing Systems . New York : ACM , 2021 : 8780 - 8794 .

YANG L , ZHANG Z L , SONG Y , et al . Diffusion models: A comprehensive survey of methods and applications [J ] . ACM Computing Surveys , 2023 , 56 ( 4 ): 1 - 39 .

CAO Z Y , LI J H , XU X R . DiffusionAAE: Enhancing hyperspectral image classification with conditional diffusion model and Adversarial Autoencoder [J ] . Ecological Informatics , 2025 , 87 : 103118 .

WANG L Z , YANG S , LIU S , et al . Not all steps are created equal: Selective diffusion distillation for image manipulation [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 7438 - 7447 .

LIU Z W , LUO P , WANG X G , et al . Deep learning face attributes in the wild [C ] // 2015 IEEE International Conference on Computer Vision . Piscataway : IEEE , 2016 : 3730 - 3738 .

CAO Q , SHEN L , XIE W D , et al . VGGFace2: A dataset for recognising faces across pose and age [C ] // 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition . Piscataway : IEEE , 2018 : 67 - 74 .

DENG J K , GUO J , VERVERAS E , et al . RetinaFace: Single-shot multi-level face localisation in the wild [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2020 : 5202 - 5211 .

DENG J K , GUO J , XUE N N , et al . ArcFace: Additive angular margin loss for deep face recognition [C ] // 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2020 : 4685 - 4694 .

TERHÖRST P , KOLF J N , DAMER N , et al . SER-FIQ: Unsupervised estimation of face image quality based on stochastic embedding robustness [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2020 : 5650 - 5659 .

MITTAL A , MOORTHY A K , BOVIK A C . No-reference image quality assessment in the spatial domain [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 12 ): 4695 - 4708 .

HEUSEL M , RAMSAUER H , UNTERTHINER T , et al . GANs trained by a two time-scale update rule converge to a local Nash equilibrium [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems . New York : ACM , 2017 : 6629 - 6640 .

VON P , PATIL S , LOZHKOV A , et al . Diffusers: State-of-the-art diffusion models [EB/OL ] . ( 2025-08-20 )[ 2025-09-30 ] . https://github.com/huggingface/diffusers https://github.com/huggingface/diffusers .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on Category Similarity-Driven Dynamic Fake Target Adversarial Attack Methods

Boosting Adversarial Transferability Through Adaptive-Learning-Rate with Data Augmentation Mechanism

An Electromagnetic Signal Adversarial Examples Generation Method Based on Saliency Map

Perturbation Initialization, Adam-Nesterov and Quasi-Hyperbolic Momentum for Adversarial Examples

RADICAL PAIR INTERACTION IN INTACT SOYBEAN LEAVES

Related Author

ZHANG Lin-jun

CHEN Jun

BAO Lei

YU Hong-xia

LU Lei-ji

BAO Lei

TAO Qing

TAO Wei

Related Institution

Department of Information Engineering, PLA Army Services University

Department of Information Engineering， PLA Army Academy of Artillery and Air Defense

PLA Academy of Military Science

Wuhan Digital Engineering Institute

North China Institute of Computer Technology

⁰