Medical Image Synthesis Using Robust Conditional GAN

LIU Shao-peng; ZHAO Hui-min; HONG Jia-ming; WU Xiao-hang; XU Fa-bao; OUYANG Jia; LIANG Peng; XIONG Jian-bin

doi:10.12263/DZXB.20210051

您当前的位置：

首页 >

文章列表页 >

Medical Image Synthesis Using Robust Conditional GAN

PAPERS | 更新时间：2025-12-08

- Medical Image Synthesis Using Robust Conditional GAN
- ACTA ELECTRONICA SINICA Vol. 51, Issue 2, Pages: 427-437(2023)
- 作者机构：
  
  1.广东技术师范大学计算机科学学院，广东广州 510665
  2.广州大学管理学院，广东广州 510006
  3.广东省大数据分析与处理重点实验室，广东广州 510006
  4.广州中医药大学医学信息工程学院，广东广州 510006
  5.中山大学中山眼科中心眼科学国家重点实验室，广东广州 510060
  6.山东大学齐鲁医院眼科，山东济南 250012
  7.广东技术师范大学网络空间安全学院，广东广州 510665
  8.广东技术师范大学自动化学院，广东广州 510665
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20210051
  CLC： TN957;
- Received：31 December 2020，
  
  Revised：2022-02-20，
  
  Published：25 February 2023
- 稿件说明：
移动端阅览
刘少鹏,赵慧民,洪佳明等.面向医学图像生成的鲁棒条件生成对抗网络[J].电子学报,2023,51(02):427-437.

LIU Shao-peng,ZHAO Hui-min,HONG Jia-ming,et al.Medical Image Synthesis Using Robust Conditional GAN[J].ACTA ELECTRONICA SINICA,2023,51(02):427-437.
刘少鹏,赵慧民,洪佳明等.面向医学图像生成的鲁棒条件生成对抗网络[J].电子学报,2023,51(02):427-437. DOI： 10.12263/DZXB.20210051.

LIU Shao-peng,ZHAO Hui-min,HONG Jia-ming,et al.Medical Image Synthesis Using Robust Conditional GAN[J].ACTA ELECTRONICA SINICA,2023,51(02):427-437. DOI： 10.12263/DZXB.20210051.

摘要

医学图像生成是计算机辅助诊断技术的关键组成，具有广泛的应用场景.当前基于生成对抗网络的端对端学习模型，依靠生成器和判别器两者对抗训练，获取真实数据的概率分布，从而指导图像生成.但标注有限的医学图像及其高分辨率特点，加大了模型训练难度，影响图像生成质量；同时，模型未纳入数据扰动因素，鲁棒性有限，容易被恶意攻击.为此，本文提出一个基于鲁棒条件生成对抗网络的医学图像生成模型——MiSrc-GAN.该模型包括精度渐进生成器、多尺度判别器以及对抗样本配对构造模块，有效融合GAN框架和对抗样本，改善判别器鲁棒性，有利于学习原始图像与待生成图像的联合概率分布.在真实数据集CSC和REFUGE上的实验表明，MiSrc-GAN生成的图像质量优于现有模型.

Abstract

Medical image synthesis is a key technology in computer aided diagnosis due to its wide applications. Recently

the end-to-end models based on generative adversarial networks (GAN) learn the true data distribution to guide the image generation via the competing generator and discriminator. However

the high resolution of the medical images and the lack of labeled samples make these models hard to train. Furthermore

they are not robust against data perturbations and vulnerable to malicious attacks. In this paper

we propose a novel image synthesis model using robust conditional GAN

namely MiSrc-GAN. The MiSrc-GAN contains a progressive resolution generator

a multi-scale discriminator and a paired adversarial examples generating module. Through effective integrating GAN framework and adversarial examples

the MiSrc-GAN is able to simultaneously improve the robustness of the multi-scale discriminator and the quality of synthetic images under the joint probability distribution of the original medical images and their translating versions. The extensive experiments show that the proposed method achieves state-of-the-art image synthesis results on both CSC and REFUGE datasets.

关键词

Keywords

references

郑光远 , 刘峡壁 , 韩光辉 . 医学影像计算机辅助检测与诊断系统综述 [J]. 软件学报 , 2018 , 29 ( 5 ): 1471 - 1514 .

ZHENG G Y , LIU X B , HAN G H . Survey on medical image computer aided detection and diagnosis systems [J]. Journal of Software , 2018 , 29 ( 5 ): 1471 - 1514 . (in Chinese)

YI X , WALIA E , BABYN P . Generative adversarial network in medical imaging: A review [J]. Medical Image Analysis , 2019 , 58 : 101552 .

GOODFELLOW I J , POUGET-ABADIE J , MIRZA M , et al . Generative adversarial nets [C]// NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 . Montreal : ACM , 2014 : 2672 - 2680 .

HUANG H , YU P S , WANG C . An introduction to image synthesis with generative adversarial nets [EB/OL]. ( 2018 )[2021]. https://arxiv.org/abs/1803.04469 https://arxiv.org/abs/1803.04469 .

田娟秀 , 刘国才 , 谷珊珊 , 等 . 医学图像分析深度学习方法研究与挑战 [J]. 自动化学报 , 2018 , 44 ( 3 ): 401 - 424 .

TIAN J X , LIU G C , GU S S , et al . Deep learning in medical image analysis and its challenges [J]. Acta Automatica Sinica , 2018 , 44 ( 3 ): 401 - 424 . (in Chinese)

YU B , WANG Y , WANG L , et al . Medical image synthesis via deep learning [J]. Advances in Experimental Medicine and Biology , 2020 , 1213 : 23 - 44 .

KAZEMINIA S , BAUR C , KUIJPER A , et al . GANs for medical image analysis [J]. Artificial Intelligence in Medicine , 2020 , 109 : 101938 .

RADFORD A , METZ L , CHINTALA S . Unsupervised representation learning with deep convolutional generative adversarial networks [EB/OL]. ( 2015 )[2021]. https://arxiv.org/abs/1511.06434 https://arxiv.org/abs/1511.06434 .

DENTON E , CHINTALA S , SZLAM A , et al . Deep generative image models using a Laplacian pyramid of adversarial networks [C]// Advances in Neural Information Processing Systems(NIPS) . Montreal : MIT Press , 2015 : 1486 - 1494 .

MIRZA M , OSINDERO S . Conditional generative adversarial nets [EB/OL]. ( 2014 )[2021]. https://arxiv.org/abs/1411.1784 https://arxiv.org/abs/1411.1784 .

ODENA A , OLAH C , SHLENS J . Conditional image synthesis with auxiliary classifier gans [EB/OL]. ( 2017 )[2021]. https://arxiv.org/abs/1610.09585 https://arxiv.org/abs/1610.09585 .

KARRAS T , AILA T , LAINE S . Progressive growing of gans for improved quality, stability, and variation [EB/OL]. ( 2018 )[2021]. https://arxiv.org/abs/1710.10196 https://arxiv.org/abs/1710.10196 .

WANG T C , LIU M Y , ZHU J Y , et al . High-resolution image synthesis and semantic manipulation with conditional GANs [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 8798 - 8807 .

SZEGEDY C , ZAREMBA W , SUTSKEVER I , et al . Intriguing properties of neural networks [EB/OL]. ( 2014 )[2021]. https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 .

XIAO C , LI B , ZHU J Y , et al . Generating adversarial examples with adversarial networks [EB/OL]. ( 2018 )[2021]. https://arxiv.org/abs/1801.02610 https://arxiv.org/abs/1801.02610 .

JANDIAL S , MANGLA P , VARSHNEY S , et al . AdvGAN: harnessing latent layers for adversary generation [C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Seoul : IEEE , 2019 : 2045 - 2048 .

LIU X Q , HSIEH C J . Rob-GAN: Generator, discriminator, and adversarial attacker [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) . Long Beach : IEEE , 2019 : 11226 - 11235 .

MADRY A , MAKELOV A , SCHMIDT L , et al . Towards deep learning models resistant to adversarial attacks [EB/OL]. ( 2018 )[2021]. https://arxiv.org/abs/1706.06083 https://arxiv.org/abs/1706.06083 .

JOHNSON J , ALAHI A , LI F F . Perceptual losses for real-time style transfer and super-resolution [C]// Computer Vision - ECCV 2016 . Cham : Springer International Publishing , 2016 : 694 - 711 .

DURUGKAR I , GEMP I M , MAHADEVAN S . Generative multi-adversarial networks [C]// Proceedings of the International Conference on Learning Representations(ICLR) . New York : ACM , 2017 : 1 - 14 .

周沛 , 陈后金 , 于泽宽 , 等 . 跨模态医学图像预测综述 [J]. 电子学报 , 2019 , 47 ( 1 ): 220 - 226 .

ZHOU P , CHEN H J , YU Z K , et al . Review of cross-modality medical image prediction [J]. Acta Electronica Sinica , 2019 , 47 ( 1 ): 220 - 226 . (in Chinese)

BEN-COHEN A , KLANG E , RASKIN S , et al . Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection [EB/OL]. ( 2019 )[2021]. https://arxiv.org/abs/1802.07846 https://arxiv.org/abs/1802.07846 .

SHELHAMER E , LONG J , DARRELL T . Fully convolutional networks for semantic segmentation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 4 ): 640 - 651 .

PAN Y S , LIU M X , LIAN C F , et al . Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for alzheimer\u2019s disease diagnosis [C]// Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 . Cham : Springer International Publishing , 2018 : 455 - 463 .

NIE D , TRULLO R , LIAN J , et al . Medical image synthesis with deep convolutional adversarial networks [J]. IEEE Transactions on Biomedical Engineering , 2018 , 65 ( 12 ): 2720 - 2730 .

WOLTERINK J M , DINKLA A M , SAVENIJE M H F , et al . Deep MR to CT synthesis using unpaired data [M]// Simulation and Synthesis in Medical Imaging . Cham : Springer International Publishing , 2017 : 14 - 23 .

ZHU J Y , PARK T , ISOLA P , et al . Unpaired image-to-image translation using cycle-consistent adversarial networks [C]// 2017 IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 2242 - 2251 .

YANG H R , SUN J , CARASS A , et al . Unsupervised MR-to-CT synthesis using structure-constrained CycleGAN [J]. IEEE Transactions on Medical Imaging , 2020 , 39 ( 12 ): 4249 - 4261 .

DAR S U , YURT M , KARACAN L , et al . Image synthesis in multi-contrast MRI with conditional generative adversarial networks [J]. IEEE Transactions on Medical Imaging , 2019 , 38 ( 10 ): 2375 - 2388 .

ZHOU T , FU H Z , CHEN G , et al . Hi-net: Hybrid-fusion network for multi-modal MR image synthesis [J]. IEEE Transactions on Medical Imaging , 2020 , 39 ( 9 ): 2772 - 2781 .

YOU Q J , WAN C , SUN J , et al . Fundus image enhancement method based on CycleGAN [C]// 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society . Berlin : IEEE , 2019 : 4500 - 4503 .

LI W Y , LIU G X , HE Y , et al . Quality improvement of adaptive optics retinal images using conditional adversarial networks [J]. Biomedical Optics Express , 2020 , 11 ( 2 ): 831 - 849 .

TOM F , SHEET D . Simulating patho-realistic ultrasound images using deep generative networks with adversarial learning [C]// 2018 IEEE 15th International Symposium on Biomedical Imaging . Washington : IEEE , 2018 : 1174 - 1177 .

ZHANG H , XU T , LI H S , et al . StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks [C]// 2017 IEEE International Conference on Computer Vision . Venice : IEEE , 2017 : 5908 - 5916 .

MA Y H , LIU Y H , CHENG J , et al . Cycle structure and illumination constrained GAN for medical image enhancement [C]// Medical Image Computing and Computer Assisted Intervention - MICCAI 2020 . Cham : Springer International Publishing , 2020 : 667 - 677 .

LIU Y , YANG J , ZHOU Y , et al . Prediction of OCT images of short-term response to anti-VEGF treatment for neovascular age-related macular degeneration using generative adversarial network [J]. British Journal of Ophthalmology , 2020 , 104 ( 12 ): 1735 - 1740 .

BAILO O , HAM D , SHIN Y M . Red blood cell image generation for data augmentation using conditional generative adversarial networks [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Long Beach : IEEE , 2019 : 1039 - 1048 .

RUAN Y N , LI D W , MARSHALL H , et al . MB-FSGAN: Joint segmentation and quantification of kidney tumor on CT by the multi-branch feature sharing generative adversarial network [J]. Medical Image Analysis , 2020 , 64 : 101721 .

XUE Y , XU T , ZHANG H , et al . SegAN: Adversarial network with multi-scale L1 loss for medical image segmentation [J]. Neuroinformatics , 2018 , 16 ( 3 ): 383 - 392 .

LIU S P , HONG J M , LU X , et al . Joint optic disc and cup segmentation using semi-supervised conditional GANs [J]. Computers in Biology and Medicine , 2019 , 115 : 103485 .

刘少鹏 , 洪佳明 , 梁杰鹏 , 等 . 面向医学图像分割的半监督条件生成对抗网络 [J]. 软件学报 , 2020 , 31 ( 8 ): 2588 - 2602 .

LIU S P , HONG J M , LIANG J P , et al . Medical image segmentation using semi-supervised conditional generative adversarial nets [J]. Journal of Software , 2020 , 31 ( 8 ): 2588 - 2602 . (in Chinese)

WOLTERINK J M , LEINER T , VIERGEVER M A , et al . Generative adversarial networks for noise reduction in low-dose CT [J]. IEEE Transactions on Medical Imaging , 2017 , 36 ( 12 ): 2536 - 2545 .

ZHOU L , SCHAEFFERKOETTER J D , THAM I W K , et al . Supervised learning with cyclegan for low-dose FDG PET image denoising [J]. Medical Image Analysis , 2020 , 65 : 101770 .

ARJOVSKY M , CHINTALA S , BOTTOU L . Wasserstein Gan [EB/OL]. ( 2017 )[2021]. https://arxiv.org/abs/1701.07875 https://arxiv.org/abs/1701.07875 .

YI X , BABYN P . Sharpness-aware low-dose CT denoising using conditional generative adversarial network [J]. Journal of Digital Imaging , 2018 , 31 ( 5 ): 655 - 669 .

袁子晗 , 蒋明峰 , 李杨 , 等 . 基于DAWGAN-GP的磁共振图像重构方法研究 [J]. 电子学报 , 2020 , 48 ( 10 ): 1883 - 1890 .

YUAN Z H , JIANG M F , LI Y , et al . Research of MRI reconstruction method by using de-aliasing Wasserstein generative adversarial networks with gradient penalty [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1883 - 1890 . (in Chinese)

LIU Z C , BICER T , KETTIMUTHU R , et al . TomoGAN: Low-dose synchrotron X-ray tomography with generative adversarial networks: Discussion [J]. Journal of the Optical Society of America A , 2020 , 37 ( 3 ): 422 - 434 .

MA X J , NIU Y H , GU L , et al . Understanding adversarial attacks on deep learning based medical image analysis systems [J]. Pattern Recognition , 2021 , 110 : 107332 .

PAUL R , SCHABATH M , GILLIES R , et al . Mitigating adversarial attacks on medical image understanding systems [C]// 2020 IEEE 17th International Symposium on Biomedical Imaging . Iowa City : IEEE , 2020 : 1517 - 1521 .

HIRANO H , KOGA K , TAKEMOTO K . Vulnerability of deep neural networks for detecting covid-19 cases from chest X-ray images to universal adversarial attacks [EB/OL]. ( 2020 )[2021]. https://arxiv.org/abs/2005.11061 https://arxiv.org/abs/2005.11061 .

ZHAO Z , DUA D , SINGH S . Generating natural adversarial examples [C]// Proceedings of the International Conference on Learning Representations (ICLR) . New York : ACM , 2018 : 1 - 15 .

CISSE M , BOJANOWSKI P , GRAVE E , et al . Parseval networks: Improving robustness to adversarial examples [C]// ICML'17: Proceedings of the 34th International Conference on Machine Learning-Volume 70 . New York : ACM , 2017 : 854 - 863 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [C]// Proceedings of the International Conference on Learning Representations . New York : ACM , 2015 : 1 - 14 .

罗会兰 , 敖阳 , 袁璞 . 一种生成对抗网络用于图像修复的方法 [J]. 电子学报 , 2020 , 48 ( 10 ): 1891 - 1898 .

LUO H L , AO Y , YUAN P . Image inpainting using generative adversarial networks [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1891 - 1898 . (in Chinese)

ZHANG R , ISOLA P , EFROS A A , et al . The unreasonable effectiveness of deep features as a perceptual metric [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City : IEEE , 2018 : 586 - 595 .

FU H Z , CHENG J , XU Y W , et al . Joint optic disc and cup segmentation based on multi-label deep network and polar transformation [J]. IEEE Transactions on Medical Imaging , 2018 , 37 ( 7 ): 1597 - 1605 .

SEVASTOPOLSKY A . Optic disc and cup segmentation methods for glaucoma detection with modification of U-Net convolutional neural network [J]. Pattern Recognition and Image Analysis , 2017 , 27 ( 3 ): 618 - 624 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Progressive Image Synthesis Method Based on Diffusion-Mamba and Scale-Invariant Loss

Robust Adversarial Defense Boundary-Based Speech Forgery Method Recognition

SwinT-Unet: Ultrasound Image Segmentation Based on Two-Channel Self-Attention Mechanism

Robust Physical Adversarial Camouflages for Image Classifiers

Related Author

ZHAO Hui-min

HONG Jia-ming

WU Xiao-hang

XU Fa-bao

OUYANG Jia

LIANG Peng

XIONG Jian-bin

LIU Shao-peng

Related Institution

School of Management， Guangzhou University

School of Medical Information Engineering， Guangzhou University of Chinese Medicine

State Key Laboratory of Ophthalmology， Zhongshan Ophthalmic Center， Sun Yat-Sen University

Department of Ophthalmology， Qilu Hospital， Cheeloo College of Medicine， Shandong University

School of Cyber Security， Guangdong Polytechnic Normal University

⁰