频率引导的双稀疏自注意力单图像去雨算法

文渊博; 高涛; 陈婷; 张千禧

doi:10.12263/DZXB.20221420

您当前的位置：

首页 >

文章列表页 >

频率引导的双稀疏自注意力单图像去雨算法

学术论文 | 更新时间：2025-12-08

- 频率引导的双稀疏自注意力单图像去雨算法
- Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining
- 电子学报 2023年51卷第10期页码：2812-2820
- 作者机构：
  
  长安大学信息工程学院，陕西西安 710064
- 作者简介：
  
  [ "文渊博男，1999年2月出生于陕西省咸阳市.现为长安大学在读博士研究生，主要从事计算机视觉和智能交通系统方面的研究工作. E-mail: wyb@chd.edu.cn" ]
  [ "高涛（通讯作者）男，1980年9月出生于陕西省富平县.2010年博士毕业于西北工业大学电子信息学院.现为长安大学教授，从事人工智能和图像处理方面的研究工作." ]
  [ "陈婷女，1982年4月出生于陕西省西安市.2011年博士毕业于西安电子科技大学通信工程学院.现为长安大学信息工程学院教授，主要从事模式识别和智能交通系统方面的研究工作. E-mail: tchenchd@126.com" ]
  [ "张千禧男，2000年5月6日出生于河南省濮阳市.现为长安大学在读硕士研究生，主要从事计算机视觉方面的研究工作. E-mail: zhangqx1216@gmail.com" ]
- 基金信息：
  
  国家自然科学基金(52172379;62001058);长安大学中央高校基本科研业务费专项资金资助项目(300102242901)
- DOI：10.12263/DZXB.20221420
  中图分类号： TP391.4
- 收稿：2022-12-21，
  
  修回：2023-03-14，
  
  纸质出版：2023-10-25
- 稿件说明：
移动端阅览
文渊博,高涛,陈婷等.频率引导的双稀疏自注意力单图像去雨算法[J].电子学报,2023,51(10):2812-2820.

WEN Yuan-bo,GAO Tao,CHEN Ting,et al.Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining[J].ACTA ELECTRONICA SINICA,2023,51(10):2812-2820.
文渊博,高涛,陈婷等.频率引导的双稀疏自注意力单图像去雨算法[J].电子学报,2023,51(10):2812-2820. DOI： 10.12263/DZXB.20221420.

WEN Yuan-bo,GAO Tao,CHEN Ting,et al.Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining[J].ACTA ELECTRONICA SINICA,2023,51(10):2812-2820. DOI： 10.12263/DZXB.20221420.

摘要

现有基于自注意力网络Transformer的单图像去雨算法尽管在合成雨图上在取得良好效果，但却造成巨大的计算负担，且无法有效处理真实雨图.对此，本文提出一种频率引导的双稀疏自注意力单图像去雨算法（Frequency-guided Dual Sparse self-Attention TransFormer，FDSATFormer）.首先，该算法利用空间稀疏因子和通道降维因子在提取准确全局信息的同时减少计算量，进而提出双稀疏自注意力特征学习网络（Dual Sparse self-attention Feature Leraning， DSFL）以解决Transformer在高分辨率雨图上难以表征自注意力的问题.其次，该算法通过探究图像去雨前后的频谱变化，提出频率引导的特征增强模块（Frequency-guided Feature Enhancer，FFE），其利用频域的全局信息指导特征编码阶段对空域特征的学习.此外，现有去雨网络的编解码结构采用相近的设计，这导致网络的整体计算负担倍增.对此，本文提出层级特征解码重建网络（Hierarchical Feature Decoding and Reconstructing network，HFDR），其利用无参的空间邻域移位操作（Spatial Neighborhood Shift，SNS）构建特征解码网络，在取得良好效果的同时进一步减少整体的计算负担.实验表明，相比表现优秀的Uformer和Restormer，本文算法所得结果的平均峰值信噪比分别提升3.13 dB和0.12 dB，平均结构相似性分别提升1.39%和1.07%.

Abstract

Existing Transformer-based algorithms for single image deraining achieve state-of-the-art performance but leading to reasonable computational loads while failing to effectively process real-world rainy images. To this end

we propose a frequency-guided dual sparse self-attention Transformer for single image deraining (FDSATFormer). Initially

our proposed method utilizes the spatial sparse factor and the channel reduction factor to extract accurate global information and significantly decreases the amount of computation. Furthermore

we present dual sparse self-attention feature learning network (DSFL) to deal with the problem that Transformer is difficult to represent self-attention on high-resolution feature maps. Meanwhile

by exploring the spectral changes of rainy image before and after removing rain streaks

we develop a frequency-guided feature enhancement module (FFE)

which exploits the global information from the frequency domain to guide the accurate learning of spatial features in network encoders. In addition

the encoder and decoder of most existing methods follow the similar principles

resulting in almost double computational burden. To handle with this issue

we propose a hierarchical feature decoding and reconstructing network (HFDR)

which uses non-parametric spatial neighborhood shift (SNS) to construct the feature decoding network and achieves fine results while further reducing the overall computing burden. Experimental results show that

our method improves the average peak signal noise ratio by 3.13 dB and 0.12 dB

and achieves performance gains of 1.39% and 1.07% in average structure similarity over the state-of-the-art Uformer and Restormer.

关键词

Keywords

references

盖杉 , 王俊生 . 基于深度学习的非局部注意力增强网络图像去雨算法研究 [J]. 电子学报 , 2020 , 48 ( 10 ): 1899 - 1908 .

GAI S , WANG J S . Image raindrop algorithm research using nonlocal attention enhanced network based on deep learning [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1899 - 1908 . (in Chinese)

FU X Y , HUANG J B , DING X H , et al . Clearing the skies: A deep network architecture for single-image rain removal [J]. IEEE Transactions on Image Processing , 2017 , 26 ( 6 ): 2944 - 2956 .

秦菁 , 文渊博 , 高涛 , 等 . 面向多天气退化图像恢复的自注意力扩散模型 [J/OL]. 上海交通大学学报 , 2023 . https://doi.org/10.16183/j.cnki.jsjtu.2023.043 https://doi.org/10.16183/j.cnki.jsjtu.2023.043 .

QIN J , WEN Y B , GAO T , et al . Transformer-based diffusion model for all-in-one weather-degraded image restoration [J/OL]. Journal of Shanghai Jiaotong University , 2023 . https://doi.org/10.16183/j.cnki.jsjtu.2023.043. https://doi.org/10.16183/j.cnki.jsjtu.2023.043. (in Chinese)

KIM J H , LEE C , SIM J Y , et al . Single-image deraining using an adaptive nonlocal means filter [C]// 2013 IEEE International Conference on Image Processing . Piscataway : IEEE , 2014 : 914 - 917 .

XU J , ZHAO W , LIU P , et al . Removing rain and snow in a single image using guided filter [C]// 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE) . Piscataway : IEEE , 2012 : 304 - 307 .

ZHENG X H , LIAO Y H , GUO W , et al . Single-image-based rain and snow removal using multi-guided filter [C]// Neural Information Processing: 20th International Conference , ICONIP 2013 . Cham : Springer , 2013: 258 - 265 .

WEI W , MENG D Y , ZHAO Q A , et al . Semi-supervised transfer learning for image rain removal [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 3872 - 3881 .

ZHANG H , PATEL V M . Density-aware single image de-raining using a multi-stream dense network [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2018 : 695 - 704 .

YASARLA R , PATEL V M . Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 8397 - 8406 .

LI X A , WU J L , LIN Z C , et al . Recurrent squeeze-and-excitation context aggregation net for single image deraining [C]// Computer Vision ‒ ECCV 2018 . Cham : Springer International Publishing , 2018 : 262 - 277 .

REN D , ZUO W , HU Q , et al . Progressive image deraining networks: A better and simpler baseline [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 3932 - 3941 .

JIANG K , WANG Z Y , YI P , et al . Multi-scale progressive fusion network for single image deraining [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 8343 - 8352 .

ZAMIR S W , ARORA A , KHAN S , et al . Multi-stage progressive image restoration [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 14816 - 14826 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J]. Advances in Neural Information Processing Systems , 2017 , 30 : 1 - 11 .

LIU Z , LIN Y T , CAO Y E , et al . Swin Transformer: Hierarchical vision Transformer using shifted windows [C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2022 : 9992 - 10002 .

高涛 , 文渊博 , 陈婷 , 等 . 基于窗口自注意力网络的单图像去雨算法 [J]. 上海交通大学学报 , 2023 , 57 ( 5 ): 613 - 623 .

GAO T , WEN Y B , CHEN T , et al . A single image deraining algorithm based on swin Transformer [J]. Journal of Shanghai Jiao Tong University , 2023 , 57 ( 5 ): 613 - 623 . (in Chinese)

WANG Z D , CUN X D , BAO J M , et al . Uformer: A general U-shaped Transformer for image restoration [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 17662 - 17672 .

ZAMIR S W , ARORA A , KHAN S , et al . Restormer: Efficient Transformer for high-resolution image restoration [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 5718 - 5729 .

ZHANG Y , XIANG Y J , CAI L , et al . Single image de-raining with high-low frequency guidance [C]// ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Piscataway : IEEE , 2022 : 2330 - 2334 .

柳长源 , 王琪 , 毕晓君 . 基于多通道多尺度卷积神经网络的单幅图像去雨方法 [J]. 电子与信息学报 , 2020 , 42 ( 9 ): 2285 - 2292 .

LIU C Y , WANG Q , BI X J . Research on rain removal method for single image based on multi-channel and multi-scale CNN [J]. Journal of Electronics & Information Technology , 2020 , 42 ( 9 ): 2285 - 2292 . (in Chinese)

YANG H , ZHOU D M , CAO J D , et al . DPNet: Detail-preserving image deraining via learning frequency domain knowledge [J]. Digital Signal Processing , 2022 , 130 : 103740 .

HUANG H B , YU A J , CHAI Z H , et al . Selective wavelet attention learning for single image deraining [J]. International Journal of Computer Vision , 2021 , 129 ( 4 ): 1282 - 1300 .

YANG H H , YANG C H H , WANG Y C F . Wavelet channel attention module with a fusion network for single image deraining [C]// 2020 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2020 : 883 - 887 .

YANG W H , LIU J Y , YANG S A , et al . Scale-free single image deraining via visibility-enhanced recurrent wavelet learning [J]. IEEE Transactions on Image Processing , 2019 , 28 ( 6 ): 2948 - 2961 .

WU X A , HUANG T Z , DENG L J , et al . A decoder-free Transformer-like architecture for high-efficiency single image deraining [C]// Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2022 : 1474 - 1480 .

CHEN P X , ZHANG M D , SHEN Y H , et al . Efficient decoder-free object detection with Transformers [C]// Lecture Notes in Computer Science . Cham : Springer , 2022 : 70 - 86 .

JOSE VALANARASU J M , YASARLA R , PATEL V M . TransWeather: Transformer-based restoration of images degraded by adverse weather conditions [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 2343 - 2353 .

LIU Z , MAO H Z , WU C Y , et al . A ConvNet for the 2020 s[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 11966 - 11976 .

YANG W H , TAN R T , FENG J S , et al . Deep joint rain detection and removal from a single image [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2017 : 1685 - 1694 .

ZHANG H , SINDAGI V , PATEL V M . Image de-raining using a conditional generative adversarial network [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 11 ): 3943 - 3956 .

LI Y , TAN R T , GUO X J , et al . Rain streak removal using layer priors [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 2736 - 2744 .

FU X Y , LIANG B R , HUANG Y , et al . Lightweight pyramid networks for image deraining [J]. IEEE Transactions on Neural Networks and Learning Systems , 2020 , 31 ( 6 ): 1794 - 1807 .

KINGMA D P , BA J . Adam: A method for stochastic optimization [EB/OL]. ( 2017-01-30 )[ 2023-09-13 ]. https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980 .

ZHANG H , CISSE M , DAUPHIN Y N , et al . MixUp: Beyond empirical risk minimization [EB/OL]. ( 2018-04-27 )[ 2022-12-19 ]. https://arxiv.org/abs/1710.09412 https://arxiv.org/abs/1710.09412 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于特征自适应选取的视觉目标跟踪算法

连续学习方法与其在视觉任务中的应用

基于因果提示蒸馏的开放世界目标检测

基于神经网络的图像风格迁移算法综述