Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining

WEN Yuan-bo; GAO Tao; CHEN Ting; ZHANG Qian-xi

doi:10.12263/DZXB.20221420

您当前的位置：

首页 >

文章列表页 >

Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining

PAPERS | 更新时间：2025-12-08

- Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining
- ACTA ELECTRONICA SINICA Vol. 51, Issue 10, Pages: 2812-2820(2023)
- 作者机构：
  
  长安大学信息工程学院，陕西西安 710064
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(52172379;62001058);Fundamental Research Funds for the Central Universities(300102242901)
- DOI：10.12263/DZXB.20221420
  CLC： TP391.4
- Received：21 December 2022，
  
  Revised：2023-03-14，
  
  Published：25 October 2023
- 稿件说明：
移动端阅览
文渊博,高涛,陈婷等.频率引导的双稀疏自注意力单图像去雨算法[J].电子学报,2023,51(10):2812-2820.

WEN Yuan-bo,GAO Tao,CHEN Ting,et al.Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining[J].ACTA ELECTRONICA SINICA,2023,51(10):2812-2820.
文渊博,高涛,陈婷等.频率引导的双稀疏自注意力单图像去雨算法[J].电子学报,2023,51(10):2812-2820. DOI： 10.12263/DZXB.20221420.

WEN Yuan-bo,GAO Tao,CHEN Ting,et al.Frequency-guided Dual Sparse Self-Attention Algorithm for Single Image Deraining[J].ACTA ELECTRONICA SINICA,2023,51(10):2812-2820. DOI： 10.12263/DZXB.20221420.

摘要

现有基于自注意力网络Transformer的单图像去雨算法尽管在合成雨图上在取得良好效果，但却造成巨大的计算负担，且无法有效处理真实雨图.对此，本文提出一种频率引导的双稀疏自注意力单图像去雨算法（Frequency-guided Dual Sparse self-Attention TransFormer，FDSATFormer）.首先，该算法利用空间稀疏因子和通道降维因子在提取准确全局信息的同时减少计算量，进而提出双稀疏自注意力特征学习网络（Dual Sparse self-attention Feature Leraning， DSFL）以解决Transformer在高分辨率雨图上难以表征自注意力的问题.其次，该算法通过探究图像去雨前后的频谱变化，提出频率引导的特征增强模块（Frequency-guided Feature Enhancer，FFE），其利用频域的全局信息指导特征编码阶段对空域特征的学习.此外，现有去雨网络的编解码结构采用相近的设计，这导致网络的整体计算负担倍增.对此，本文提出层级特征解码重建网络（Hierarchical Feature Decoding and Reconstructing network，HFDR），其利用无参的空间邻域移位操作（Spatial Neighborhood Shift，SNS）构建特征解码网络，在取得良好效果的同时进一步减少整体的计算负担.实验表明，相比表现优秀的Uformer和Restormer，本文算法所得结果的平均峰值信噪比分别提升3.13 dB和0.12 dB，平均结构相似性分别提升1.39%和1.07%.

Abstract

Existing Transformer-based algorithms for single image deraining achieve state-of-the-art performance but leading to reasonable computational loads while failing to effectively process real-world rainy images. To this end

we propose a frequency-guided dual sparse self-attention Transformer for single image deraining (FDSATFormer). Initially

our proposed method utilizes the spatial sparse factor and the channel reduction factor to extract accurate global information and significantly decreases the amount of computation. Furthermore

we present dual sparse self-attention feature learning network (DSFL) to deal with the problem that Transformer is difficult to represent self-attention on high-resolution feature maps. Meanwhile

by exploring the spectral changes of rainy image before and after removing rain streaks

we develop a frequency-guided feature enhancement module (FFE)

which exploits the global information from the frequency domain to guide the accurate learning of spatial features in network encoders. In addition

the encoder and decoder of most existing methods follow the similar principles

resulting in almost double computational burden. To handle with this issue

we propose a hierarchical feature decoding and reconstructing network (HFDR)

which uses non-parametric spatial neighborhood shift (SNS) to construct the feature decoding network and achieves fine results while further reducing the overall computing burden. Experimental results show that

our method improves the average peak signal noise ratio by 3.13 dB and 0.12 dB

and achieves performance gains of 1.39% and 1.07% in average structure similarity over the state-of-the-art Uformer and Restormer.

关键词

Keywords

references

盖杉 , 王俊生 . 基于深度学习的非局部注意力增强网络图像去雨算法研究 [J]. 电子学报 , 2020 , 48 ( 10 ): 1899 - 1908 .

GAI S , WANG J S . Image raindrop algorithm research using nonlocal attention enhanced network based on deep learning [J]. Acta Electronica Sinica , 2020 , 48 ( 10 ): 1899 - 1908 . (in Chinese)

FU X Y , HUANG J B , DING X H , et al . Clearing the skies: A deep network architecture for single-image rain removal [J]. IEEE Transactions on Image Processing , 2017 , 26 ( 6 ): 2944 - 2956 .

秦菁 , 文渊博 , 高涛 , 等 . 面向多天气退化图像恢复的自注意力扩散模型 [J/OL]. 上海交通大学学报 , 2023 . https://doi.org/10.16183/j.cnki.jsjtu.2023.043 https://doi.org/10.16183/j.cnki.jsjtu.2023.043 .

QIN J , WEN Y B , GAO T , et al . Transformer-based diffusion model for all-in-one weather-degraded image restoration [J/OL]. Journal of Shanghai Jiaotong University , 2023 . https://doi.org/10.16183/j.cnki.jsjtu.2023.043. https://doi.org/10.16183/j.cnki.jsjtu.2023.043. (in Chinese)

KIM J H , LEE C , SIM J Y , et al . Single-image deraining using an adaptive nonlocal means filter [C]// 2013 IEEE International Conference on Image Processing . Piscataway : IEEE , 2014 : 914 - 917 .

XU J , ZHAO W , LIU P , et al . Removing rain and snow in a single image using guided filter [C]// 2012 IEEE International Conference on Computer Science and Automation Engineering (CSAE) . Piscataway : IEEE , 2012 : 304 - 307 .

ZHENG X H , LIAO Y H , GUO W , et al . Single-image-based rain and snow removal using multi-guided filter [C]// Neural Information Processing: 20th International Conference , ICONIP 2013 . Cham : Springer , 2013: 258 - 265 .

WEI W , MENG D Y , ZHAO Q A , et al . Semi-supervised transfer learning for image rain removal [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 3872 - 3881 .

ZHANG H , PATEL V M . Density-aware single image de-raining using a multi-stream dense network [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2018 : 695 - 704 .

YASARLA R , PATEL V M . Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 8397 - 8406 .

LI X A , WU J L , LIN Z C , et al . Recurrent squeeze-and-excitation context aggregation net for single image deraining [C]// Computer Vision ‒ ECCV 2018 . Cham : Springer International Publishing , 2018 : 262 - 277 .

REN D , ZUO W , HU Q , et al . Progressive image deraining networks: A better and simpler baseline [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 3932 - 3941 .

JIANG K , WANG Z Y , YI P , et al . Multi-scale progressive fusion network for single image deraining [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 8343 - 8352 .

ZAMIR S W , ARORA A , KHAN S , et al . Multi-stage progressive image restoration [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 14816 - 14826 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J]. Advances in Neural Information Processing Systems , 2017 , 30 : 1 - 11 .

LIU Z , LIN Y T , CAO Y E , et al . Swin Transformer: Hierarchical vision Transformer using shifted windows [C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2022 : 9992 - 10002 .

高涛 , 文渊博 , 陈婷 , 等 . 基于窗口自注意力网络的单图像去雨算法 [J]. 上海交通大学学报 , 2023 , 57 ( 5 ): 613 - 623 .

GAO T , WEN Y B , CHEN T , et al . A single image deraining algorithm based on swin Transformer [J]. Journal of Shanghai Jiao Tong University , 2023 , 57 ( 5 ): 613 - 623 . (in Chinese)

WANG Z D , CUN X D , BAO J M , et al . Uformer: A general U-shaped Transformer for image restoration [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 17662 - 17672 .

ZAMIR S W , ARORA A , KHAN S , et al . Restormer: Efficient Transformer for high-resolution image restoration [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 5718 - 5729 .

ZHANG Y , XIANG Y J , CAI L , et al . Single image de-raining with high-low frequency guidance [C]// ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Piscataway : IEEE , 2022 : 2330 - 2334 .

柳长源 , 王琪 , 毕晓君 . 基于多通道多尺度卷积神经网络的单幅图像去雨方法 [J]. 电子与信息学报 , 2020 , 42 ( 9 ): 2285 - 2292 .

LIU C Y , WANG Q , BI X J . Research on rain removal method for single image based on multi-channel and multi-scale CNN [J]. Journal of Electronics & Information Technology , 2020 , 42 ( 9 ): 2285 - 2292 . (in Chinese)

YANG H , ZHOU D M , CAO J D , et al . DPNet: Detail-preserving image deraining via learning frequency domain knowledge [J]. Digital Signal Processing , 2022 , 130 : 103740 .

HUANG H B , YU A J , CHAI Z H , et al . Selective wavelet attention learning for single image deraining [J]. International Journal of Computer Vision , 2021 , 129 ( 4 ): 1282 - 1300 .

YANG H H , YANG C H H , WANG Y C F . Wavelet channel attention module with a fusion network for single image deraining [C]// 2020 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2020 : 883 - 887 .

YANG W H , LIU J Y , YANG S A , et al . Scale-free single image deraining via visibility-enhanced recurrent wavelet learning [J]. IEEE Transactions on Image Processing , 2019 , 28 ( 6 ): 2948 - 2961 .

WU X A , HUANG T Z , DENG L J , et al . A decoder-free Transformer-like architecture for high-efficiency single image deraining [C]// Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization , 2022 : 1474 - 1480 .

CHEN P X , ZHANG M D , SHEN Y H , et al . Efficient decoder-free object detection with Transformers [C]// Lecture Notes in Computer Science . Cham : Springer , 2022 : 70 - 86 .

JOSE VALANARASU J M , YASARLA R , PATEL V M . TransWeather: Transformer-based restoration of images degraded by adverse weather conditions [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 2343 - 2353 .

LIU Z , MAO H Z , WU C Y , et al . A ConvNet for the 2020 s[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2022 : 11966 - 11976 .

YANG W H , TAN R T , FENG J S , et al . Deep joint rain detection and removal from a single image [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2017 : 1685 - 1694 .

ZHANG H , SINDAGI V , PATEL V M . Image de-raining using a conditional generative adversarial network [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 11 ): 3943 - 3956 .

LI Y , TAN R T , GUO X J , et al . Rain streak removal using layer priors [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 2736 - 2744 .

FU X Y , LIANG B R , HUANG Y , et al . Lightweight pyramid networks for image deraining [J]. IEEE Transactions on Neural Networks and Learning Systems , 2020 , 31 ( 6 ): 1794 - 1807 .

KINGMA D P , BA J . Adam: A method for stochastic optimization [EB/OL]. ( 2017-01-30 )[ 2023-09-13 ]. https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980 .

ZHANG H , CISSE M , DAUPHIN Y N , et al . MixUp: Beyond empirical risk minimization [EB/OL]. ( 2018-04-27 )[ 2022-12-19 ]. https://arxiv.org/abs/1710.09412 https://arxiv.org/abs/1710.09412 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Visual Object Tracking Algorithm Based on Adaptive Feature Selection

Continual Learning Methods and Applications in Computer Vision

Open World Object Detection Based on Causal Prompt Distillation

Neural Network Based Image Style Transfer: A Survey

Related Author

GAO Tao

WANG Cai-xia

AN Qi

ZHOU Hong-ce

LI Yi-peng

FANG Yan

Wei Yun-chao

Cong Run-min

Related Institution

Jilin Province Technology Research Center of Photoelectric Detection and Intelligent Information Processing

School of Electronic Information and Engineering, Changchun University of Science and Technology

School of Computer Science and Technology, Harbin Institute of Technology

School of Control Science and Engineering, Shandong University

School of Computer Science and Technology, Beijing Jiaotong University

⁰