

浏览全部资源
扫码关注微信
南京大学计算机学院,江苏南京 210023
Received:08 January 2025,
Revised:2025-07-15,
Published:25 July 2025
移动端阅览
沈伟露, 刘杰, 唐杰, 等. 基于多尺度特征自适应调制的单图像超分辨率网络[J]. 电子学报, 2025, 53(07): 2324-2341.
SHEN Wei-lu, LIU Jie, TANG Jie, et al. Multi-Scale Feature Adaptive Modulation for Single Image Super-Resolution[J]. Acta Electronica Sinica, 2025, 53(07): 2324-2341.
沈伟露, 刘杰, 唐杰, 等. 基于多尺度特征自适应调制的单图像超分辨率网络[J]. 电子学报, 2025, 53(07): 2324-2341. DOI:10.12263/DZXB.20250032
SHEN Wei-lu, LIU Jie, TANG Jie, et al. Multi-Scale Feature Adaptive Modulation for Single Image Super-Resolution[J]. Acta Electronica Sinica, 2025, 53(07): 2324-2341. DOI:10.12263/DZXB.20250032
基于Transformer的图像恢复方法在单图像超分辨率(Single Image Super-Resolution,SISR)任务中展现了卓越的性能,这得益于其自注意力(Self-Attention,SA)机制能够有效捕捉非局部信息,从而实现更高质量的高分辨率(High Resolution,HR)图像重建.然而,SA机制中的矩阵乘法操作需要消耗大量计算资源,这使得基于Transformer的模型通常难以适配计算能力和内存受限的低功耗设备.此外,SA机制的低通特性限制了其捕获高频局部细节的能力,从而导致平滑的重建结果.为了解决以上问题,本文提出了一种基于多尺度特征自适应调制的单图像超分辨率网络(Multi-scale Feature Adaptive Modulation Network,MFAMNet),其核心是多尺度特征自适应调制(Multi-scale Feature Adaptive Modulation,MFAM)模块,该模块通过下采样操作获取不同尺度的低频内容,计算输入特征的全局方差来调制处理后的低频特征,然后使用调制后的特征自适应地聚合输入特征,从而实现对非局部信息的高效建模.在聚合输入特征之后,本文引入通道注意力机制,从通道维度对融合特征进行细化处理,以增强所有通道间公共信息的提取能力,同时实现跨通道权重的动态重分配.此外,由于MFAM从远程角度处理输入特征,因此需要补充局部上下文信息.为此,本文还设计了空间增强模块(Spatial Enhancement Module,SEM),作为复杂自注意机制的有效替代方案,以显著提高空间局部聚合能力,在空间和通道维度上进一步细化从MFAM输出的特征.大量实验表明:所提出的MFAMNet,在公共基准数据集上实现了重建性能和计算效率之间的更佳权衡.特别是在4倍超分辨率(Super-Resolution,SR)下,与最新的自调制特征聚合网络(Self-Modulation Feature Aggregation Network,SMFANet)相比,MFAMNet在五个公共测试集上的平均性能提高了0.15 dB,而模型复杂性,如每秒浮点运算次数(FLoating-point Operations Per second,FLOPs),与前者几乎相同.
Transformer-based image restoration methods have demonstrated remarkable performance in single image super-resolution tasks
owing to their self-attention (SA) mechanism
which effectively captures non-local information
thereby achieving higher-quality high-resolution image reconstruction. However
the matrix multiplication operations in the self-attention mechanism consume substantial computational resources
making Transformer-based models generally challenging to deploy on low-power devices with limited computational capabilities and memory. Additionally
the low-pass characteristics of the SA mechanism restrict its ability to capture high-frequency local details
leading to overly smooth reconstruction results. To address these issues
we propose a multi-scale feature adaptive modulation network (MFAMNet) for single image super-resolution
whose core is the multi-scale feature adaptive modulation (MFAM) module. This module obtains low-frequency content at different scales through downsampling operations
computes the global variance of the input features to modulate the processed low-frequency features
and then adaptively aggregates the input features using the modulated features
thereby achieving efficient modeling of non-local information. After aggregating the input features
we introduce a channel attention mechanism to refine the fused features from the channel dimension
enhancing the extraction of shared information across all channels while dynamically reallocating cross-channel weights. Furthermore
since MFAM processes input features from a long-range perspective
it is necessary to supplement local contextual information. To this end
we also design a spatial enhancement module (SEM) as an effective alternative to complex self-attention mechanisms
significantly improving spatial local aggregation capabilities and further refining the features output from MFAM in both spatial and channel dimensions. Extensive experiments demonstrate that the proposed MFAMNet achieves a better trade-off between reconstruction performance and computational efficiency on public benchmark datasets. Notably
in 4× super-resolution
self-modulation feature aggregation network (MFAMNet) improves the average performance by 0.15 dB compared to the state-of-the-art self-modulation feature aggregation network (SMFANet) on five public test sets
while maintaining nearly the same model complexity
e.g.
floating-point operations per second (FLOPs).
KEYS R . Cubic convolution interpolation for digital image processing [J ] . IEEE Transactions on Acoustics, Speech, and Signal Processing , 1981 , 29 ( 6 ): 1153 - 1160 .
YANG J C , WRIGHT J , HUANG T S , et al . Image super-resolution via sparse representation [J ] . IEEE Transactions on Image Processing , 2010 , 19 ( 11 ): 2861 - 2873 .
YANG J C , WRIGHT J , HUANG T , et al . Image super-resolution as sparse representation of raw image patches [C ] // 2008 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2008 : 1 - 8 .
XU H T , ZHAI G T , YANG X K . Single image super-resolution with detail enhancement based on local fractal analysis of gradient [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2013 , 23 ( 10 ): 1740 - 1754 .
WANG L F , XIANG S M , MENG G F , et al . Edge-directed single-image super-resolution via adaptive gradient magnitude self-interpolation [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2013 , 23 ( 8 ): 1289 - 1299 .
ZHANG K B , GAO X B , TAO D C , et al . Single image super-resolution with non-local means and steering kernel regression [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 11 ): 4544 - 4556 .
FREEMAN W T , JONES T R , PASZTOR E C . Example-based super-resolution [J ] . IEEE Computer Graphics and Applications , 2002 , 22 ( 2 ): 56 - 65 .
CHANG H , YEUNG D Y , XIONG Y M . Super-resolution through neighbor embedding [C ] // Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2004 . CVPR 2004. Piscataway : IEEE , 2004: I .
SCHULTER S , LEISTNER C , BISCHOF H . Fast and accurate image upscaling with super-resolution forests [C ] // 2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2015 : 3791 - 3799 .
YU J F , GAO X B , TAO D C , et al . A unified learning framework for single image super-resolution [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2014 , 25 ( 4 ): 780 - 792 .
DENG C , XU J , ZHANG K B , et al . Similarity constraints-based structured output regression machine: An approach to image super-resolution [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2016 , 27 ( 12 ): 2472 - 2485 .
YANG W M , TIAN Y P , ZHOU F , et al . Consistent coding scheme for single-image super-resolution via independent dictionaries [J ] . IEEE Transactions on Multimedia , 2016 , 18 ( 3 ): 313 - 325 .
AHN N , KANG B , SOHN K A . Fast, accurate, and lightweight super-resolution with cascading residual network [M ] // Computer Vision-ECCV 2018 . Cham : Springer International Publishing , 2018 : 256 - 272 .
DONG C , LOY C C , HE K M , et al . Image super-resolution using deep convolutional networks [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2016 , 38 ( 2 ): 295 - 307 .
DONG C , LOY C C , TANG X O . Accelerating the super-resolution convolutional neural network [M ] // Computer Vision - ECCV 2016 . Cham : Springer International Publishing , 2016 : 391 - 407 .
HUI Z , GAO X B , YANG Y C , et al . Lightweight image super-resolution with information multi-distillation network [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York : ACM , 2019 : 2024 - 2032 .
LI Z Y , LIU Y Q , CHEN X Y , et al . Blueprint separable residual network for efficient image super-resolution [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2022 : 832 - 842 .
LIM B , SON S , KIM H , et al . Enhanced deep residual networks for single image super-resolution [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2017 : 1132 - 1140 .
SHI W Z , CABALLERO J , HUSZÁR F , et al . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 1874 - 1883 .
SUN L , DONG J X , TANG J H , et al . Spatially-adaptive feature modulation for efficient image super-resolution [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2023 : 13144 - 13153 .
SUN L , PAN J , TANG J . Shufflemixer: An efficient convnet for image super-resolution [J ] . Advances in Neural Information Processing Systems , 2022 , 35 : 17314 - 17326 .
ZHANG Y L , LI K P , LI K , et al . Image Super-resolution using Very Deep Residual Channel Attention Networks [M ] // Computer Vision-ECCV 2018 . Cham : Springer International Publishing , 2018 : 294 - 310 .
高丹丹 , 周登文 , 王婉君 , 等 . 特征频率分组融合的轻量级图像超分辨率重建 [J ] . 计算机辅助设计与图形学学报 , 2023 , 35 ( 7 ): 1020 - 1031 .
GAO D D , ZHOU D W , WANG W J , et al . Lightweight super-resolution via grouping fusion of feature frequencies [J ] . Journal of Computer-Aided Design & Computer Graphics , 2023 , 35 ( 7 ): 1020 - 1031 . (in Chinese)
吴靖 , 叶晓晶 , 黄峰 , 等 . 基于深度学习的单帧图像超分辨率重建综述 [J ] . 电子学报 , 2022 , 50 ( 9 ): 2265 - 2294 .
WU J , YE X J , HUANG F , et al . A review of single image super-resolution reconstruction based on deep learning [J ] . Acta Electronica Sinica , 2022 , 50 ( 9 ): 2265 - 2294 . (in Chinese)
缪永伟 , 张新杰 , 任瀚实 , 等 . 基于通道多尺度融合的场景深度图超分辨率网络 [J ] . 计算机辅助设计与图形学学报 , 2023 , 35 ( 1 ): 37 - 47 .
MIAO Y W , ZHANG X J , REN H S , et al . A channel multi-scale fusion network for scene depth map super-resolution [J ] . Journal of Computer-Aided Design & Computer Graphics , 2023 , 35 ( 1 ): 37 - 47 . (in Chinese)
周登文 , 刘子涵 , 刘玉铠 . 基于像素对比学习的图像超分辨率算法 [J ] . 自动化学报 , 2024 , 50 ( 1 ): 181 - 193 .
ZHOU D W , LIU Z H , LIU Y K . Pixel-wise contrastive learning for single image super-resolution [J ] . Acta Automatica Sinica , 2024 , 50 ( 1 ): 181 - 193 . (in Chinese)
王云涛 , 赵蔺 , 刘李漫 , 等 . 基于组-信息蒸馏残差网络的轻量级图像超分辨率重建 [J ] . 自动化学报 , 2024 , 50 ( 10 ): 2063 - 2078 .
WANG Y T , ZHAO L , LIU L M , et al . G-IDRN: A group-information distillation residual network for lightweight image super-resolution [J ] . Acta Automatica Sinica , 2024 , 50 ( 10 ): 2063 - 2078 . (in Chinese)
KONG F Y , LI M X , LIU S W , et al . Residual local feature network for efficient super-resolution [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2022 : 765 - 775 .
LIU J , TANG J , WU G S . Residual feature distillation network for lightweight image super-resolution [M ] // Computer Vision - ECCV 2020 Workshops . Cham : Springer International Publishing , 2020 : 41 - 55 .
MICHELINI P N , LU Y H , JIANG X Q . Edge-SR: Super-resolution for the masses [C ] // 2022 IEEE/CVF Winter Conference on Applications of Computer Vision . Piscataway : IEEE , 2022 : 4019 - 4028 .
ZHAO H Y , KONG X T , HE J W , et al . Efficient Image Super-resolution using Pixel Attention [M ] // Computer Vision - ECCV 2020 Workshops . Cham : Springer International Publishing , 2020 : 56 - 72 .
HE Z B , DAI T , LU J , et al . Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution [C ] // 2020 IEEE International Conference on Image Processing . Piscataway : IEEE , 2020 : 518 - 522 .
CHU X X , ZHANG B , MA H L , et al . Fast, accurate and lightweight super-resolution with neural architecture search [C ] // Proceedings of the 25th International Conference on Pattern Recognition . Piscataway : IEEE , 2021 : 59 - 64 .
SONG D H , XU C , JIA X , et al . Efficient residual dense block search for image super-resolution [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2020 , 34 ( 7 ): 12007 - 12014 .
ZHANG X D , ZENG H , ZHANG L . Edge-oriented convolution block for real-time super resolution on mobile devices [C ] // Proceedings of the 29th ACM International Conference on Multimedia . New York : ACM , 2021 : 4034 - 4043 .
LEDIG C , THEIS L , HUSZÁR F , et al . Photo-realistic single image super-resolution using a generative adversarial network [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2017 : 105 - 114 .
WANG X T , YU K , WU S X , et al . ESRGAN: Enhanced super-resolution generative adversarial networks [M ] // Computer Vision-ECCV 2018 Workshops . Cham : Springer International Publishing , 2019 : 63 - 79 .
SAHARIA C , HO J , CHAN W , et al . Image super-resolution via iterative refinement [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 4 ): 4713 - 4726 .
RAMESH A , PAVLOV M , GOH G , et al . Zero-shot text-to-image generation [C ] // International Conference on Machine Learning . Cambridge : PMLR , 2021 : 8821 - 8831 .
ROMBACH R , BLATTMANN A , LORENZ D , et al . High-resolution image synthesis with latent diffusion models [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 10674 - 10685 .
DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16 x 16 words: Transformers for image recognition at scale[EB/OL ] . ( 2021-06-03 )[ 2024-10-31 ] . https://arXiv.org/abs/2010.11929 https://arXiv.org/abs/2010.11929 .
CHEN H T , WANG Y H , GUO T Y , et al . Pre-trained image processing transformer [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 12294 - 12305 .
CHOI H , LEE J , YANG J . N-gram in swin transformers for efficient lightweight image super-resolution [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 2071 - 2081 .
LI M , MA B , ZHANG Y L . Lightweight image super-resolution with pyramid clustering transformer [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2023 , PP(99): 1.
LIANG J Y , CAO J Z , SUN G L , et al . SwinIR: Image restoration using swin transformer [C ] // 2021 IEEE/CVF International Conference on Computer Vision Workshops . Piscataway : IEEE , 2021 : 1833 - 1844 .
LIU J , CHEN C , TANG J , et al . From coarse to fine: Hierarchical pixel integration for lightweight image super-resolution [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2023 , 37 ( 2 ): 1666 - 1674 .
WANG H , CHEN X H , NI B B , et al . Omni aggregation networks for lightweight image super-resolution [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 22378 - 22387 .
ZHANG A P , REN W Q , LIU Y , et al . Lightweight image super-resolution with superpixel token interaction [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 12682 - 12691 .
ZHANG X D , ZENG H , GUO S , et al . Efficient long-range attention network for image super-resolution [M ] // Computer Vision - ECCV 2022 . Cham : Springer Nature Switzerland , 2022 : 649 - 667 .
ZHOU Y P , LI Z , GUO C L , et al . SRFormer: Permuted self-attention for single image super-resolution [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 12734 - 12745 .
唐述 , 曾琬凌 , 杨书丽 , 等 . 基于Transformer的块内块间双聚合的单图像超分辨率重建网络 [J ] . 计算机学报 , 2024 , 47 ( 12 ): 2783 - 2802 .
TANG S , ZENG W L , YANG S L , et al . Intra-block and inter-block dual aggregation transformer for single image super-resolution [J ] . Chinese Journal of Computers , 2024 , 47 ( 12 ): 2783 - 2802 . (in Chinese)
毕修平 , 陈实 , 张乐飞 . 轻量级图像超分辨率的蓝图可分离卷积Transformer网络 [J ] . 中国图象图形学报 , 2024 , 29 ( 4 ): 875 - 889 .
BI X P , CHEN S , ZHANG L F . Blueprint separable convolution Transformer network for lightweight image super-resolution [J ] . Journal of Image and Graphics , 2024 , 29 ( 4 ): 875 - 889 . (in Chinese)
LIU Z , LIN Y T , CAO Y , et al . Swin transformer: Hierarchical vision transformer using shifted windows [C ] // 2021 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2022 : 9992 - 10002 .
ZAMIR S W , ARORA A , KHAN S , et al . Restormer: Efficient transformer for high-resolution image restoration [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 5718 - 5729 .
ZHANG J N , PENG H W , WU K , et al . MiniViT: Compressing vision transformers with weight multiplexing [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 12135 - 12144 .
DONG J X , PAN J S , YANG Z B , et al . Multi-scale residual low-pass filter network for image deblurring [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 12311 - 12320 .
PARK N , KIM S . How do vision transformers work? [EB/OL ] . ( 2022-06-08 )[ 2024-10-31 ] . https://arXiv.org/abs/2202.06709 https://arXiv.org/abs/2202.06709 .
LI W , ZHOU K , QI L , et al . Lapar: Linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 20343 - 20355 .
WANG L G , DONG X Y , WANG Y Q , et al . Exploring sparsity in image super-resolution for efficient inference [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 4915 - 4924 .
ZHENG M J , SUN L , DONG J X , et al . SMFANet: A lightweight self-modulation feature aggregation network for efficient image super-resolution [M ] // Computer Vision-ECCV 2024 . Cham : Springer Nature Switzerland , 2024 : 359 - 375 .
KIM J , LEE J K , LEE K M . Accurate image super-resolution using very deep convolutional networks [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2016 : 1646 - 1654 .
TAI Y , YANG J , LIU X M . Image super-resolution via deep recursive residual network [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2017 : 2790 - 2798 .
LU Z S , LI J C , LIU H , et al . Transformer for single image super-resolution [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2022 : 456 - 465 .
WANG H , CHEN X H , NI B B , et al . Omni aggregation networks for lightweight image super-resolution [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 22378 - 22387 .
CHEN X Y , WANG X T , ZHOU J T , et al . Activating more pixels in image super-resolution transformer [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 22367 - 22377 .
LI A , ZHANG L , LIU Y , et al . Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution [C ] // 2023 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2024 : 12480 - 12490 .
LI Y W , FAN Y C , XIANG X Y , et al . Efficient and explicit modelling of image hierarchies for image restoration [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 18278 - 18289 .
ZHOU L , CAI H M , GU J J , et al . Efficient image super-resolution using vast-receptive-field attention [M ] // Computer Vision - ECCV 2022 Workshops . Cham : Springer Nature Switzerland , 2023 : 256 - 272 .
MAO Y Y , ZHANG N H , WANG Q , et al . Multi-level dispersion residual network for efficient image super-resolution [C ] // 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway : IEEE , 2023 : 1660 - 1669 .
MIAO X , LI S J , LI Z , et al . Multi-scale gated network for efficient image super-resolution [J ] . The Visual Computer , 2025 , 41 ( 2 ): 1227 - 1239 .
ZAMFIR E , WU Z , MEHTA N , et al . See more details: Efficient image super-resolution by experts mining [C ] // Proceedings of the 41st International Conference on Machine Learning . Cambridge : PMLR , 2024 : 1 - 16 .
ZHAO X L , LI L Z , XIE C X , et al . Efficient single image super-resolution with entropy attention and receptive field augmentation [C ] // Proceedings of the 32nd ACM International Conference on Multimedia . New York : ACM , 2024 : 1302 - 1310 .
LI F , CONG R M , WU J J , et al . SRConvNet: A transformer-style ConvNet for lightweight image super-resolution [J ] . International Journal of Computer Vision , 2025 , 133 ( 1 ): 173 - 189 .
CHANG K R , SUN J , YANG B , et al . A multi-scale enhanced large-kernel attention transformer network for lightweight image super-resolution [J ] . Signal, Image and Video Processing , 2025 , 19 ( 3 ): 204 .
YANG X , HONG C M , ZHANG P P . GRFN: A group residual feature network for lightweight image super-resolution [J ] . Circuits, Systems, and Signal Processing , 2025 , 44 ( 5 ): 3513 - 3533 .
GU J J , DONG C . Interpreting super-resolution networks with local attribution maps [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2021 : 9195 - 9204 .
VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017) . Long Beach : Curran Associates, Inc. , 2017 : 5998 - 6008 .
0
Views
18
下载量
0
CSCD
Publicity Resources
Related Articles
Related Author
Related Institution
京公网安备11010802024621