MW-DAN: Multilevel Wavelet-Deep Aggregation Network for Hyperspectral Image Super-Resolution

FANG Jian; YANG Jing-xiang; XIAO Liang

doi:10.12263/DZXB.20220800

您当前的位置：

首页 >

文章列表页 >

MW-DAN: Multilevel Wavelet-Deep Aggregation Network for Hyperspectral Image Super-Resolution

PAPERS | 更新时间：2026-04-10

- MW-DAN: Multilevel Wavelet-Deep Aggregation Network for Hyperspectral Image Super-Resolution
- ACTA ELECTRONICA SINICA Vol. 52, Issue 1, Pages: 201-216(2024)
- 作者机构：
  
  1.南京理工大学计算机科学与工程学院，江苏南京 210094
  2.江苏省光谱成像与智能感知重点实验室，江苏南京 210094
- 作者简介：
- 基金信息：
  
  Special Fund for Basic Research Business Expenses of Central Universities(30920021134)
- DOI：10.12263/DZXB.20220800
  CLC： TP751;
- Received：08 July 2022，
  
  Revised：2022-09-20，
  
  Published：25 January 2024
- 稿件说明：
移动端阅览
方健,杨劲翔,肖亮.基于多层小波深度聚合网络的高光谱图像超分辨率方法[J].电子学报,2024,52(01):201-216.

FANG Jian, YANG Jing-xiang, XIAO Liang.MW-DAN: Multilevel Wavelet-Deep Aggregation Network for Hyperspectral Image Super-Resolution[J].Acta Electronica Sinica, 2024, 52(01): 201-216.
方健,杨劲翔,肖亮.基于多层小波深度聚合网络的高光谱图像超分辨率方法[J].电子学报,2024,52(01):201-216. DOI：10.12263/DZXB.20220800

FANG Jian, YANG Jing-xiang, XIAO Liang.MW-DAN: Multilevel Wavelet-Deep Aggregation Network for Hyperspectral Image Super-Resolution[J].Acta Electronica Sinica, 2024, 52(01): 201-216. DOI：10.12263/DZXB.20220800

摘要

利用低空间分辨率高光谱（Low Resolution HyperSpectral Image，LR-HSI）和高空间分辨率多光谱图像（High Resolution MultiSpectral Image，HR-MSI）的有机结合，实现高光谱空间分辨率增强，是当前高光谱图像处理的热点问题.目前，深度学习已成为高光谱-多光谱图像融合超分辨率的代表性方法，然而如何有效挖掘两者的互补空谱信息，实现空间结构和细节注入，在提升高光谱图像空间分辨率的同时保持高保真光谱信息，依然存在诸多挑战.本文提出了一种多层小波深度聚合网络（Multilevel Wavelet-Deep Aggregation Network，MW-DAN）.该网络有机结合非抽取小波（UnDecimated Wavelet Transform，UDWT）分解和深度残差网络，建立双分支互补信息融合网络，提升图像重建性能.其中，通过深度残差网络中引入跳层汇聚连接，设计信息聚合型结构，并对多光谱图像进行UDWT方向子带分解，逐层注入到网络中间隐层，增强了方向子带结构的细节注入和光谱保真能力.整个网络通过LR-HSI，HR-MSI和HR-HSI（High Resolution HyperSpectral Image）端对端训练，能够学习性能优越的空-谱融合的超分辨非线性映射.大量仿真数据集和真实数据集上的大量融合实验表明，本文提出的方法在客观评价指标、光谱保持和视觉效果上优于目前主流的深度学习方法.

Abstract

The utilization of the synergistic fusion of low resolution hyperspectral image (LR-HSI) and high resolution multispectral image (HR-MSI) for the purpose of achieving enhanced hyperspectral spatial resolution has emerged as a prominent and actively pursued research area within the domain of hyperspectral image processing. At the present time

deep learning has become an efficient tool for HSI-MSI fusion. Despite the potential of deep learning

there are still some challenging

such as how to effectively mine the complementary information of HSI and MSI

how to inject the spatial structure and detail of MSI into HSI

and how to maintain the spectral fidelity of HSI. This study proposes a multilevel wavelet-deep aggregation network (MW-DAN). It has dual branches

which combine undecimated wavelet transform (UDWT) with deep residual network to promote the image reconstruction. Particularly

the UDWT directional subband decomposition of MSIs is performed by introducing jumper aggregation connections in the deep residual network to design an information aggregation type structure

and injected into the middle hidden layer of the network layer by layer to enhance the detail injection and spectral fidelity of the directional subband structure. The entire network are trained from LR-HSI

HR-MSI and HR-HSI in an end-to-end fashion. It could learn the spatial-spectral fusion nonlinear mapping with superior performance. Experimental results on simulation and real datasets show that the proposed method is superior to the state-of-the-art fusion methods in terms of objective evaluation index

spectral fidelity and visual performance.

关键词

Keywords

references

潘宗序 , 禹晶 , 肖创柏 , 等 . 基于光谱相似性的高光谱图像超分辨率算法 [J]. 自动化学报 , 2014 , 40 ( 12 ): 2797 - 2807 .

PAN Z X , YU J , XIAO C B , et al . Spectral similarity-based super resolution for hyperspectral images [J]. Acta Automatica Sinica , 2014 , 40 ( 12 ): 2797 - 2807 . (in Chinese)

徐金环 , 沈煜 , 刘鹏飞 , 等 . 联合核稀疏多元逻辑回归和TV-L1错误剔除的高光谱图像分类算法 [J]. 电子学报 , 2018 , 46 ( 1 ): 175 - 184 .

XU J H , SHEN Y , LIU P F , et al . Hyperspectral image classification combining kernel sparse multinomial logistic regression and TV-L1 error rejection [J]. Acta Electronica Sinica , 2018 , 46 ( 1 ): 175 - 184 . (in Chinese)

张号逵 , 李映 , 姜晔楠 . 深度学习在高光谱图像分类领域的研究现状与展望 [J]. 自动化学报 , 2018 , 44 ( 6 ): 961 - 977 .

ZHANG H K , LI Y , JIANG Y N . Deep learning for hyperspectral imagery classification: The state of the art and prospects [J]. Acta Automatica Sinica , 2018 , 44 ( 6 ): 961 - 977 . (in Chinese)

YOKOYA N , GROHNFELDT C , CHANUSSOT J . Hyperspectral and multispectral data fusion: A comparative review of the recent literature [J]. IEEE Geoscience and Remote Sensing Magazine , 2017 , 5 ( 2 ): 29 - 56 .

CARPER W R , LILLESAND T M , KIEFER R . The use of intensity-hue-saturation transformations for merging SPOT panchromatic and multispectral image data [J]. Photogrammetric Engineering and Remote Sensing , 1990 , 56 : 457 - 467 .

CHAVEZ P S J , KWARTENG A Y . Extracting spectral contrast in Landsat thematic mapper image data using selective principal component analysis [J]. Photogrammetric Engineering & Remote Sensing , 1989 , 55 ( 3 ): 339 - 348 .

AIAZZI B , BARONTI S , SELVA M . Improving component substitution pansharpening through multivariate regression of MS+Pan data [J]. IEEE Transactions on Geoscience and Remote Sensing , 2007 , 45 ( 10 ): 3230 - 3239 .

CHEN Z , PU H Y , WANG B , et al . Fusion of hyperspectral and multispectral images: A novel framework based on generalization of pan-sharpening methods [J]. IEEE Geoscience and Remote Sensing Letters , 2014 , 11 ( 8 ): 1418 - 1422 .

RANCHIN T , WALD L . Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation [J]. Photogrammetric Engineering & Remote Sensing , 2000 , 66 : 49 - 61 .

SELVA M , AIAZZI B , BUTERA F , et al . Hyper-sharpening: A first approach on SIM-GA data [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2015 , 8 ( 6 ): 3008 - 3024 .

NUNEZ J , OTAZU X , FORS O , et al . Multiresolution-based image fusion with additive wavelet decomposition [J]. IEEE Transactions on Geoscience and Remote Sensing , 1999 , 37 ( 3 ): 1204 - 1211 .

GOMEZ R B , JAZAERI A , KAFATOS M . Wavelet-based hyperspectral and multispectral image fusion [C]// Proceeding SPIE 4383, Geo-Spatial Image and Data Exploitation II . Orlando : SPIE , 2001 , 4383 : 36 - 42 .

ZHANG Y F , HE M Y . Multi-spectral and hyperspectral image fusion using 3-D wavelet transform [J]. Journal of Electronics (China) , 2007 , 24 ( 2 ): 218 - 224 .

AKHTAR N , SHAFAIT F , MIAN A . Bayesian sparse representation for hyperspectral image super resolution [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2015 : 3631 - 3640 .

ZHANG Y F , DE BACKER S , SCHEUNDERS P . Noise-resistant wavelet-based Bayesian fusion of multispectral and hyperspectral images [J]. IEEE Transactions on Geoscience and Remote Sensing , 2009 , 47 ( 11 ): 3834 - 3843 .

SIMÕES M , BIOUCAS-DIAS J , ALMEIDA L B , et al . A convex formulation for hyperspectral image superresolution via subspace-based regularization [J]. IEEE Transactions on Geoscience and Remote Sensing , 2015 , 53 ( 6 ): 3373 - 3388 .

BUNGERT L , COOMES D A , EHRHARDT M J , et al . Blind image fusion for hyperspectral imaging with the directional total variation [J]. Inverse Problems , 2018 , 34 ( 4 ): 044003 .

KAWAKAMI R , MATSUSHITA Y , WRIGHT J , et al . High-resolution hyperspectral imaging via matrix factorization [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2011 : 2329 - 2336 .

YOKOYA N , YAIRI T , IWASAKI A . Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion [J]. IEEE Transactions on Geoscience and Remote Sensing , 2012 , 50 ( 2 ): 528 - 537 .

LANARAS C , BALTSAVIAS E , SCHINDLER K . Hyperspectral super-resolution by coupled spectral unmixing [C]// 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2016 : 3586 - 3594 .

KWON H , TAI Y W . RGB-guided hyperspectral image upsampling [C]// 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2016 : 307 - 315 .

DONG W S , FU F Z , SHI G M , et al . Hyperspectral image super-resolution via non-negative structured sparse representation [J]. IEEE Transactions on Image Processing , 2016 , 25 ( 5 ): 2337 - 2352 .

LIU J J , WU Z B , XIAO L , et al . A truncated matrix decomposition for hyperspectral image super-resolution [J]. IEEE Transactions on Image Processing , 2020 , 29 : 8028 - 8042 .

DIAN R W , FANG L Y , LI S T . Hyperspectral image super-resolution via non-local sparse tensor factorization [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2017 : 3862 - 3871 .

DIAN R W , LI S T , FANG L Y . Learning a low tensor-train rank representation for hyperspectral image super-resolution [J]. IEEE Transactions on Neural Networks and Learning Systems , 2019 , 30 ( 9 ): 2672 - 2683 .

XU Y , WU Z B , CHANUSSOT J , et al . Nonlocal patch tensor sparse representation for hyperspectral image super-resolution [J]. IEEE Transactions on Image Processing , 2019 , 28 ( 6 ): 3034 - 3047 .

XU Y , WU Z B , CHANUSSOT J , et al . Nonlocal coupled tensor CP decomposition for hyperspectral and multispectral image fusion [J]. IEEE Transactions on Geoscience and Remote Sensing , 2020 , 58 ( 1 ): 348 - 362 .

HE W , CHEN Y , YOKOYA N , et al . Hyperspectral super-resolution via coupled tensor ring factorization [J]. Pattern Recognition , 2022 , 122 : 108280 .

LECUN Y , BENGIO Y , HINTON G . Deep learning [J]. Nature , 2015 , 521 ( 7553 ): 436 - 444 .

HAN X H , SHI B X , ZHENG Y Q . SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution [C]// 2018 25th IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2018 : 2506 - 2510 .

CHANG Y , YAN L X , FANG H Z , et al . HSI-DeNet: Hyperspectral image restoration via convolutional neural network [J]. IEEE Transactions on Geoscience and Remote Sensing , 2019 , 57 ( 2 ): 667 - 682 .

YANG J X , ZHAO Y Q , CHAN J . Hyperspectral and multispectral image fusion via deep two-branches convolutional neural network [J]. Remote Sensing , 2018 , 10 ( 5 ): 800 .

DIAN R W , LI S T , GUO A J , et al . Deep hyperspectral image sharpening [J]. IEEE Transactions on Neural Networks and Learning Systems , 2018 , 29 ( 11 ): 5345 - 5355 .

WANG W , ZENG W H , HUANG Y , et al . Deep blind hyperspectral image fusion [C]// 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2020 : 4149 - 4158 .

ZHANG L , NIE J T , WEI W , et al . Deep blind hyperspectral image super-resolution [J]. IEEE Transactions on Neural Networks and Learning Systems , 2021 , 32 ( 6 ): 2388 - 2400 .

YANG J X , XIAO L , ZHAO Y Q , et al . Variational regularization network with attentive deep prior for hyperspectral-multispectral image fusion [J]. IEEE Transactions on Geoscience and Remote Sensing , 2022 , 60 : 1 - 17 .

QU Y , QI H R , KWAN C . Unsupervised sparse dirichlet-net for hyperspectral image super-resolution [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 2511 - 2520 .

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern recognition . 2016 : 770 - 778 .

HAN X H , CHEN Y W . Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution [C]// 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM) . Piscataway : IEEE , 2019 : 266 - 270 .

XIAO J J , YUAN Q Q , LI J , et al . Enhanced residual dense network joint with GRUS for multispectral and hyperspectral image fusion [C]// 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS . Piscataway : IEEE , 2021 : 2448 - 2451 .

LIU J , ZHANG W J , TANG Y T , et al . Residual feature aggregation network for image super-resolution [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 2356 - 2365 .

王相海 , 赵晓阳 , 毕晓昀 , 等 . 小波域多角度轮廓模板变分模型的单幅图像超分辨率重建 [J]. 电子学报 , 2018 , 46 ( 9 ): 2256 - 2262 .

WANG X H , ZHAO X Y , BI X Y , et al . Single image super-resolution reconstruction approach based on multi-angle contour templates variational calculus model in wavelet domain [J]. Acta Electronica Sinica , 2018 , 46 ( 9 ): 2256 - 2262 . (in Chinese)

STARCK J L , FADILI J , MURTAGH F . The undecimated wavelet decomposition and its reconstruction [J]. IEEE Transactions on Image Processing , 2007 , 16 ( 2 ): 297 - 309 .

王相海 , 赵晓阳 , 王鑫莹 , 等 . 非抽取小波边缘学习深度残差网络的单幅图像超分辨率重建 [J]. 电子学报 , 2022 , 50 ( 7 ): 1753 - 1765 .

WANG X H , ZHAO X Y , WANG X Y , et al . Single image super-resolution reconstruction using deep residual networks with non-decimated wavelet edge learning [J]. Acta Electronica Sinica , 2022 , 50 ( 7 ): 1753 - 1765 . (in Chinese)

SHENSA M J . The discrete wavelet transform: Wedding the a trous and Mallat algorithms [C]// IEEE Transactions on Signal Processing . Piscataway : IEEE , 2002 : 2464 - 2482 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [J]. IEEE Transactions on Signal Processing , 1992 , 61 ( 3 ): 211 - 231 .

LIM B , SON S , KIM H , et al . Enhanced deep residual networks for single image super-resolution [C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE , 2017 : 1132 - 1140 .

TAI Y , YANG J , LIU X , et al . Memnet: A persistent memory network for image restoration [C]// Proceedings of the IEEE International Conference on Computer Vision . Piscataway : IEEE , 2017 : 4549 - 4557 .

KINGMA D P , BA J . Adam: A method for stochastic optimization [EB/OL]. ( 2014-12-22 )[ 2022-07-08 ]. https://arxiv.org/abs/1412.6980 https://arxiv.org/abs/1412.6980 .

DIAN R W , LI S T , KANG X D . Regularizing hyperspectral and multispectral image fusion by CNN denoiser [J]. IEEE Transactions on Neural Networks and Learning Systems , 2021 , 32 ( 3 ): 1124 - 1135 .

DONG W S , ZHOU C , WU F F , et al . Model-guided deep hyperspectral image super-resolution [J]. IEEE Transactions on Image Processing , 2021 , 30 : 5754 - 5768 .

XIE Q , ZHOU M H , ZHAO Q , et al . MHF-net: An interpretable deep network for multispectral and hyperspectral image fusion [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022 , 44 ( 3 ): 1457 - 1473 .

SCARPA G , VITALE S , COZZOLINO D . Target-adaptive CNN-based pansharpening [J]. IEEE Transactions on Geoscience and Remote Sensing , 2018 , 56 ( 9 ): 5443 - 5457 .

YANG J X , ZHAO Y Q , YI C , et al . No-reference hyperspectral image quality assessment via quality-sensitive features learning [J]. Remote Sensing , 2017 , 9 ( 4 ): 305 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Textual Semantic Guidance for Infrared and Visible Image Fusion

SSCDenseNet:A Spectral-Spatial Convolutional Dense Network for Hyperspectral Image Classification

A Low-Light Image Enhancement Algorithm Using the Hybrid Strategy of Deep Learning and Image Fusion

An Infrared Polarization Based UAV Detection Method for Complex Environment

Related Author

XIAO Liang

YANG Jing-Xiang

ZHU Mingrui

CHEN Xiru

WEI Xin

WANG Nannan

GAO Xinbo

LIU Qi-chao

Related Institution

Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense， Nanjing University of Science and Technology

State Key Laboratory of Integrated Services Networks, Xidian University

School of Computer Science and Engineering, Nanjing University of Science and Technology

School of Information Engineering, Nanchang University

School of Automation, Northwestern Polytechnical University

⁰