Shared Super-Resolution Dual-Branch Network for Spatiotemporal Fusion of Remote Sensing Images

FANG Shuai; ZHANG Xiao-xi; ZHANG Jing

doi:10.12263/DZXB.20240324

您当前的位置：

首页 >

文章列表页 >

Shared Super-Resolution Dual-Branch Network for Spatiotemporal Fusion of Remote Sensing Images

PAPERS | 更新时间：2025-12-08

- Shared Super-Resolution Dual-Branch Network for Spatiotemporal Fusion of Remote Sensing Images
- ACTA ELECTRONICA SINICA Vol. 53, Issue 2, Pages: 581-594(2025)
- 作者机构：
  
  1.合肥工业大学计算机与信息学院，安徽合肥 230000
  2.工业安全与应急技术安徽省重点实验室，安徽合肥 230000
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(61872327;61175033)
- DOI：10.12263/DZXB.20240324
  CLC： TP751.1;
- Received：09 April 2024，
  
  Revised：2024-09-18，
  
  Published：25 February 2025
- 稿件说明：
移动端阅览
方帅, 张小溪, 张晶. 共享超分的双分支遥感图像时空融合网络[J]. 电子学报, 2025, 53(02): 581-594.

FANG Shuai, ZHANG Xiao-xi, ZHANG Jing. Shared Super-Resolution Dual-Branch Network for Spatiotemporal Fusion of Remote Sensing Images[J]. Acta Electronica Sinica, 2025, 53(02): 581-594.
方帅, 张小溪, 张晶. 共享超分的双分支遥感图像时空融合网络[J]. 电子学报, 2025, 53(02): 581-594. DOI：10.12263/DZXB.20240324

FANG Shuai, ZHANG Xiao-xi, ZHANG Jing. Shared Super-Resolution Dual-Branch Network for Spatiotemporal Fusion of Remote Sensing Images[J]. Acta Electronica Sinica, 2025, 53(02): 581-594. DOI：10.12263/DZXB.20240324

摘要

本文从空间维度和时间维度分析了场景弱变化区域和类型变化区域的融合规律、物理模型的差异性和效果上的互补性，提出了共享超分辨率的双分支（Shared Super-Resolution Dual-Branch，SSRDB）遥感图像时空融合算法.该算法具有如下3个特点：（1）构建了互补性的网络框架，虽然该框架是端到端的深度学习模型，但每个模块有各自的物理意义和任务，通过增加中间监督，分别实现空间维的超分建模，时间维的变化预测建模，以及两者优势互补的融合建模；（2）对变化预测的数学表示进行推演，利用一个非线性补偿模块，使得两分支共享超分模块，在共享超分模块和递归复用超分单元的双重策略下，显著降低了网络参数；（3）递归超分模块使用固定的2倍率超分单元，在有效监督和有效参考下，渐进式进行特征增强与图像重建，这可以有效提高超分精度，且通过调整超分单元个数，灵活适应不同倍率差异的时空融合任务.SSRDB算法在空间和光谱特性上以及变化区域上都展现了优秀的融合效果，RMSE（Root Mean Squared Error）、SAM（Spectral Angle Mapper）和SSIM（Structural Similarity）3个定量评价指标显示，在CIA（Coleambally lrrigation Area）数据集上分别优于次优方法7.067%、2.065%、0.563%；在LGC（Lower Gwydir Catchment）数据集上分别优于次优方法5.319%、5.490%、1.455%；在Nanjing数据集上分别优于次优方法6.486%、16.290%、0.481%.

Abstract

In this paper

we analyze the fusion law of scene weak change region and type change region

the difference of physical model and the complementarity of effect from spatial and temporal dimensions

and propose a shared super-resolution dual-branch (Shared Super-Resolution Dual-Branch

SSRDB) remote sensing image spatio-temporal fusion algorithm. The algorithm has the following three characteristics: (1) A complementary network framework is constructed. Although the framework is an end-to-end deep learning model

each module has its own physical meaning and task. By adding intermediate supervision

the super-resolution modeling of spatial dimension

the change prediction modeling of time dimension and the fusion modeling of the two advantages are realized respectively. (2) The mathematical representation of the change prediction is deduced

and a nonlinear compensation module is used to make the two branches share the super-resolution module. Under the dual strategy of sharing super-resolution module and recursive multiplexing super-resolution unit

the network parameters are significantly reduced. (3) The recursive super-resolution module uses fixed 2-magnification super-resolution units to gradually enhance features and reconstruct images under effective supervision and reference

which can effectively improve the precision of super-resolution

and flexibly adapt to spatio-temporal fusion tasks with different magnification differences by adjusting the number of super-resolution units. The SSRDB algorithm shows excellent fusion effect in spatial and spectral characteristics and change regions. The three quantitative evaluation indexes of RMSE (Root Mean Squared Error)、SAM (Spectral Angle Mapper) and SSIM (Structural Similarity) show that it is superior to the sub-optimal method on the CIA (Coleambally lrrigation Area) dataset by 7.067%

2.065% and 0.563%

respectively. On the LGC (Lower Gwydir Catchment) dataset

it is superior to the sub-optimal method by 5.319%

5.490% and 1.455%

respectively. On the Nanjing dataset

it is superior to the suboptimal method by 6.486%

16.290% and 0.481%

respectively.

关键词

Keywords

references

ZHANG Y . A time-series approach to detect urbanized areas using biophysical indicators and landsat satellite imagery [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2021 , 14 : 9210 - 9222 .

ARÉVALO P , OLOFSSON P , WOODCOCK C E . Continuous monitoring of land change activities and post-disturbance dynamics from landsat time series: A test methodology for REDD+ reporting [J ] . Remote Sensing of Environment , 2020 , 238 : 111051 .

YANG Y , ANDERSON M C , GAO F , et al . Studying drought-induced forest mortality using high spatiotemporal resolution evapotranspiration data from thermal satellite imaging [J ] . Remote Sensing of Environment , 2021 , 265 : 112640 .

LUO X , WANG M C , DAI G M , et al . A novel technique to compute the revisit time of satellites and its application in remote sensing satellite optimization design [J ] . International Journal of Aerospace Engineering , 2017 , 2017 : 6469439 .

GAO F , MASEK J , SCHWALLER M , et al . On the blending of the landsat and MODIS surface reflectance: Predicting daily landsat surface reflectance [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2006 , 44 ( 8 ): 2207 - 2218 .

HILKER T , WULDER M A , COOPS N C , et al . A new data fusion model for high spatial- and temporal-resolution mapping of forest disturbance based on Landsat and MODIS [J ] . Remote Sensing of Environment , 2009 , 113 ( 8 ): 1613 - 1627 .

ZHU X L , CHEN J , GAO F , et al . An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions [J ] . Remote Sensing of Environment , 2010 , 114 ( 11 ): 2610 - 2623 .

FU D J , CHEN B Z , WANG J , et al . An improved image fusion approach based on enhanced spatial and temporal the adaptive reflectance fusion model [J ] . Remote Sensing , 2013 , 5 ( 12 ): 6346 - 6360 .

ZHUKOV B , OERTEL D , LANZL F , et al . Unmixing-based multisensor multiresolution image fusion [J ] . IEEE Transactions on Geoscience and Remote Sensing , 1999 , 37 ( 3 ): 1212 - 1226 .

WU M Q , NIU Z , WANG C Y , et al . Use of MODIS and Landsat time series data to generate high-resolution temporal synthetic Landsat data using a spatial and temporal reflectance fusion model [J ] . Journal of Applied Remote Sensing , 2012 , 6 ( 1 ): 063507 .

ZHANG W , LI A N , JIN H A , et al . An enhanced spatial and temporal data fusion model for fusing landsat and MODIS surface reflectance to generate high temporal landsat-like data [J ] . Remote Sensing , 2013 , 5 ( 10 ): 5346 - 5368 .

ZHU X L , HELMER E H , GAO F , et al . A flexible spatiotemporal method for fusing satellite images with different resolutions [J ] . Remote Sensing of Environment , 2016 , 172 : 165 - 177 .

WANG Q M , ATKINSON P M . Spatio-temporal fusion for daily Sentinel-2 images [J ] . Remote Sensing of Environment , 2018 , 204 : 31 - 42 .

HUANG B , SONG H H . Spatiotemporal reflectance fusion via sparse representation [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2012 , 50 ( 10 ): 3707 - 3716 .

WU B , HUANG B , ZHANG L P . An error-bound-regularized sparse coding for spatiotemporal reflectance fusion [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2015 , 53 ( 12 ): 6791 - 6803 .

WEI J B , WANG L Z , LIU P , et al . Spatiotemporal fusion of remote sensing images with structural sparsity and semi-coupled dictionary learning [J ] . Remote Sensing , 2017 , 9 ( 1 ): 21 .

SONG H H , LIU Q S , WANG G J , et al . Spatiotemporal satellite image fusion using deep convolutional neural networks [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2018 , 11 ( 3 ): 821 - 829 .

TAN Z Y , YUE P , DI L P , et al . Deriving high spatiotemporal remote sensing images using deep convolutional network [J ] . Remote Sensing , 2018 , 10 ( 7 ): 1066 .

TAN Z Y , DI L P , ZHANG M D , et al . An enhanced deep convolutional model for spatiotemporal image fusion [J ] . Remote Sensing , 2019 , 11 ( 24 ): 2898 .

ZHANG H Y , SONG Y Y , HAN C , et al . Remote sensing image spatiotemporal fusion using a generative adversarial network [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2021 , 59 ( 5 ): 4273 - 4286 .

LEDIG C , THEIS L , HUSZÁR F , et al . Photo-realistic single image super-resolution using a generative adversarial network [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2017 : 105 - 114 .

TAN Z Y , GAO M L , LI X H , et al . A flexible reference-insensitive spatiotemporal fusion model for remote sensing images using conditional generative adversarial network [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2021 , 60 : 5601413 .

LIU X , DENG C W , CHANUSSOT J , et al . StfNet: A two-stream convolutional neural network for spatiotemporal image fusion [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2019 , 57 ( 9 ): 6552 - 6564 .

SONG B Z , LIU P , LI J , et al . MLFF-GAN: A multilevel feature fusion with GAN for spatiotemporal remote sensing images [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2022 , 60. 60 : 4410816 .

SONG Y Y , ZHANG H Y , HUANG H , et al . Remote sensing image spatiotemporal fusion via a generative adversarial network with one prior image pair [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2022 , 60 : 5528117 .

YANG G Q , QIAN Y R , LIU H , et al . MSFusion: Multistage for remote sensing image spatiotemporal fusion based on texture transformer and convolutional neural network [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2022 , 15 : 4653 - 4666 .

EMELYANOVA I V , MCVICAR T R , VAN NIEL T G , et al . Assessing the accuracy of blending landsat-MODIS surface reflectances in two landscapes with contrasting spatial and temporal dynamics: A framework for algorithm selection [J ] . Remote Sensing of Environment , 2013 , 133 : 193 - 209 .

WANG Z Y , FANG S , ZHANG J . Spatiotemporal fusion model of remote sensing images combining single-band and multi-band prediction [J ] . Remote Sensing , 2023 , 15 ( 20 ): 4936 .

YUHAS R H , GOETZ A F H , BOARDMAN J W . Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm [C ] // Summaries of the 3rd Annual JPL Airborne Geoscience Workshop . Pasadena : JPL , 1992 : 147 - 149 .

WANG Z , BOVIK A C , SHEIKH H R , et al . Image quality assessment: From error visibility to structural similarity [J ] . IEEE Transactions on Image Processing , 2004 , 13 ( 4 ): 600 - 612 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Operator Fusion Method and Hardware Architecture Design Based on Non-Standard Operators

UHD Multi-Exposure Image Fusion via Dense Local-Global Feature Aggregation

Lightweight Fully-Connected Tensorial Mapping Network for Hyperspectral Image Classification

Cross-CNN: An Animation Cross-Frame Sketch Colorization Algorithm Based on Hybrid Model with CNN and Transformer

Related Author

WANG Ying

GAO Lan

ZHANG Zhe

LIU Xin

WU Yi-xiong

ZHANG Wei-gong

JIA Xiu-yi

LIN Qiao-wan-ni

Related Institution

College of Information Engineering, Capital Normal University

School of Mathematical Science, Capital Normal University

Faculty of Software Technologics, Shanxi Agricultural University

School of Computer Science and Engineering, Nanjing University of Science and Technology

School of Computer Science and Engineering, Xi’an University of Technology

⁰