Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification

HE Xin; CHEN Yu-shi; GU Yan-feng; LIU Tian-zhu

doi:10.12263/DZXB.20250882

您当前的位置：

首页 >

文章列表页 >

Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification

Special Issue\: Recipients of CIE Science and Technology Awards | 更新时间：2026-04-24

- Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification
- ACTA ELECTRONICA SINICA Vol. 53, Issue 12, Pages: 4267-4278(2025)
- 作者机构：
  
  哈尔滨工业大学电子与信息工程学院，黑龙江哈尔滨 150001
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62025107;62371169);Natural Science Foundation of Heilongjiang Province of China(LH2024F015)
- DOI：10.12263/DZXB.20250882
  CLC： TP751;
- Received：06 November 2025，
  
  Accepted：06 December 2025，
  
  Published：25 December 2025
- 稿件说明：
移动端阅览
何欣, 陈雨时, 谷延锋, 等. 面向不确定性融合的多模态遥感数据分类[J]. 电子学报, 2025, 53(12): 4267-4278.

HE Xin, CHEN Yu-shi, GU Yan-feng, et al. Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification[J]. Acta Electronica Sinica, 2025, 53(12): 4267-4278.
何欣, 陈雨时, 谷延锋, 等. 面向不确定性融合的多模态遥感数据分类[J]. 电子学报, 2025, 53(12): 4267-4278. DOI：10.12263/DZXB.20250882

HE Xin, CHEN Yu-shi, GU Yan-feng, et al. Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification[J]. Acta Electronica Sinica, 2025, 53(12): 4267-4278. DOI：10.12263/DZXB.20250882

摘要

分类是多模态遥感数据解译的关键技术和热点问题.近年来，深度学习方法在多模态遥感数据像素级分类中取得了长足的进展.然而，多模态遥感数据包含的不同模态经过特征提取后，预测结果存在不一致的问题，即预测结果不确定性，影响多模态遥感数据分类方法的分类精度.为降低预测结果不确定性，本文提出了一种面向不确定性融合的多模态遥感数据分类框架.该框架联合提取不同模态（如合成孔径雷达数据、激光雷达数据、高光谱数据等）的空间特征与通道特征，并构建对应的神经网络，通过设计特定的证据融合函数，实现基于证据信息的有效融合.在融合过程中，当预测结果存在冲突时，设计基于冲突感知的动态权重调整机制，通过折扣因子自适应地降低冲突模态的权重，动态重分配证据质量，从而有效降低多模态遥感数据分类方法的不确定性.在此基础上，为进一步预测结果的差异，本文在模型参数优化过程中引入一致性损失函数，以约束各模态预测结果的一致性.实验在3种国际公开的多模态遥感数据集上进行，并与6种主流方法进行对比，结果表明本文所提方法在分类性能上均取得了显著提升.

Abstract

Classification is a key technique and a hot topic in the interpretation of multimodal remote sensing data. In recent years

deep learning methods have achieved significant progress in pixel-level classification of multimodal remote sensing data. However

the different modalities contained in multimodal remote sensing data exhibit variability in their predicted results after feature extraction

which is referred to as predictive uncertainty. This uncertainty negatively impacts the classification accuracy of multimodal remote sensing data classification methods. To reduce the prediction uncertainty

this paper proposes an uncertainty-aware fusion framework for multimodal remote sensing data classification. From the perspective of evidence quality

the framework jointly extracts spatial and channel features from different modalities (e.g.

synthetic aperture radar data

light detection and ranging data

hyperspectral image

etc.) and constructs corresponding evidential neural networks. A specifically designed evidential fusion function is employed to effectively integrate multimodal evidential information. During the fusion process

when conflicting predictions arise from different modalities

a conflict-aware dynamic weight adjustment mechanism is introduced. This mechanism adaptively reduces the weight of conflicting modalities using a discount factor and dynamically reallocates the quality of evidence

thereby effectively reducing model uncertainty. Furthermore

to further minimize the discrepancy among predictions from different modalities

a consistency loss function is incorporated into the model parameter optimization process to constrain the consistency of predictions across modalities. Experiments are conducted on three publicly available international multimodal remote sensing datasets

and the proposed method is compared with six state-of-the-art approaches. The results demonstrate that the proposed method achieves significant improvements in classification performance.

关键词

Keywords

references

SUN X , TIAN Y , LU W X , et al . From single- to multi-modal remote sensing imagery interpretation: A survey and taxonomy [J ] . Science China Information Sciences , 2023 , 66 ( 4 ): 140301 .

ZHAO X D , ZHANG M M , TAO R , et al . Fractional Fourier image transformer for multimodal remote sensing data classification [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2024 , 35 ( 2 ): 2314 - 2326 .

KIEU N , NGUYEN K , NAZIB A , et al . Multimodal colearning meets remote sensing: Taxonomy, state of the art, and future works [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2024 , 17 : 7386 - 7409 .

HONG D F , LI C Y , ZHANG B , et al . Multimodal artificial intelligence foundation models: Unleashing the power of remote sensing big data in earth observation [J ] . The Innovation Geoscience , 2024 , 2 ( 1 ): 100055 .

贾永红 , 李德仁 . 多源遥感影像像素级融合分类与决策级分类融合法的研究 [J ] . 武汉大学学报(信息科学版) , 2001 , 26 ( 5 ): 430 - 434 .

JIA Y H , LI D R . An approach of classification based on pixel level and decision level fusion of multi-source images in remote sensing [J ] . Editoral Board of Geomatics and Information Science of Wuhan University , 2001 , 26 ( 5 ): 430 - 434 . (in Chinese)

张良培 , 沈焕锋 . 遥感数据融合的进展与前瞻 [J ] . 遥感学报 , 2016 , 20 ( 5 ): 1050 - 1061 .

ZHANG L P , SHEN H F . Progress and future of remote sensing data fusion [J ] . Journal of Remote Sensing , 2016 , 20 ( 5 ): 1050 - 1061 . (in Chinese)

WU X , HONG D F , CHANUSSOT J . Convolutional neural networks for multimodal remote sensing data classification [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2022 , 60 : 5517010 .

付琨 , 王佩瑾 , 冯瑛超 , 等 . 遥感跨模态智能解译: 模型、数据与应用 [J ] . 中国科学(信息科学) , 2023 , 53 ( 8 ): 1529 - 1559 .

FU K , WANG P J , FENG Y C , et al . Cross-modal intelligent interpretation of remote sensing: Model, data and application [J ] . Science in China (Information Sciences) , 2023 , 53 ( 8 ): 1529 - 1559 . (in Chinese)

TU B , REN Q , LI J , et al . NCGLF2: Network combining global and local features for fusion of multisource remote sensing data [J ] . Information Fusion , 2024 , 104 : 102192 .

LI J X , HONG D F , GAO L R , et al . Deep learning in multimodal remote sensing data fusion: A comprehensive review [J ] . International Journal of Applied Earth Observation and Geoinformation , 2022 , 112 : 102926 .

YANG B , WANG X , XING Y , et al . Modality fusion vision transformer for hyperspectral and LiDAR data collaborative classification [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2024 , 17 : 17052 - 17065 .

ROY S K , DERIA A , HONG D F , et al . Multimodal fusion transformer for remote sensing image classification [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2023 , 61 : 5515620 .

YAO J , ZHANG B , LI C Y , et al . Extended vision transformer (ExViT) for land use and land cover classification: A multimodal deep learning framework [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2023 , 61 : 5514415 .

LI Y S , LI D X , XIE W Y , et al . Semi-mamba: Mamba-driven semi-supervised multimodal remote sensing feature classification [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2025 , 35 ( 10 ): 9837 - 9849 .

ZHANG G L , ZHANG Z X , DENG J W , et al . S 2 CrossMamba: Spatial-spectral cross-mamba for multimodal remote sensing image classification [J ] . IEEE Geoscience and Remote Sensing Letters , 2024 , 21 : 5510705 .

MA M R , ZHAO J X , MA W P , et al . A mamba-aware spatial-spectral cross-modal network for remote sensing classification [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2025 , 63 : 4402515 .

WEN J G , WU X D , YOU D Q , et al . The main inherent uncertainty sources in trend estimation based on satellite remote sensing data [J ] . Theoretical and Applied Climatology , 2023 , 151 ( 1 ): 915 - 934 .

WOODCOCK C E . Uncertainty in Remote Sensing [M ] // Uncertainty in Remote Sensing and GIS . Hoboken : Wiley , 2002 : 19 - 24 .

KOLLER C , SHAHRAD M , ZHU X X . Uncertainty-guided representation learning in local climate zone classification [C ] // IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium . Piscataway : IEEE , 2022 : 183 - 186 .

XU Y H , BAI T , YU W K , et al . AI security for geoscience and remote sensing: Challenges and future trends [J ] . IEEE Geoscience and Remote Sensing Magazine , 2023 , 11 ( 2 ): 60 - 85 .

ZINNER T , HAUSMANN P , EWALD F , et al . Ground-based imaging remote sensing of ice clouds: Uncertainties caused by sensor, method and atmosphere [J ] . Atmospheric Measurement Techniques , 2016 , 9 ( 9 ): 4615 - 4632 .

HAN Z B , ZHANG C Q , FU H Z , et al . Trusted multi-view classification with dynamic evidential fusion [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 2 ): 2551 - 2566 .

HAN Z B , YANG F , HUANG J Z , et al . Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 20675 - 20685 .

HAN Z B , YANG Y F , ZHANG C Q , et al . Selective learning: Towards robust calibration with dynamic regularization [EB/OL ] . ( 2024-07-14 )[ 2025-11-10 ] . https://arXiv.org/abs/2402.08384 https://arXiv.org/abs/2402.08384 .

RUSSELL R L , REALE C . Multivariate uncertainty in deep learning [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2022 , 33 ( 12 ): 7937 - 7943 .

TYRALIS H , PAPACHARALAMPOUS G . A review of predictive uncertainty estimation with machine learning [J ] . Artificial Intelligence Review , 2024 , 57 ( 4 ): 94 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Semi-Supervised Medical Image Segmentation Based on Suspicious Pixel Mutual Correction

Multi-Level Description of Granules From an Outsider’s Perspective

Sequential Three-Way Decision Model Based on the Game Between Uncertainty and Error Classification Rate

Multi-Branch Structure Based Local Channel Attention with Uncertainty

Related Author

SUN Fei-man

WANG Meng-xi

DU Xiao-gang

LEI Tao

YANG Zi-yao

GONG Mao-guo

LI Jin-hai

ZHANG Li

Related Institution

Key Laboratory of Collaborative Intelligent Systems, Ministry of Education, Xidian University

School of Mathematics, Southwest Jiaotong University

Shaanxi Joint Laboratory of Artificial Intelligence, Shaanxi University of Science and Technology

School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology

School of Computer Science and Technology, Henan Polytechnic University

⁰