面向不确定性融合的多模态遥感数据分类

何欣; 陈雨时; 谷延锋; 刘天竹

doi:10.12263/DZXB.20250882

您当前的位置：

首页 >

文章列表页 >

面向不确定性融合的多模态遥感数据分类

中国电子学会科学技术奖特约专栏 | 更新时间：2026-04-24

- 面向不确定性融合的多模态遥感数据分类
- Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification
- 电子学报 2025年53卷第12期页码：4267-4278
- 作者机构：
  
  哈尔滨工业大学电子与信息工程学院，黑龙江哈尔滨 150001
- 作者简介：
  
  [ "何欣女，1994年1月出生于黑龙江省齐齐哈尔市.现为哈尔滨工业大学副研究员.主要研究方向为遥感图像智能处理.E-mail: hexin1@hit.edu.cn" ]
  [ "陈雨时男，1978年9月出生于河南省信阳市.现为哈尔滨工业大学教授、博士生导师.主要研究方向为遥感图像处理及深度学习理论和应用.E-mail: chenyushi@hit.edu.cn" ]
  [ "谷延锋男，1977年12月出生于黑龙江省佳木斯市.现为哈尔滨工业大学电子与信息工程学院教授、博士生导师.主要研究方向为空天智能信息处理、高光谱遥感、多维度信息探测与处理系统.E-mail: guyf@hit.edu.cn" ]
  [ "刘天竹女，1990年12月出生于黑龙江省哈尔滨市.现为哈尔滨工业大学电子与信息工程学院研究员、博士生导师.主要研究方向为多模遥感信息智能处理.中国电子学会会员编号：E190025027M.E-mail: tzliu@hit.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(62025107;62371169);黑龙江省自然科学基金(LH2024F015)
- DOI：10.12263/DZXB.20250882
  中图分类号： TP751;
- 收稿：2025-11-06，
  
  录用：2025-12-06，
  
  纸质出版：2025-12-25
- 稿件说明：
移动端阅览
何欣, 陈雨时, 谷延锋, 等. 面向不确定性融合的多模态遥感数据分类[J]. 电子学报, 2025, 53(12): 4267-4278.

HE Xin, CHEN Yu-shi, GU Yan-feng, et al. Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification[J]. Acta Electronica Sinica, 2025, 53(12): 4267-4278.
何欣, 陈雨时, 谷延锋, 等. 面向不确定性融合的多模态遥感数据分类[J]. 电子学报, 2025, 53(12): 4267-4278. DOI：10.12263/DZXB.20250882

HE Xin, CHEN Yu-shi, GU Yan-feng, et al. Uncertainty Fusion-Based Multimodal Remote Sensing Data Classification[J]. Acta Electronica Sinica, 2025, 53(12): 4267-4278. DOI：10.12263/DZXB.20250882

摘要

分类是多模态遥感数据解译的关键技术和热点问题.近年来，深度学习方法在多模态遥感数据像素级分类中取得了长足的进展.然而，多模态遥感数据包含的不同模态经过特征提取后，预测结果存在不一致的问题，即预测结果不确定性，影响多模态遥感数据分类方法的分类精度.为降低预测结果不确定性，本文提出了一种面向不确定性融合的多模态遥感数据分类框架.该框架联合提取不同模态（如合成孔径雷达数据、激光雷达数据、高光谱数据等）的空间特征与通道特征，并构建对应的神经网络，通过设计特定的证据融合函数，实现基于证据信息的有效融合.在融合过程中，当预测结果存在冲突时，设计基于冲突感知的动态权重调整机制，通过折扣因子自适应地降低冲突模态的权重，动态重分配证据质量，从而有效降低多模态遥感数据分类方法的不确定性.在此基础上，为进一步预测结果的差异，本文在模型参数优化过程中引入一致性损失函数，以约束各模态预测结果的一致性.实验在3种国际公开的多模态遥感数据集上进行，并与6种主流方法进行对比，结果表明本文所提方法在分类性能上均取得了显著提升.

Abstract

Classification is a key technique and a hot topic in the interpretation of multimodal remote sensing data. In recent years

deep learning methods have achieved significant progress in pixel-level classification of multimodal remote sensing data. However

the different modalities contained in multimodal remote sensing data exhibit variability in their predicted results after feature extraction

which is referred to as predictive uncertainty. This uncertainty negatively impacts the classification accuracy of multimodal remote sensing data classification methods. To reduce the prediction uncertainty

this paper proposes an uncertainty-aware fusion framework for multimodal remote sensing data classification. From the perspective of evidence quality

the framework jointly extracts spatial and channel features from different modalities (e.g.

synthetic aperture radar data

light detection and ranging data

hyperspectral image

etc.) and constructs corresponding evidential neural networks. A specifically designed evidential fusion function is employed to effectively integrate multimodal evidential information. During the fusion process

when conflicting predictions arise from different modalities

a conflict-aware dynamic weight adjustment mechanism is introduced. This mechanism adaptively reduces the weight of conflicting modalities using a discount factor and dynamically reallocates the quality of evidence

thereby effectively reducing model uncertainty. Furthermore

to further minimize the discrepancy among predictions from different modalities

a consistency loss function is incorporated into the model parameter optimization process to constrain the consistency of predictions across modalities. Experiments are conducted on three publicly available international multimodal remote sensing datasets

and the proposed method is compared with six state-of-the-art approaches. The results demonstrate that the proposed method achieves significant improvements in classification performance.

关键词

Keywords

references

SUN X , TIAN Y , LU W X , et al . From single- to multi-modal remote sensing imagery interpretation: A survey and taxonomy [J ] . Science China Information Sciences , 2023 , 66 ( 4 ): 140301 .

ZHAO X D , ZHANG M M , TAO R , et al . Fractional Fourier image transformer for multimodal remote sensing data classification [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2024 , 35 ( 2 ): 2314 - 2326 .

KIEU N , NGUYEN K , NAZIB A , et al . Multimodal colearning meets remote sensing: Taxonomy, state of the art, and future works [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2024 , 17 : 7386 - 7409 .

HONG D F , LI C Y , ZHANG B , et al . Multimodal artificial intelligence foundation models: Unleashing the power of remote sensing big data in earth observation [J ] . The Innovation Geoscience , 2024 , 2 ( 1 ): 100055 .

贾永红 , 李德仁 . 多源遥感影像像素级融合分类与决策级分类融合法的研究 [J ] . 武汉大学学报(信息科学版) , 2001 , 26 ( 5 ): 430 - 434 .

JIA Y H , LI D R . An approach of classification based on pixel level and decision level fusion of multi-source images in remote sensing [J ] . Editoral Board of Geomatics and Information Science of Wuhan University , 2001 , 26 ( 5 ): 430 - 434 . (in Chinese)

张良培 , 沈焕锋 . 遥感数据融合的进展与前瞻 [J ] . 遥感学报 , 2016 , 20 ( 5 ): 1050 - 1061 .

ZHANG L P , SHEN H F . Progress and future of remote sensing data fusion [J ] . Journal of Remote Sensing , 2016 , 20 ( 5 ): 1050 - 1061 . (in Chinese)

WU X , HONG D F , CHANUSSOT J . Convolutional neural networks for multimodal remote sensing data classification [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2022 , 60 : 5517010 .

付琨 , 王佩瑾 , 冯瑛超 , 等 . 遥感跨模态智能解译: 模型、数据与应用 [J ] . 中国科学(信息科学) , 2023 , 53 ( 8 ): 1529 - 1559 .

FU K , WANG P J , FENG Y C , et al . Cross-modal intelligent interpretation of remote sensing: Model, data and application [J ] . Science in China (Information Sciences) , 2023 , 53 ( 8 ): 1529 - 1559 . (in Chinese)

TU B , REN Q , LI J , et al . NCGLF2: Network combining global and local features for fusion of multisource remote sensing data [J ] . Information Fusion , 2024 , 104 : 102192 .

LI J X , HONG D F , GAO L R , et al . Deep learning in multimodal remote sensing data fusion: A comprehensive review [J ] . International Journal of Applied Earth Observation and Geoinformation , 2022 , 112 : 102926 .

YANG B , WANG X , XING Y , et al . Modality fusion vision transformer for hyperspectral and LiDAR data collaborative classification [J ] . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2024 , 17 : 17052 - 17065 .

ROY S K , DERIA A , HONG D F , et al . Multimodal fusion transformer for remote sensing image classification [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2023 , 61 : 5515620 .

YAO J , ZHANG B , LI C Y , et al . Extended vision transformer (ExViT) for land use and land cover classification: A multimodal deep learning framework [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2023 , 61 : 5514415 .

LI Y S , LI D X , XIE W Y , et al . Semi-mamba: Mamba-driven semi-supervised multimodal remote sensing feature classification [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2025 , 35 ( 10 ): 9837 - 9849 .

ZHANG G L , ZHANG Z X , DENG J W , et al . S 2 CrossMamba: Spatial-spectral cross-mamba for multimodal remote sensing image classification [J ] . IEEE Geoscience and Remote Sensing Letters , 2024 , 21 : 5510705 .

MA M R , ZHAO J X , MA W P , et al . A mamba-aware spatial-spectral cross-modal network for remote sensing classification [J ] . IEEE Transactions on Geoscience and Remote Sensing , 2025 , 63 : 4402515 .

WEN J G , WU X D , YOU D Q , et al . The main inherent uncertainty sources in trend estimation based on satellite remote sensing data [J ] . Theoretical and Applied Climatology , 2023 , 151 ( 1 ): 915 - 934 .

WOODCOCK C E . Uncertainty in Remote Sensing [M ] // Uncertainty in Remote Sensing and GIS . Hoboken : Wiley , 2002 : 19 - 24 .

KOLLER C , SHAHRAD M , ZHU X X . Uncertainty-guided representation learning in local climate zone classification [C ] // IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium . Piscataway : IEEE , 2022 : 183 - 186 .

XU Y H , BAI T , YU W K , et al . AI security for geoscience and remote sensing: Challenges and future trends [J ] . IEEE Geoscience and Remote Sensing Magazine , 2023 , 11 ( 2 ): 60 - 85 .

ZINNER T , HAUSMANN P , EWALD F , et al . Ground-based imaging remote sensing of ice clouds: Uncertainties caused by sensor, method and atmosphere [J ] . Atmospheric Measurement Techniques , 2016 , 9 ( 9 ): 4615 - 4632 .

HAN Z B , ZHANG C Q , FU H Z , et al . Trusted multi-view classification with dynamic evidential fusion [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 2 ): 2551 - 2566 .

HAN Z B , YANG F , HUANG J Z , et al . Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2022 : 20675 - 20685 .

HAN Z B , YANG Y F , ZHANG C Q , et al . Selective learning: Towards robust calibration with dynamic regularization [EB/OL ] . ( 2024-07-14 )[ 2025-11-10 ] . https://arXiv.org/abs/2402.08384 https://arXiv.org/abs/2402.08384 .

RUSSELL R L , REALE C . Multivariate uncertainty in deep learning [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2022 , 33 ( 12 ): 7937 - 7943 .

TYRALIS H , PAPACHARALAMPOUS G . A review of predictive uncertainty estimation with machine learning [J ] . Artificial Intelligence Review , 2024 , 57 ( 4 ): 94 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于可疑像素相互修正的半监督医学图像分割

旁观者视角下粒的多层次描述

基于不确定性与错误分类率博弈的序贯三支决策模型

基于多分支结构的不确定性局部通道注意力机制