基于Dempster-Shafer证据推理的EEG-fNIRS运动想象分类决策层融合方法

康冉斓; 李玉榕; 史武翔; 李吉祥

doi:10.12263/DZXB.20240885

您当前的位置：

首页 >

文章列表页 >

基于Dempster-Shafer证据推理的EEG-fNIRS运动想象分类决策层融合方法

学术论文 | 更新时间：2025-12-08

- 基于Dempster-Shafer证据推理的EEG-fNIRS运动想象分类决策层融合方法
- Decision-Level Fusion of EEG-fNIRS for Motor Imagery Classification Based on Dempster-Shafer Evidence Reasoning
- 电子学报 2025年53卷第3期页码：941-950
- 作者机构：
  
  1.福州大学电气工程与自动化学院，福建福州 350108
  2.福建省医疗器械和医药技术重点实验室，福建福州 350108
- 作者简介：
  
  [ "康冉斓女，2001年10月生，湖南娄底人.现为福州大学硕士研究生.主要研究方向为基于EEG-fNIRS多模态融合的运动想象型脑机接口技术.E-mail: kranlan2001@163.com" ]
  [ "李玉榕女，1973年2月生，福建莆田人.现为福州大学电气工程与自动化学院教授、博士生导师.主要研究方向为多模态电生理信号建模与智能康复技术.E-mail: liyurong@fzu.edu.cn" ]
  [ "史武翔男，1995年10月生，河南焦作人.于2021年获得福州大学工程学硕士学位，现为福州大学博士研究生.主要研究方向为生物医学信号处理以及深度学习.E-mail: shiwuxiang@foxmail.com" ]
  [ "李吉祥男，1990年2月生，河南太康人.2018年毕业于郑州轻工业学院控制理论与控制工程专业，现为福州大学博士研究生.主要研究方向为运动想象脑机接口技术.E-mail: 1490758827@qq.com" ]
- 基金信息：
  
  国家自然科学基金(62373108)
- DOI：10.12263/DZXB.20240885
  中图分类号： TP391;
- 收稿：2024-10-07，
  
  修回：2025-02-20，
  
  纸质出版：2025-03-25
- 稿件说明：
移动端阅览
康冉斓, 李玉榕, 史武翔, 等. 基于Dempster-Shafer证据推理的EEG-fNIRS运动想象分类决策层融合方法[J]. 电子学报, 2025, 53(03): 941-950.

KANG Ran-lan, LI Yu-rong, SHI Wu-xiang, et al. Decision-Level Fusion of EEG-fNIRS for Motor Imagery Classification Based on Dempster-Shafer Evidence Reasoning[J]. Acta Electronica Sinica, 2025, 53(03): 941-950.
康冉斓, 李玉榕, 史武翔, 等. 基于Dempster-Shafer证据推理的EEG-fNIRS运动想象分类决策层融合方法[J]. 电子学报, 2025, 53(03): 941-950. DOI：10.12263/DZXB.20240885

KANG Ran-lan, LI Yu-rong, SHI Wu-xiang, et al. Decision-Level Fusion of EEG-fNIRS for Motor Imagery Classification Based on Dempster-Shafer Evidence Reasoning[J]. Acta Electronica Sinica, 2025, 53(03): 941-950. DOI：10.12263/DZXB.20240885

摘要

为解决传统基于脑电信号（ElectroencEphaloGraphy，EEG）的单模态脑机接口（Brain-Computer Interface，BCI）技术存在的空间分辨率低、易受噪声干扰等问题，越来越多的研究开始关注基于EEG信号和功能近红外光谱（functional Near-InfRared Spectroscopy，fNIRS）信号融合的BCI研究.然而，这两种异构信号之间的融合具有挑战性，本文创新性地提出一种基于深度学习和证据理论的端对端信号融合方法，用于运动想象（Motor Imagery，MI）分类.对于EEG信号，本文通过双尺度时间卷积和深度可分离卷积提取其时空特征信息，并引入混合注意力模块以增强网络对重要特征的感知能力.对于fNIRS信号，本文通过全通道的空间卷积探索大脑不同区域之间的激活差异，并通过并联时间卷积和门控循环单元（Gated Recurrent Unit，GRU）模块捕获更丰富的时间特征信息.在决策融合阶段，首先将两种信号分别解码得到的决策输出利用Dirichlet分布参数估计，以量化不确定性；然后使用Dempster-Shafer理论（Dempster-Shafer Theory，DST）进行双层推理，从而融合来自两种基本信念分配（Basic Belief Assignment，BBA）方法和不同模态的证据，得到最终的分类结果.本文基于公开数据集TU-Berlin-A进行模型的测试评估，获得了83.26%的平均准确率，相较于最先进研究提升了3.78个百分点，该结果为基于EEG和fNIRS信号的融合研究提供了新的思路和方法.

Abstract

To address the issues of low spatial resolution and susceptibility to noise in traditional single-modality brain-computer interface (BCI) technologies based on electroencephalography (EEG)

an increasing number of studies have focused on BCI research that combines EEG signals with functional near-infrared spectroscopy (fNIRS) signals. However

integrating these two heterogeneous signals poses challenges. This paper proposes an innovative end-to-end signal fusion method based on deep learning and evidence theory for motor imagery (MI) classification. The spatiotemporal feature information of EEG signals is extracted using dual-scale temporal convolution and depth wise separable convolution

with a hybrid attention module introduced to enhance the network’s ability to perceive important features. For fNIRS signals

spatial convolution across all channels explores activation differences between different brain regions

while parallel temporal convolution and gated recurrent unit (GRU) capture richer temporal feature information. During the decision fusion stage

the decision outputs obtained from decoding each signal are first utilized to estimate uncertainty using Dirichlet distribution parameter estimation. Subsequently

Dempster-Shafer theory (DST) is employed for dual-layer reasoning

effectively merging evidence from the two basic belief assignment (BBA) methods and different modalities to obtain the decoding results. The proposed model is evaluated on the publicly available TU-Berlin-A dataset

achieving an average accuracy of 83.26%

which represents a 3.78 percentage points improvement compared to the state-of-the-art research. This provides new ideas and approaches for fusion studies based on EEG and fNIRS signals.

关键词

Keywords

references

WOLPAW J R , BIRBAUMER N , HEETDERKS W J , et al . Brain-computer interface technology: A review of the first international meeting [J ] . IEEE Transactions on Rehabilitation Engineering , 2000 , 8 ( 2 ): 164 - 173 .

SCHALK G , MCFARLAND D J , HINTERBERGER T , et al . BCI2000: A general-purpose brain-computer interface (BCI) system [J ] . IEEE Transactions on Bio-Medical Engineering , 2004 , 51 ( 6 ): 1034 - 1043 .

AL-QAYSI Z T , ZAIDAN B B , ZAIDAN A A , et al . A review of disability EEG based wheelchair control system: Coherent taxonomy, open challenges and recommendations [J ] . Computer Methods and Programs in Biomedicine , 2018 , 164 : 221 - 237 .

JEONG J H , SHIM K H , KIM D J , et al . Brain-controlled robotic arm system based on multi-directional CNN-BiLSTM network using EEG signals [J ] . IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2020 , 28 ( 5 ): 1226 - 1238 .

MENG J J , ZHANG S Y , BEKYO A , et al . Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks [J ] . Scientific Reports , 2016 , 6 : 38565 .

HASSAN M , WENDLING F . Electroencephalography source connectivity: Aiming for high resolution of brain networks in time and space [J ] . IEEE Signal Processing Magazine , 2018 , 35 ( 3 ): 81 - 96 .

SAWANGJAI P , HOMPOONSUP S , LEELAARPORN P , et al . Consumer grade EEG measuring sensors as research tools: A review [J ] . IEEE Sensors Journal , 2020 , 20 ( 8 ): 3996 - 4024 .

QUARESIMA V , FERRARI M . Functional near-infrared spectroscopy (fNIRS) for assessing cerebral cortex function during human behavior in natural/social situations: A concise review [J ] . Organizational Research Methods , 2019 , 22 ( 1 ): 46 - 68 .

BUCCINO A P , KELES H O , OMURTAG A . Hybrid EEG-fNIRS asynchronous brain-computer interface for multiple motor tasks [J ] . PLoS One , 2016 , 11 ( 1 ): e0146610 .

REZAZADEH SERESHKEH A , YOUSEFI R , WONG A T , et al . Development of a ternary hybrid fNIRS-EEG brain-computer interface based on imagined speech [J ] . Brain-Computer Interfaces , 2019 , 6 ( 4 ): 128 - 140 .

SHIN J , VON LUHMANN A , BLANKERTZ B , et al . Open access dataset for EEG+NIRS single-trial classification [J ] . IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2017 , 25 ( 10 ): 1735 - 1745 .

XU T , ZHOU Z K , YANG Y L , et al . Motor imagery decoding enhancement based on hybrid EEG-fNIRS signals [J ] . IEEE Access , 2023 , 11 : 65277 - 65288 .

JIANG X Y , GU X , XU K , et al . Independent decision path fusion for bimodal asynchronous brain-computer interface to discriminate multiclass mental states [J ] . IEEE Access , 2019 , 7 : 165303 - 165317 .

SUN Z , HUANG Z H , DUAN F , et al . A novel multimodal approach for hybrid brain-computer interface [J ] . IEEE Access , 2020 , 8 : 89909 - 89918 .

HE Q , FENG L F , JIANG G Q , et al . Multimodal multitask neural network for motor imagery classification with EEG and fNIRS signals [J ] . IEEE Sensors Journal , 2022 , 22 ( 21 ): 20695 - 20706 .

WU N , JASTRZĘBSKI S , CHO K , et al . Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks [EB/OL ] . ( 2022-09-16 )[ 2025-03-11 ] . https://arxiv.org/abs/2202.05306v3 https://arxiv.org/abs/2202.05306v3 .

GAO Y Y , JIA B , HOUSTON M , et al . Hybrid EEG-fNIRS brain computer interface based on common spatial pattern by using EEG-informed general linear model [J ] . IEEE Transactions on Instrumentation and Measurement , 2023 , 72 : 4006110 .

SUN D D , WANG M H , LI A . A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data [J ] . IEEE/ACM Transactions on Computational Biology and Bioinformatics , 2018 , 16 ( 3 ): 841 - 850 .

NODA K , ARIE H , SUGA Y , et al . Multimodal integration learning of robot behavior using deep neural networks [J ] . Robotics and Autonomous Systems , 2014 , 62 ( 6 ): 721 - 736 .

TZIRAKIS P , TRIGEORGIS G , NICOLAOU M A , et al . End-to-end multimodal emotion recognition using deep neural networks [J ] . IEEE Journal of Selected Topics in Signal Processing , 2017 , 11 ( 8 ): 1301 - 1309 .

LI Y , ZHANG X , MING D . Early-stage fusion of EEG and fNIRS improves classification of motor imagery [J ] . Frontiers in Neuroscience , 2023 , 16 : 1062889 .

KWAK Y , SONG W J , KIM S E . FGANet: FNIRS-guided attention network for hybrid EEG-fNIRS brain-computer interfaces [J ] . IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2022 , 30 : 329 - 339 .

SHAFER G . Dempster-shafer theory [J ] . Encyclopedia of Artificial Intelligence , 1992 , 1 : 330 - 331 .

DENOEUX T . A neural network classifier based on Dempster-Shafer theory [J ] . IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans , 2000 , 30 ( 2 ): 131 - 150 .

TONG Z , XU P , DENŒUX T . An evidential classifier based on Dempster-Shafer theory and deep learning [J ] . Neurocomputing , 2021 , 450 : 275 - 293 .

LAWHERN V J , SOLON A J , WAYTOWICH N R , et al . EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces [J ] . Journal of Neural Engineering , 2018 , 15 ( 5 ): 056013 .

HU J , SHEN L , SUN G . Squeeze-and-excitation networks [C ] // 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 7132 - 7141 .

WANG Z H , FANG J H , ZHANG J . Rethinking delayed hemodynamic responses for fNIRS classification [J ] . IEEE Transactions on Neural Systems and Rehabilitation Engineering , 2023 , 31 : 4528 - 4538 .

SHAO Z M , DOU W B , PAN Y . Dual-level deep evidential fusion: Integrating multimodal information for enhanced reliable decision-making in deep learning [J ] . Information Fusion , 2024 , 103 : 102113 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

面向时序异常检测的可变视距多向扫描方法

基于稀疏平滑自蒸馏的差分隐私深度学习方法

基于非一般类算子融合方法及硬件架构设计

基于注意力融合多尺度特征的解压缩点云质量增强方法

基于深度压缩感知的联合信源信道编码方法研究