动态场景下深度自监督多曝光图像融合方法

张雨童; 邓欣; 徐迈

doi:10.12263/DZXB.20220893

您当前的位置：

首页 >

文章列表页 >

动态场景下深度自监督多曝光图像融合方法

学术论文 | 更新时间：2026-04-10

- 动态场景下深度自监督多曝光图像融合方法
- Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes
- 电子学报 2024年52卷第1期页码：264-273
- 作者机构：
  
  1.北京航空航天大学电子信息工程学院，北京 100191
  2.北京航空航天大学网络空间安全学院，北京 100191
- 作者简介：
  
  [ "张雨童男，1999年2月生，四川成都人.2020年在北京航空航天大学获得学士学位.现为北京航空航天大学硕士研究生.主要研究方向为深度学习和图像融合. E-mail: yutongzhang@buaa.edu.cn" ]
  [ "邓欣女，1991年1月生，山东威海人.博士毕业于英国伦敦帝国理工学院获博士学位.现为北京航空航天大学网络空间安全学院副研究员.主要研究方向为多模态图像处理和可解释神经网络.E-mail: cindydeng@buaa.edu.cn" ]
  [ "徐迈男，1981年2月生，江苏无锡人.博士毕业于英国伦敦帝国理工学院获博士学位.现为北京航空航天大学电子信息工程学院教授.主要研究方向为图像处理和人工智能.中国电子学会会员编号：E190014800S.E-mail: maixu@buaa.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(62001016)
- DOI：10.12263/DZXB.20220893
  中图分类号： TP391;
- 收稿：2022-07-28，
  
  修回：2022-11-08，
  
  纸质出版：2024-01-25
- 稿件说明：
移动端阅览
张雨童,邓欣,徐迈.动态场景下深度自监督多曝光图像融合方法[J].电子学报,2024,52(01):264-273.

ZHANG Yu-tong, DENG Xin, XU Mai.Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes[J].Acta Electronica Sinica, 2024, 52(01): 264-273.
张雨童,邓欣,徐迈.动态场景下深度自监督多曝光图像融合方法[J].电子学报,2024,52(01):264-273. DOI：10.12263/DZXB.20220893

ZHANG Yu-tong, DENG Xin, XU Mai.Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes[J].Acta Electronica Sinica, 2024, 52(01): 264-273. DOI：10.12263/DZXB.20220893

摘要

近年来，面向动态场景的多曝光图像融合技术取得重大进展.其中，基于深度学习的方法在视觉效果和运算效率上都远超传统算法，成为高动态范围成像技术的主流.然而，现有基于深度学习的融合方法都以有监督学习的方式实现，过度依赖真值图像，难以被广泛应用于实际场景中.本文提出了一个基于深度自监督学习的动态多曝光图像融合网络，主要贡献包括：设计自监督的动态多曝光融合网络框架，探索高动态范围图像与低动态范围图像序列的内在关联；提出基于注意力机制的全局去伪影模块，使用全局文本模块减少动态融合产生的运动伪影，增强图像细节；提出融合重建模块，通过残差和稠密连接实现多层次特征之间的信息流动；设计运动掩膜引导的自监督损失函数，用于网络的高效训练.实验表明，与现有方法相比，本文提出的方法在高动态范围图像重建的主观和客观质量上均表现较好，运算效率显著提升.

Abstract

In recent years

significant progress has been made in multi-exposure image fusion in dynamic scenes. In particular

the deep learning based methods have shown great visual performance in dynamic multi-exposure image fusion

which have become the mainstream methods in high dynamic range (HDR) imaging. However

the current deep learning based methods are mostly implemented in a supervised manner

which heavily rely on the ground-truth images. That makes it difficult for them to work in real scenes. In this paper

we propose a self-supervised multi-exposure image fusion network for dynamic scenes. The main contributions of this paper are as follows: we design a self-supervised fusion network to explore the latent relationship between HDR and low dynamic range (LDR) images; we propose an attention mechanism based global deghosting module

to reduce the ghosting artifacts caused by moving objects; we propose a merging reconstruction module with residual and dense connections

to improve the reconstruction details; we design a motion mask guided self-supervised loss function to train the proposed network efficiently. Experimental results demonstrate the effectiveness of the proposed method. Compared with the state-of-the-art methods

our method achieves higher objective and subjective quality on reconstructed HDR images

with faster running speed.

关键词

Keywords

references

MA K D , ZENG K , WANG Z . Perceptual quality assessment for multi-exposure image fusion [J]. IEEE Transactions on Image Processing , 2015 , 24 ( 11 ): 3345 - 3356 .

PRABHAKAR K R , SRIKAR V S , BABU R V . DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs [C]// 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2017 : 4724 - 4732 .

DENG X , DRAGOTTI P L . Deep convolutional neural network for multi-modal image restoration and fusion [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021 , 43 ( 10 ): 3333 - 3348 .

DENG X , ZHANG Y T , XU M , et al . Deep coupled feedback network for joint exposure fusion and image super-resolution [J]. IEEE Transactions on Image Processing , 2021 , 30 : 3098 - 3112 .

BOGONI L . Extending dynamic range of monochrome and color images through fusion [C]// Proceedings 15th International Conference on Pattern Recognition . Piscataway : IEEE , 2002 : 7 - 12 .

KANG S B , UYTTENDAELE M , WINDER S , et al . High dynamic range video [J]. ACM Transactions on Graphics , 22 ( 3 ): 319 - 325 .

JINNO T , OKUDA M . Motion blur free HDR image acquisition using multiple exposures [C]// 2008 15th IEEE International Conference on Image Processing . Piscataway : IEEE , 2008 : 1304 - 1307 .

ZIMMER H , BRUHN A , WEICKERT J . Freehand HDR imaging of moving scenes with simultaneous resolution enhancement [J]. Computer Graphics Forum , 2011 , 30 ( 2 ): 405 - 414 .

SEN P , KALANTARI N K , YAESOUBI M , et al . Robust patch-based HDR reconstruction of dynamic scenes [J]. ACM Transactions on Graphics , 31 ( 6 ): 203 .

HU J , GALLO O , PULLI K , et al . HDR deghosting: How to deal with saturation? [C]// 2013 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2013 : 1163 - 1170 .

GALLO O , GELFANDZ N , CHEN W C , et al . Artifact-free high dynamic range imaging [C]// 2009 IEEE International Conference on Computational Photography (ICCP) . Piscataway : IEEE , 2009 : 1 - 7 .

OH T H , LEE J Y , TAI Y W , et al . Robust high dynamic range imaging by rank minimization [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2015 , 37 ( 6 ): 1219 - 1232 .

MA K D , LI H , YONG H W , et al . Robust multi-exposure image fusion: A structural patch decomposition approach [J]. IEEE Transactions on Image Processing , 2017 , 26 ( 5 ): 2519 - 2532 .

LI H , MA K D , YONG H W , et al . Fast multi-scale structural patch decomposition for multi-exposure image fusion [J]. IEEE Transactions on Image Processing , 2020 : 32310768 .

LI H , CHAN T N , QI X B , et al . Detail-preserving multi-exposure fusion with edge-preserving structural patch decomposition [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 31 ( 11 ): 4293 - 4304 .

KALANTARI N K , RAMAMOORTHI R . Deep high dynamic range imaging of dynamic scenes [J]. ACM Transactions on Graphics , 36 ( 4 ): 144 .

WU S Z , XU J R , TAI Y W , et al . Deep high dynamic range imaging with large foreground motions [C]// Computer Vision - ECCV 2018: 15th European Conference . New York : ACM , 2018 : 120 - 135 .

PRABHAKAR K R , ARORA R , SWAMINATHAN A , et al . A fast, scalable, and reliable deghosting method for extreme exposure fusion [C]// 2019 IEEE International Conference on Computational Photography (ICCP) . Piscataway : IEEE , 2019 : 1 - 8 .

YAN Q S , GONG D , SHI Q F , et al . Attention-guided network for ghost-free high dynamic range imaging [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 1751 - 1760 .

YAN Q S , ZHANG L , LIU Y , et al . Deep HDR imaging via a non-local network [J]. IEEE Transactions on Image Processing , 2020 , 29 : 4308 - 4322 .

LIU Z , LIN W J , LI X P , et al . ADNet: Attention-guided deformable convolutional network for high dynamic range imaging [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE , 2021 : 463 - 470 .

YAN Q S , GONG D , SHI J Q , et al . Dual-attention-guided network for ghost-free high dynamic range imaging [J]. International Journal of Computer Vision , 2022 , 130 ( 1 ): 76 - 94 .

PRABHAKAR K R , SENTHIL G , AGRAWAL S , et al . Labeled from unlabeled: Exploiting unlabeled data for few-shot deep HDR deghosting [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 4873 - 4883 .

SUN D Q , YANG X D , LIU M Y , et al . PWC-net: CNNs for optical flow using pyramid, warping, and cost volume [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 8934 - 8943 .

GROSSBERG M D , NAYAR S K . Determining the camera response from images: What is knowable? [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2003 , 25 ( 11 ): 1455 - 1467 .

CAO Y , XU J R , LIN S , et al . GCNet: Non-local networks meet squeeze-excitation networks and beyond [C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway : IEEE , 2019 : 1971 - 1980 .

ZHANG Y L , TIAN Y P , KONG Y , et al . Residual dense network for image super-resolution [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 2472 - 2481 .

WANG X L , GIRSHICK R , GUPTA A , et al . Non-local neural networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 7794 - 7803 .

HU J , SHEN L , SUN G . Squeeze-and-excitation networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 7132 - 7141 .

TSAI D Y , LEE Y , MATSUYAMA E . Information entropy measure for evaluation of image quality [J]. Journal of Digital Imaging , 2008 , 21 ( 3 ): 338 - 347 .

GU K , WANG S Q , ZHAI G T , et al . Blind quality assessment of tone-mapped images via analysis of information, naturalness, and structure [J]. IEEE Transactions on Multimedia , 2016 , 18 ( 3 ): 432 - 443 .

FANG Y M , ZHU H W , MA K D , et al . Perceptual evaluation for multi-exposure image fusion of dynamic scenes [J]. IEEE Transactions on Image Processing , 2019 : 31535996 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于无标签视频数据的深度预测学习方法综述

基于红外偏振成像的复杂干扰环境下无人机目标检测方法

基于视觉与深度学习的无人机自主着陆场景感知方法研究进展

融合动作描述生成与跨模态语义对齐的骨架动作识别方法