Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes

ZHANG Yu-tong; DENG Xin; XU Mai

doi:10.12263/DZXB.20220893

您当前的位置：

首页 >

文章列表页 >

Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes

PAPERS | 更新时间：2026-04-10

- Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes
- ACTA ELECTRONICA SINICA Vol. 52, Issue 1, Pages: 264-273(2024)
- 作者机构：
  
  1.北京航空航天大学电子信息工程学院，北京 100191
  2.北京航空航天大学网络空间安全学院，北京 100191
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62001016)
- DOI：10.12263/DZXB.20220893
  CLC： TP391;
- Received：28 July 2022，
  
  Revised：2022-11-08，
  
  Published：25 January 2024
- 稿件说明：
移动端阅览
张雨童,邓欣,徐迈.动态场景下深度自监督多曝光图像融合方法[J].电子学报,2024,52(01):264-273.

ZHANG Yu-tong, DENG Xin, XU Mai.Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes[J].Acta Electronica Sinica, 2024, 52(01): 264-273.
张雨童,邓欣,徐迈.动态场景下深度自监督多曝光图像融合方法[J].电子学报,2024,52(01):264-273. DOI：10.12263/DZXB.20220893

ZHANG Yu-tong, DENG Xin, XU Mai.Deep Self-Supervised Multi-Exposure Image Fusion for Dynamic Scenes[J].Acta Electronica Sinica, 2024, 52(01): 264-273. DOI：10.12263/DZXB.20220893

摘要

近年来，面向动态场景的多曝光图像融合技术取得重大进展.其中，基于深度学习的方法在视觉效果和运算效率上都远超传统算法，成为高动态范围成像技术的主流.然而，现有基于深度学习的融合方法都以有监督学习的方式实现，过度依赖真值图像，难以被广泛应用于实际场景中.本文提出了一个基于深度自监督学习的动态多曝光图像融合网络，主要贡献包括：设计自监督的动态多曝光融合网络框架，探索高动态范围图像与低动态范围图像序列的内在关联；提出基于注意力机制的全局去伪影模块，使用全局文本模块减少动态融合产生的运动伪影，增强图像细节；提出融合重建模块，通过残差和稠密连接实现多层次特征之间的信息流动；设计运动掩膜引导的自监督损失函数，用于网络的高效训练.实验表明，与现有方法相比，本文提出的方法在高动态范围图像重建的主观和客观质量上均表现较好，运算效率显著提升.

Abstract

In recent years

significant progress has been made in multi-exposure image fusion in dynamic scenes. In particular

the deep learning based methods have shown great visual performance in dynamic multi-exposure image fusion

which have become the mainstream methods in high dynamic range (HDR) imaging. However

the current deep learning based methods are mostly implemented in a supervised manner

which heavily rely on the ground-truth images. That makes it difficult for them to work in real scenes. In this paper

we propose a self-supervised multi-exposure image fusion network for dynamic scenes. The main contributions of this paper are as follows: we design a self-supervised fusion network to explore the latent relationship between HDR and low dynamic range (LDR) images; we propose an attention mechanism based global deghosting module

to reduce the ghosting artifacts caused by moving objects; we propose a merging reconstruction module with residual and dense connections

to improve the reconstruction details; we design a motion mask guided self-supervised loss function to train the proposed network efficiently. Experimental results demonstrate the effectiveness of the proposed method. Compared with the state-of-the-art methods

our method achieves higher objective and subjective quality on reconstructed HDR images

with faster running speed.

关键词

Keywords

references

MA K D , ZENG K , WANG Z . Perceptual quality assessment for multi-exposure image fusion [J]. IEEE Transactions on Image Processing , 2015 , 24 ( 11 ): 3345 - 3356 .

PRABHAKAR K R , SRIKAR V S , BABU R V . DeepFuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs [C]// 2017 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2017 : 4724 - 4732 .

DENG X , DRAGOTTI P L . Deep convolutional neural network for multi-modal image restoration and fusion [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021 , 43 ( 10 ): 3333 - 3348 .

DENG X , ZHANG Y T , XU M , et al . Deep coupled feedback network for joint exposure fusion and image super-resolution [J]. IEEE Transactions on Image Processing , 2021 , 30 : 3098 - 3112 .

BOGONI L . Extending dynamic range of monochrome and color images through fusion [C]// Proceedings 15th International Conference on Pattern Recognition . Piscataway : IEEE , 2002 : 7 - 12 .

KANG S B , UYTTENDAELE M , WINDER S , et al . High dynamic range video [J]. ACM Transactions on Graphics , 22 ( 3 ): 319 - 325 .

JINNO T , OKUDA M . Motion blur free HDR image acquisition using multiple exposures [C]// 2008 15th IEEE International Conference on Image Processing . Piscataway : IEEE , 2008 : 1304 - 1307 .

ZIMMER H , BRUHN A , WEICKERT J . Freehand HDR imaging of moving scenes with simultaneous resolution enhancement [J]. Computer Graphics Forum , 2011 , 30 ( 2 ): 405 - 414 .

SEN P , KALANTARI N K , YAESOUBI M , et al . Robust patch-based HDR reconstruction of dynamic scenes [J]. ACM Transactions on Graphics , 31 ( 6 ): 203 .

HU J , GALLO O , PULLI K , et al . HDR deghosting: How to deal with saturation? [C]// 2013 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2013 : 1163 - 1170 .

GALLO O , GELFANDZ N , CHEN W C , et al . Artifact-free high dynamic range imaging [C]// 2009 IEEE International Conference on Computational Photography (ICCP) . Piscataway : IEEE , 2009 : 1 - 7 .

OH T H , LEE J Y , TAI Y W , et al . Robust high dynamic range imaging by rank minimization [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2015 , 37 ( 6 ): 1219 - 1232 .

MA K D , LI H , YONG H W , et al . Robust multi-exposure image fusion: A structural patch decomposition approach [J]. IEEE Transactions on Image Processing , 2017 , 26 ( 5 ): 2519 - 2532 .

LI H , MA K D , YONG H W , et al . Fast multi-scale structural patch decomposition for multi-exposure image fusion [J]. IEEE Transactions on Image Processing , 2020 : 32310768 .

LI H , CHAN T N , QI X B , et al . Detail-preserving multi-exposure fusion with edge-preserving structural patch decomposition [J]. IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 31 ( 11 ): 4293 - 4304 .

KALANTARI N K , RAMAMOORTHI R . Deep high dynamic range imaging of dynamic scenes [J]. ACM Transactions on Graphics , 36 ( 4 ): 144 .

WU S Z , XU J R , TAI Y W , et al . Deep high dynamic range imaging with large foreground motions [C]// Computer Vision - ECCV 2018: 15th European Conference . New York : ACM , 2018 : 120 - 135 .

PRABHAKAR K R , ARORA R , SWAMINATHAN A , et al . A fast, scalable, and reliable deghosting method for extreme exposure fusion [C]// 2019 IEEE International Conference on Computational Photography (ICCP) . Piscataway : IEEE , 2019 : 1 - 8 .

YAN Q S , GONG D , SHI Q F , et al . Attention-guided network for ghost-free high dynamic range imaging [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2019 : 1751 - 1760 .

YAN Q S , ZHANG L , LIU Y , et al . Deep HDR imaging via a non-local network [J]. IEEE Transactions on Image Processing , 2020 , 29 : 4308 - 4322 .

LIU Z , LIN W J , LI X P , et al . ADNet: Attention-guided deformable convolutional network for high dynamic range imaging [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE , 2021 : 463 - 470 .

YAN Q S , GONG D , SHI J Q , et al . Dual-attention-guided network for ghost-free high dynamic range imaging [J]. International Journal of Computer Vision , 2022 , 130 ( 1 ): 76 - 94 .

PRABHAKAR K R , SENTHIL G , AGRAWAL S , et al . Labeled from unlabeled: Exploiting unlabeled data for few-shot deep HDR deghosting [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2021 : 4873 - 4883 .

SUN D Q , YANG X D , LIU M Y , et al . PWC-net: CNNs for optical flow using pyramid, warping, and cost volume [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 8934 - 8943 .

GROSSBERG M D , NAYAR S K . Determining the camera response from images: What is knowable? [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2003 , 25 ( 11 ): 1455 - 1467 .

CAO Y , XU J R , LIN S , et al . GCNet: Non-local networks meet squeeze-excitation networks and beyond [C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) . Piscataway : IEEE , 2019 : 1971 - 1980 .

ZHANG Y L , TIAN Y P , KONG Y , et al . Residual dense network for image super-resolution [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 2472 - 2481 .

WANG X L , GIRSHICK R , GUPTA A , et al . Non-local neural networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 7794 - 7803 .

HU J , SHEN L , SUN G . Squeeze-and-excitation networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2018 : 7132 - 7141 .

TSAI D Y , LEE Y , MATSUYAMA E . Information entropy measure for evaluation of image quality [J]. Journal of Digital Imaging , 2008 , 21 ( 3 ): 338 - 347 .

GU K , WANG S Q , ZHAI G T , et al . Blind quality assessment of tone-mapped images via analysis of information, naturalness, and structure [J]. IEEE Transactions on Multimedia , 2016 , 18 ( 3 ): 432 - 443 .

FANG Y M , ZHU H W , MA K D , et al . Perceptual evaluation for multi-exposure image fusion of dynamic scenes [J]. IEEE Transactions on Image Processing , 2019 : 31535996 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A Survey on Deep Predictive Learning Based on Unlabeled Videos

An Infrared Polarization Based UAV Detection Method for Complex Environment

Research Progress of UAV Autonomous Landing Scene Perception Methods Based on Vision and Deep Learning

Leveraging Action Description Generation and Cross-Modal Semantic Alignment for Skeleton-Based Action Recognition

Related Author

PAN Min-ting

WANG Yun-bo

ZHU Xiang-ming

GAO Si-yu

LONG Ming-sheng

YANG Xiao-kang

QIAO Xin-bo

GUO Yang

Related Institution

MoE Key laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University

School of Software, Tsinghua University

School of Automation, Northwestern Polytechnical University

Naval Aeronautical University

College of Electronic and Information Engineering, Nanjing University of Aeronautics and Astronautics

⁰