基于景深先验引导与环境光优化的图像去雾

doi:10.12263/DZXB.20201127

摘要/Abstract

摘要：

针对暗通道先验存在的边缘去雾不彻底及细节信息丢失等问题，提出了一种景深引导网络（Depth Guided Network，DGN）与环境光优化的图像去雾方法.首先，通过DGN中的图像去模糊分支与景深先验信息将有雾图像复原为初始清晰图像，同时提取图像中的景深信息，根据DGN中的景深细调网络恢复出景深图的边界与结构；然后，为了使细调后景深信息转化为图像特征，利用空间特征变换（Spatial Feature Transform，SFT）层将图像进行迭代变换，同时结合缩放及平移与初始清晰图像进行特征融合，根据景深引导将初始清晰图像进行优化；最后，为了进一步使复原图像具有适宜色彩与较高对比度，将动态环境光细化后用于图像去雾，进一步优化图像细节信息.实验表明：从主观对比来看，得到的复原图像平滑性较好，动态环境光可以增强图像的细节信息，得到对比度较高的图像；从客观对比分析来看，本文方法对合成有雾图像处理后的结构相似性与峰值信噪比的均值分别为88.78%和22.98 dB，对真实有雾图像处理后的细节强度与色彩还原度的均值分别为0.436 8和0.794.综合对比其他去雾方法，本文方法在性能指标最优的同时运行时间最短，能够适用于复杂环境场景的实时性去雾，且具有较好的鲁棒性.

关键词: 景深引导网络, 去模糊分支, 细调分支, 空间特征变换, 动态环境光, 图像去雾

Abstract:

Under the dark channel prior, there are incomplete edge defogs and loss of detailed information. Therefore, an image depthing method based on a depth guided network(DGN) and optimization of ambient light is proposed. First, the foggy image was restored to the initial clear image by using the image deblurring branch and depth of field prior information in the DGN. Meanwhile, the depth of field information in the image was extracted. The depth-of-field fine-tuning network in DGN was used to restore the boundaries and structures in the depth-of-field map. Then, the spatial feature transform(SFT) layer was used to transform the image iteratively. The purpose of transforming the fine-tuned depth information into image features was achieved. Moreover, zooming, panning, and initial clear image features were merged. Depth of field guidance was used to optimize the original image. Finally, the dynamic ambient light was refined and used for image defogging. The detailed information of the image was further optimized. The purpose of the restored image with suitable color and higher contrast was achieved. Experiments show that the smoothness of the restored image obtained from the subjective comparison is better. The dynamic ambient light optimization can enhance the detailed information of the image. Images with better contrast can be obtained. The mean value of SSIM and PSNR obtained by synthesizing foggy images is 88.78% and 22.98 dB, respectively. The average value of the detail intensity and the color reproduction obtained from the real foggy image are 0.436 8 and 0.794, respectively. When compared with other defogging methods, it has the best performance index and the shortest time. Furthermore, its advantage lies in better real-time defogging. Moreover, it has good robustness.

Key words: depth guided network, image deblurring branch, depth refinement branch, spatial feature transform, dynamic ambient light, image dehazing

中图分类号:

TP391.4

麻文刚, 张亚东, 郭进. 基于景深先验引导与环境光优化的图像去雾[J]. 电子学报, 2022, 50(7): 1708-1721.

MA Wen-gang, ZHANG Ya-dong, GUO Jin. Image Dehazing Based on Priori Guidance of Depth of Field and Optimization of Ambient Light[J]. Acta Electronica Sinica, 2022, 50(7): 1708-1721.

图/表 23

图1 本文方法去雾流程

图2 景深引导的DGN结构图

表1 景深细调分支

不同网络层	输入大小	输出大小	核大小	步长	填充	求和
Conv1	1	32	7	1	3	—
SC1	32	64	3	2	1	—
SC2	64	128	3	2	1	—
Conv2	32	32	7	1	3	—
SFT	32	32	—	—	—	—
Res2~4	32	32	—	—	—	—
Res6~8	64	64	—	—	—	—
Res10~15	128	128	—	—	—	—
Res17~19	64	64	—	—	—	—
Res21~23	32	32	—	—	—	—
TC1	128	64	4	2	1	Res8
TC2	64	32	4	2	1	Res4
Tanh	1	1	—	—	—	Input

表2 去模糊分支

不同网络层	输入大小	输出大小	核大小	步长	填充	求和
Conv1	3	64	7	1	3	—
SC1	64	128	3	2	1	—
SC2	128	256	3	2	1	—
Conv2	64	3	7	1	3	—
SFT	64	64	—	—	—	—
Res2~4	64	64	—	—	—	—
Res6~8	128	128	—	—	—	—
Res10~15	256	256	—	—	—	—
Res17~19	128	128	—	—	—	—
Res21~23	64	64	—	—	—	—
TC1	256	128	4	2	1	Res8
TC2	128	64	4	2	1	Res4
Tanh	3	3	—	—	—	Input

图3 空间特征变换(SFT)结构图

表3 损失函数有效性分析

损失函数	PSNR/dB(↑)	SSIM/%(↑)
无 $ϖ a$	16.48	79.64
无 $ϖ b$	17.69	81.26
无 $ϖ c$	19.81	83.93
本文方法	24.69	86.67

表3 损失函数有效性分析

损失函数	PSNR/dB(↑)	SSIM/%(↑)
无 $ϖ a$	16.48	79.64
无 $ϖ b$	17.69	81.26
无 $ϖ c$	19.81	83.93
本文方法	24.69	86.67

表4 损失函数权重有效性分析

权重组合	PSNR/dB(↑)	SSIM/%(↑)
$δ a$ =0.01, $δ b$ =0.01, $δ c$ =1	22.17	82.17
$δ a$ =1, $δ b$ =1, $δ c$ =1	20.42	83.59
$δ a$ =1, $δ b$ =0.01, $δ c$ =0.01	21.38	81.78
本文权重组合	24.69	86.67

表4 损失函数权重有效性分析

权重组合	PSNR/dB(↑)	SSIM/%(↑)
$δ a$ =0.01, $δ b$ =0.01, $δ c$ =1	22.17	82.17
$δ a$ =1, $δ b$ =1, $δ c$ =1	20.42	83.59
$δ a$ =1, $δ b$ =0.01, $δ c$ =0.01	21.38	81.78
本文权重组合	24.69	86.67

图4 不同ρ值的图像处理效果对比

图5 不同增益系数下的去雾效果对比(a) 有雾图像 (b) k=1去雾效果 (c) k=2去雾效果

图6 不同分割值的平滑损失值对比

图7 环境光优化效果对比(a) 有雾图像 (b) 未使用环境光 (c) 粗略环境光 (d) 细化环境光 (e) 复原图像

图8 有雾图像

图9 本文方法去雾效果

图10 去雾过程效果对比图(a) 原图 (b) 去模糊优化 (c) (b) +细调优化 (d) (c)+粗略动态环境光 (e) (d) +细化动态环境光

图11 合成景物1(去雾效果对比)(a) 原图 (b) He方法 (c) Gibson方法 (d) Cai方法 (e) 文献[18]方法 (f) 文献[19]方法 (g) 本文方法

图12 合成景物2(远处建筑景物)(a) 原图 (b) He方法 (c) Gibson方法 (d) Cai方法 (e) 文献[18]方法 (f) 文献[19]方法 (g) 本文方法

表5 合成有雾图像的PSNR指标对比/dB(↑)

Image	He方法	Gibson方法	Cai方法	文献[18]方法	文献[19]方法	本文方法
1	14.615 0	15.236 5	17.742 1	19.325 4	20.369 8	20.236 5
2	16.325 4	18.965 7	19.658 7	20.365 8	21.468 8	21.325 4
3	20.365 8	20.658 7	21.589 7	22.354 7	22.698 7	23.658 7
4	19.647 8	20.674 8	21.695 2	22.968 7	23.654 7	24.365 4
5	21.657 4	21.968 9	22.639 7	23.658 7	23.962 2	24.654 7
6	20.365 4	20.365 8	23.648 9	22.369 4	25.365 1	23.643 2

表6 合成有雾图像的SSIM值对比/%(↑)

Image	He 方法	Gibson 方法	Cai 方法	文献[18] 方法	文献[19] 方法	本文方法
1	75.16	77.36	79.45	79.45	78.58	82.58
2	83.23	84.32	80.56	81.16	85.29	88.39
3	84.65	84.78	86.78	79.78	82.69	89.63
4	79.36	80.56	83.52	85.38	86.38	91.45
5	85.32	86.42	88.23	87.74	87.89	94.26
6	86.34	87.62	87.51	88.63	89.63	86.35

图13 真实景物1(近景景物)(a) 原图 (b) He方法 (c) Gibson方法 (d) Cai方法 (e) 文献[18]方法 (f) 文献[19]方法 (g) 本文方法

图14 真实景物2(远景景物)(a) 原图 (b) He方法 (c) Gibson方法 (d) Cai方法 (e) 文献[18]方法 (f) 文献[19]方法 (g) 本文方法

表7 真实有雾图像的实验结果数据分析

Image	He方法		Gibson方法		Cai方法		文献[18]方法		文献[19]方法		本文方法
Image	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$
1	0.345 8	0.654	0.394 5	0.697	0.412 2	0.712	0.446 5	0.728	0.465 8	0.719	0.485 2	0.743
2	0.258 7	0.528	0.276 8	0.612	0.303 4	0.687	0.345 1	0.712	0.377 4	0.725	0.401 4	0.765
3	0.324 4	0.621	0.362 7	0.643	0.389 7	0.674	0.404 2	0.741	0.412 5	0.758	0.402 6	0.821
4	0.298 7	0.517	0.312 4	0.589	0.336 8	0.632	0.365 2	0.697	0.386 7	0.732	0.366 7	0.758
5	0.369 4	0.674	0.387 4	0.748	0.413 8	0.789	0.452 7	0.823	0.469 6	0.831	0.484 2	0.802
6	0.325 7	0.687	0.354 7	0.695	0.396 5	0.687	0.412 4	0.756	0.395 2	0.824	0.451 2	0.832
7	0.369 2	0.564	0.298 5	0.624	0.412 9	0.724	0.396 5	0.742	0.432 5	0.782	0.479 8	0.806
8	0.298 2	0.632	0.362 4	0.728	0.406 2	0.756	0.423 4	0.825	0.412 6	0.734	0.428 2	0.834
9	0.325 4	0.654	0.354 7	0.784	0.417 8	0.792	0.395 2	0.754	0.425 7	0.766	0.432 4	0.786

表7 真实有雾图像的实验结果数据分析

Image	He方法		Gibson方法		Cai方法		文献[18]方法		文献[19]方法		本文方法
Image	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$	$A ν$	$M (h, h *)$
1	0.345 8	0.654	0.394 5	0.697	0.412 2	0.712	0.446 5	0.728	0.465 8	0.719	0.485 2	0.743
2	0.258 7	0.528	0.276 8	0.612	0.303 4	0.687	0.345 1	0.712	0.377 4	0.725	0.401 4	0.765
3	0.324 4	0.621	0.362 7	0.643	0.389 7	0.674	0.404 2	0.741	0.412 5	0.758	0.402 6	0.821
4	0.298 7	0.517	0.312 4	0.589	0.336 8	0.632	0.365 2	0.697	0.386 7	0.732	0.366 7	0.758
5	0.369 4	0.674	0.387 4	0.748	0.413 8	0.789	0.452 7	0.823	0.469 6	0.831	0.484 2	0.802
6	0.325 7	0.687	0.354 7	0.695	0.396 5	0.687	0.412 4	0.756	0.395 2	0.824	0.451 2	0.832
7	0.369 2	0.564	0.298 5	0.624	0.412 9	0.724	0.396 5	0.742	0.432 5	0.782	0.479 8	0.806
8	0.298 2	0.632	0.362 4	0.728	0.406 2	0.756	0.423 4	0.825	0.412 6	0.734	0.428 2	0.834
9	0.325 4	0.654	0.354 7	0.784	0.417 8	0.792	0.395 2	0.754	0.425 7	0.766	0.432 4	0.786

图15 不同噪声等级下的复原能力对比

表8 各方法去雾的时间对比/s

Image	He方法	Gibson方法	Cai方法	文献[18]方法	文献[19]方法	本文方法
合成景物1	2.826 574	1.635 221	0.922 695	0.963 325	0.862 691	0.595 200
合成景物2	2.652 600	1.323 500	0.968 500	0.965 200	0.793 600	0.523 452
真实景物1	3.102 300	2.213 600	1.369 800	1.039 600	0.963 500	0.662 300
真实景物2	2.963 258	2.326 927	1.132 596	1.236 982	0.954 921	0.754 150

参考文献 25

1	嵇晓强, 戴明, 尹传历, 等. 航拍降质图像的去雾处理[J]. 光学精密工程, 2011, 19(7): 1659-1668.
	JI X Q, DAI M, YIN C L, et al. Haze removal for aerial degraded images[J]. Optics and Precision Engineering, 2011, 19(7): 1659-1668. (in Chinese)
2	李佳童, 章毓晋. 图像去雾算法的改进和主客观性能评价[J]. 光学精密工程, 2017, 25(3): 735-741.
	LI J T, ZHANG Y J. Improvements of image haze removal algorithm and its subjective and objective performance evaluation[J]. Optics and Precision Engineering, 2017, 25(3): 735-741. (in Chinese)
3	TAN R T. Visibility in bad weather from a single image[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage: IEEE, 2008: 1-8.
4	FATTAL R. Single image dehazing[J]. ACM Transactions on Graphics, 2008, 27(3): 1-9.
5	HE L Y, ZHAO J Z, ZHENG N N, et al. Haze removal using the difference-structure-preservation prior[J]. IEEE Transactions on Image Processing, 2017, 26(3): 1063-1075.
6	GIBSON K B, VÕ D T, NGUYEN T Q. An investigation of dehazing effects on image and video coding[J]. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2012, 21(2): 662-673.
7	刘冬梅, 常发亮. 基于非下采样轮廓小波变换增强的从粗到精的显著性检测[J]. 光学学报, 2019, 39(1): 0115003.
	LIU D M, CHANG F L. Coarse-to-fine saliency detection based on non-subsampled contourlet transform enhancement[J]. Acta Optica Sinica, 2019, 39(1): 0115003. (in Chinese)
8	陈永, 郭红光, 艾亚鹏. 基于双域分解的多尺度深度学习单幅图像去雾[J]. 光学学报, 2020, 40(2): 0210003.
	CHEN Y, GUO H G, AI Y P. Single image dehazing of multiscale deep-learning based on dual-domain decomposition[J]. Acta Optica Sinica, 2020, 40(2): 0210003. (in Chinese)
9	刘海波, 杨杰, 吴正平, 等. 基于暗通道先验和Retinex理论的快速单幅图像去雾方法[J]. 自动化学报, 2015, 41(7): 1264-1273.
	LIU H B, YANG J, WU Z P, et al. A fast single image dehazing method based on dark channel prior and retinex theory[J]. Acta Automatica Sinica, 2015, 41(7): 1264-1273. (in Chinese)
10	陈书贞, 任占广, 练秋生. 基于改进暗通道和导向滤波的单幅图像去雾算法[J]. 自动化学报, 2016, 42(3): 455-465.
	CHEN S Z, REN Z G, LIAN Q S. Single image dehazing algorithm based on improved dark channel prior and guided filter[J]. Acta Automatica Sinica, 2016, 42(3): 455-465. (in Chinese)
11	张小刚, 唐美玲, 陈华, 等. 一种结合双区域滤波和图像融合的单幅图像去雾算法[J]. 自动化学报, 2014, 40(8): 1733-1739.
	ZHANG X G, TANG M L, CHEN H, et al. A dehazing method in single image based on double-area filter and image fusion[J]. Acta Automatica Sinica, 2014, 40(8): 1733-1739. (in Chinese)
12	TAREL J P, HAUTIÈRE N. Fast visibility restoration from a single color or gray level image[C]//2009 IEEE 12th International Conference on Computer Vision. Kyoto: IEEE, 2009: 2201-2208.
13	何林远, 毕笃彦, 熊磊, 等. 基于亮度反馈的彩色雾霾图像增强算法[J]. 电子学报, 2015, 43(10): 1978-1983.
	HE L Y, BI D Y, XIONG L, et al. Color image haze removal algorithm based on luminance feedback[J]. Acta Electronica Sinica, 2015, 43(10): 1978-1983. (in Chinese)
14	孙伟, 李大健, 刘宏娟, 等. 基于大气散射模型的单幅图像快速去雾[J]. 光学精密工程, 2013, 21(4): 1040-1046.
	SUN W, LI D J, LIU H J, et al. Fast single image fog removal based on atmospheric scattering model[J]. Optics and Precision Engineering, 2013, 21(4): 1040-1046. (in Chinese)
15	TANG K T, YANG J C, WANG J. Investigating haze-relevant features in a learning framework for image dehazing[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus: IEEE, 2014: 2995-3002.
16	CAI B L, XU X M, JIA K, et al. DehazeNet: An end-to-end system for single image haze removal[J]. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2016, 25(11): 5187-5198.
17	王柯俨, 王迪, 赵熹, 等. 基于卷积神经网络的联合估计图像去雾算法[J]. 吉林大学学报(工学版), 2020, 50(5): 1771-1777.
	WANG K Y, WANG D, ZHAO X, et al. Image dehazing based on joint estimation via convolutional neural network[J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1771-1777. (in Chinese)
18	肖进胜, 周景龙, 雷俊锋, 等. 基于霾层学习的单幅图像去雾算法[J]. 电子学报, 2019, 47(10): 2142-2148.
	XIAO J S, ZHOU J L, LEI J F, et al. Single image dehazing algorithm based on the learning of hazy layers[J]. Acta Electronica Sinica, 2019, 47(10): 2142-2148. (in Chinese)
19	陈清江, 张雪. 基于并联卷积神经网络的图像去雾[J]. 自动化学报, 2021, 47(7): 1739-1748.
	CHEN Q J, ZHANG X. Single image dehazing based on multiple convolutional neural networks[J]. Acta Automatica Sinica, 2021, 47(7): 1739-1748. (in Chinese)
20	HUANG S C, CHEN B H, CHENG Y J. An efficient visibility enhancement algorithm for road scenes captured by intelligent transportation systems[J]. IEEE Transactions on Intelligent Transportation Systems, 2014, 15(5): 2321-2332.
21	KIM J H, JANG W D, SIM J Y, et al. Optimized contrast enhancement for real-time image and video dehazing[J]. Journal of Visual Communication and Image Representation, 2013, 24(3): 410-425.
22	ZHU Q S, MAI J M, SHAO L. A fast single image haze removal algorithm using color attenuation prior[J]. IEEE Transactions on Image Processing: A Publication of the IEEE Signal Processing Society, 2015, 24(11): 3522-3533.
23	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015)[2020]. .
24	KIM J H, JANG W D, SIM J Y, et al. Optimized contrast enhancement for real-time image and video dehazing[J]. Journal of Visual Communication and Image Representation, 2013, 24(3): 410-425.
25	李大鹏, 禹晶, 肖创柏. 图像去雾的无参考客观质量评测方法[J]. 中国图象图形学报, 2011, 16(9): 1753-1757.
	LI D P, YU J, XIAO C B. No-reference quality assessment method for defogged images[J]. Journal of Image and Graphics, 2011, 16(9): 1753-1757. (in Chinese)

[1]	肖进胜, 周景龙, 雷俊锋, 刘恩雨, 舒成. 基于霾层学习的单幅图像去雾算法[J]. 电子学报, 2019, 47(10): 2142-2148.
[2]	刘杰平, 黄炳坤, 韦岗. 一种快速的单幅图像去雾算法[J]. 电子学报, 2017, 45(8): 1896-1901.
[3]	肖进胜, 高威, 邹白昱, 姚远, 章勇勤. 基于天空约束暗通道先验的图像去雾[J]. 电子学报, 2017, 45(2): 346-352.
[4]	张登银, 鞠铭烨, 王雪梅. 一种基于暗通道先验的快速图像去雾算法[J]. 电子学报, 2015, 43(7): 1437-1443.
[5]	南栋, 毕笃彦, 马时平, 何林远, 娄小龙. 基于景深约束的单幅雾天图像去雾算法[J]. 电子学报, 2015, 43(3): 500-504.
[6]	何林远, 毕笃彦, 熊磊, 南栋, 周理. 基于亮度反馈的彩色雾霾图像增强算法[J]. 电子学报, 2015, 43(10): 1978-1983.