双注意力引导的细节和结构信息融合图像去雾网络

doi:10.12263/DZXB.20211549

电子学报 ›› 2023, Vol. 51 ›› Issue (1): 160-171.DOI: 10.12263/DZXB.20211549

双注意力引导的细节和结构信息融合图像去雾网络

高继蕊¹^,², 李华锋¹^,², 张亚飞¹^,², 谢明鸿¹, 李凡¹^,²

^1.昆明理工大学信息工程与自动化学院，云南昆明 650500
^2.云南省人工智能重点实验室，云南昆明 650500

收稿日期:2021-11-19 修回日期:2022-04-25 出版日期:2023-01-25
- 作者简介:
- 高继蕊女，1996年9月出生于云南省临沧市.昆明理工大学信息工程与自动化学院硕士研究生.主要研究方向为计算机视觉与图像处理.E-mail: 2945266153@qq.com
  李华锋男，1983年7月出生于安徽省阜阳市.昆明理工大学信息工程与自动化学院教授，博士生导师.主要研究方向为计算机视觉.E-mail: hfchina99@163.com
  张亚飞（通讯作者）女，1981年5月出生于河南省洛阳市.昆明理工大学信息工程与自动化学院副教授，硕士生导师.主要研究方向为图像处理、模式识别.
  谢明鸿男，1976年12月出生于云南省昭通市.昆明理工大学信息工程与自动化学院高级工程师，硕士生导师.主要研究方向为计算机视觉.E-mail: minghongxie@163.com
  李凡男，1986年8月出生于云南省楚雄市.昆明理工大学信息工程与自动化学院副教授，硕士生导师.主要研究方向为计算机视觉.E-mail: 478263823@qq.com
- 基金资助:
- 国家自然科学基金 (62161015)

Dual Attention-Guided Detail and Structure Information Fusion Network for Image Dehazing

GAO Ji-rui¹^,², LI Hua-feng¹^,², ZHANG Ya-fei¹^,², XIE Ming-hong¹, LI Fan¹^,²

^1.Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, Yunnan 650500, China
^2.Yunnan Key Laboratory of Artificial Intelligence, Kunming, Yunnan 650500, China

Received:2021-11-19 Revised:2022-04-25 Online:2023-01-25 Published:2023-02-23
- Supported by:
- National Natural Science Foundation of China (62161015)

摘要/Abstract

摘要：

雾图像结构信息弱化、边缘细节信息丢失，严重影响其在高水平视觉任务的使用.现有大部分去雾方法对图像细节信息的恢复并不理想，影响了图像去雾的整体效果.为此，本文提出一种双注意力引导的细节和结构信息融合去雾网络.该网络主要由空间-通道双注意力联合模块、细节和结构信息融合模块以及多尺度特征重建模块组成.其中，空间-通道双注意力联合模块通过联合空间和通道两个维度的注意力进行特征提取，实现雾图像中细节和结构信息的增强；细节和结构信息融合模块将结构信息和边缘细节信息融合为注意力权重和逆向注意力权重，以进一步增强这两种信息；多尺度特征重建模块将提取到的特征重建为清晰图像.实验结果表明，本文方法的去雾效果在定量评价和视觉效果上均优于对比方法.

关键词: 图像去雾, 图像恢复, 信息融合, 注意力, 特征重建

Abstract:

Haze weakens the structural information of an image and makes the edge information lost, which negatively affects the performance of high-level vision tasks. The details recovered by most existing dehazing methods are unsatisfactory, affecting the overall effect of image dehazing. To this end, this paper proposes a dual-attention guided detail and structure information fusion network composed of spatial-channel dual attention joint module, detail and structure information fusion module, and multi-scale feature reconstruction module. The spatial-channel dual attention joint module performs feature extraction by combining spatial attention and channel attention to enhance details and structural information in the hazy image. The detail and structure information fusion module fuses structure and edge into attention weights and inverse attention weights to further enhance both information. The multi-scale feature reconstruction module reconstructs the extracted features into a clear image. The experiment results show that the dehazing effect of the proposed method is superior to that of the compared methods in both quantitative evaluation and visual effect.

Key words: image dehazing, image restoration, information fusion, attention, feature reconstruction

中图分类号:

TP399

高继蕊, 李华锋, 张亚飞, 等. 双注意力引导的细节和结构信息融合图像去雾网络[J]. 电子学报, 2023, 51(1): 160-171.

Ji-rui GAO, Hua-feng LI, Ya-fei ZHANG, et al. Dual Attention-Guided Detail and Structure Information Fusion Network for Image Dehazing[J]. Acta Electronica Sinica, 2023, 51(1): 160-171.

图/表 13

图1 本文提出方法的总体框架图

图2 空间-通道双注意力联合模块结构图

图3 细节和结构信息融合模块结构图

图4 多尺度特征重建模块结构图

图5 不同方法在SOTS-outdoor测试集上的测试效果

图6 不同方法在HSTS-Syn测试集上的测试效果

图7 不同方法在HAZERD测试集上的测试效果

图8 去雾前后图像细节信息对比

图9 不同方法去雾细节信息恢复情况对比

表1 不同方法在不同测试集上去雾结果的定量评价结果（PSNR/SSIM/LPIPS）

方法	SOTS-outdoor	HSTS-Syn	HAZERD
CAP(TIP’15)^[13]	18.12/0.758 1/0.135	19.71/0.809 5/0.114	14.12/0.708 2/0.267
DehazeNet(TIP’16)^[15]	22.61/0.863 3/0.075	23.92/0.888 0/0.065	15.52/0.760 1/0.236
EPDN(CVPR’19)^[21]	19.96/0.834 2/0.144	21.35/0.883 3/0.098	15.70/0.754 7/0.267
FFA-Net(AAAI’20)^[22]	30.89/0.936 7/0.038	31.36/0.927 9/0.031	16.26/0.791 0/0.253
DA(CVPR’20)^[29]	25.79/0.886 4/0.197	26.26/0.864 7/0.170	17.65/0.804 3/0.243
TBNN(CVPRW’21)^[30]	15.21/0.780 6/0.186	13.65/0.747 9/0.186	12.03/0.745 7/0.293
AOD-Net(ICCV’17)^[43]	19.65/0.854 8/0.097	19.77/0.839 4/0.099	15.45/0.749 3/0.269
DPSID(ITM’18)^[44]	18.44/0.861 7/0.119	18.45/0.849 5/0.120	15.52/0.772 4/0.224
本文方法	29.43/0.968 5/0.014	29.76/0.972 9/0.011	16.93/0.812 5/0.227

表2 本文提出的方法中不同模块消融实验的结果

方法	PSNR	SSIM	LPIPS
BL	26.50	0.954 9	0.022 4
BL+S-C	27.90	0.962 8	0.017 7
BL+S-C+DS	28.75	0.963 3	0.016 5
BL+S-C+DS+MR	29.43	0.968 5	0.014 3
BL+CP+DS+MR	27.70	0.957 8	0.019 7
W/O $p$	29.03	0.966 2	0.016 7
W/O $p ˜$	28.05	0.962 1	0.018 1
W/O MFEB	28.80	0.965 3	0.015 5

表2 本文提出的方法中不同模块消融实验的结果

方法	PSNR	SSIM	LPIPS
BL	26.50	0.954 9	0.022 4
BL+S-C	27.90	0.962 8	0.017 7
BL+S-C+DS	28.75	0.963 3	0.016 5
BL+S-C+DS+MR	29.43	0.968 5	0.014 3
BL+CP+DS+MR	27.70	0.957 8	0.019 7
W/O $p$	29.03	0.966 2	0.016 7
W/O $p ˜$	28.05	0.962 1	0.018 1
W/O MFEB	28.80	0.965 3	0.015 5

表3 不同方法的模型大小、参数量和计算量大小比较

方法	Model size/MB	Params/M	FLOPs/G
CAP	—	—	—
DehazeNet	0.030	0.008	0.394
AOD-Net	0.009	0.002	0.088
EPDN	66.300	17.379	3.677
DA	208.354	54.591	30.041
FFA-Net	25.390	4.456	220.551
TBNN	192.769	50.352	64.736
DPSID	—	—	—
Ours	5.410	1.342	71.562

图 10 损失函数Ltotal权重参数分析图

参考文献 44

1	PANG Y W, LI Y Z, SHEN J B, et al. Towards bridging semantic gap to improve semantic segmentation[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 4229-4238.
2	MA S, PANG Y, PAN J, et al. Preserving details in semantics-aware context for scene parsing[J]. Science China Information Sciences, 2020, 63(2): 120106.
3	ZHANG Z J, PANG Y W. CGNet: Cross-guidance network for semantic segmentation[J]. Science China Information Sciences, 2020, 63(2): 120104.
4	NIE J, ANWER R M, CHOLAKKAL H, et al. Enriched feature guided refinement network for object detection[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 9536-9545.
5	PANG Y W, XIE J, KHAN M H, et al. Mask-guided attention network for occluded pedestrian detection[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, : 4966-4974.
6	CAO J L, PANG Y W, HAN J G, et al. Hierarchical shot detector[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 9704-9713.
7	LI Y Z, PANG Y W, SHEN J B, et al. NETNet: Neighbor erasing and transferring network for better single shot object detection[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 13346-13355.
8	NI W P, GAO X B, WANG Y. Single satellite image dehazing via linear intensity transformation and local property analysis[J]. Neurocomputing, 2016, 175: 25-39.
9	MCCARTNEY E J, HALL F F. Optics of the atmosphere: Scattering by molecules and particles[J]. Physics Today, 1977, 30(5): 76-77.
10	NARASIMHAN S G, NAYAR S K. Chromatic framework for vision in bad weather[C]//Proceedings IEEE Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 598-605.
11	NARASIMHAN S G, NAYAR S K. Vision and the atmosphere[J]. International Journal of Computer Vision, 2002, 48(3): 233-254.
12	HE K M, SUN J, TANG X O. Single image haze removal using dark channel prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(12): 2341-2353.
13	ZHU Q S, MAI J M, SHAO L. A fast single image haze removal algorithm using color attenuation prior[J]. IEEE Transactions on Image Processing, 2015, 24(11): 3522-3533.
14	BERMAN D, TREIBITZ T, AVIDAN S. Non-local image dehazing[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1674-1682.
15	CAI B L, XU X M, JIA K, et al. DehazeNet: An end-to-end system for single image haze removal[J]. IEEE Transactions on Image Processing, 2016, 25(11): 5187-5198.
16	REN W, LIU S, ZHANG H, et al. Single image dehazing via multi-scale convolutional neural networks[C]//European Conference on Computer Vision. Amsterdam: Springer Cham, 2016: 154-169.
17	ZHANG H, PATEL V M. Densely connected pyramid dehazing network[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3194-3203.
18	LI Y N, MIAO Q G, OUYANG W L, et al. LAP-Net: Level-aware progressive network for image dehazing[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3275-3284.
19	REN W Q, MA L, ZHANG J W, et al. Gated fusion network for single image dehazing[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 3253-3261.
20	LIU X H, MA Y R, SHI Z H, et al. GridDehazeNet: Attention-based multi-scale network for image dehazing[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 7314-7323.
21	QU Y Y, CHEN Y Z, HUANG J Y, et al. Enhanced pix2pix dehazing network[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, : 8152-8160.
22	QIN X, WANG Z L, BAI Y C, et al. FFA-Net: Feature fusion attention network for single image dehazing[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI Press, 2020: 11908-11915.
23	DONG H, PAN J S, XIANG L, et al. Multi-scale boosted dehazing network with dense feature fusion[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2154-2164.
24	ZHANG X Q, JIANG R H, WANG T, et al. Single image dehazing via dual-path recurrent network[J]. IEEE Transactions on Image Processing, 2021, 30: 5211-5222.
25	张登银, 鞠铭烨, 王雪梅. 一种基于暗通道先验的快速图像去雾算法[J]. 电子学报, 2015, 43(7): 1437-1443.
	ZHANG D Y, JU M Y, WANG X M. A fast image daze removal algorithm using dark channel prior[J]. Acta Electronica Sinica, 2015, 43(7): 1437-1443. (in Chinese)
26	刘杰平, 黄炳坤, 韦岗. 一种快速的单幅图像去雾算法[J]. 电子学报, 2017, 45(8): 1896-1901.
	LIU J P, HUANG B K, WEI G. A fast effective single image dehazing algorithm[J]. Acta Electronica Sinica, 2017, 45(8): 1896-1901. (in Chinese)
27	FATTAL R. Dehazing using color-lines[J]. ACM Transactions on Graphics, 2014, 34(1): 13.
28	肖进胜, 周景龙, 雷俊锋, 等. 基于霾层学习的单幅图像去雾算法[J]. 电子学报, 2019, 47(10): 2142-2148.
	XIAO J S, ZHOU J L, LEI J F, et al. Single image dehazing algorithm based on the learning of hazy layers[J]. Acta Electronica Sinica, 2019, 47(10): 2142-2148. (in Chinese)
29	SHAO Y J, LI L, REN W Q, et al. Domain adaptation for image dehazing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2805-2814.
30	YU Y K, LIU H, FU M H, et al. A two-branch neural network for non-homogeneous dehazing via ensemble learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Nashville: IEEE, 2021: 193-202.
31	MNIH V, HEESS N, GRAVES A. Recurrent models of visual attention[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal: MIT Press, 2014: 2204-2212.
32	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
33	Fan D P, Ji G P, Zhou T, et al. PraNet: Parallel reverse attention network for polyp segmentation[C]//International Conference on Medical Image Computing and Computer Assisted Intervention. Lima: Springer, 2020: 263-273.
34	LI S L, ZHAO M, FANG Z Y, et al. Image super-resolution using lightweight multiscale residual dense network[J]. International Journal of Optics, 2020, 2020: 2852865.
35	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10)[2021-11-10]]. .
36	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
37	LI B Y, REN W Q, FU D P, et al. Benchmarking single-image dehazing and beyond[J]. IEEE Transactions on Image Processing, 2019, 28(1): 492-505.
38	ZHANG Y F, DING L, SHARMA G. HazeRD: An outdoor scene dataset and benchmark for single image dehazing[C]//2017 IEEE International Conference on Image Processing. Beijing: IEEE, 2017: 3205-3209.
39	Mannos J, Sakrison D. The effects of a visual fidelity criterion of the encoding of images[J]. IEEE Transactions on Information Theory, 1974, 20(4): 525-536.
40	ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 586-595.
41	Kingma D P, Ba J. Adam: A method for stochastic optimization[EB/OL].(2014-10-22)[2021-11-10]. .
42	HE T, ZHANG Z, ZHANG H, et al. Bag of tricks for image classification with convolutional neural networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 558-567.
43	LI B Y, PENG X L, WANG Z Y, et al. AOD-net: All-in-one dehazing network[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 4780-4788.
44	GAO Y Y, HU H M, LI B, et al. Detail preserved single image dehazing algorithm based on airlight refinement[J]. IEEE Transactions on Multimedia, 2019, 21(2): 351-362.

[1]	隗昊, 唐焕玲, 周爱, 张益嘉, 陈飞, 鲁明羽. 基于双路分段注意力神经张量网络的临床文本关系抽取[J]. 电子学报, 2023, 51(3): 658-665.
[2]	贾童瑶, 卓力, 李嘉锋, 张菁. 基于深度学习的单幅图像去雾研究进展[J]. 电子学报, 2023, 51(1): 231-245.
[3]	何滢婕, 刘月峰, 边浩东, 郭威, 张小燕. 基于Informer的电池荷电状态估算及其稀疏优化方法[J]. 电子学报, 2023, 51(1): 50-56.
[4]	王硕, 王坚, 王亚男, 宋亚飞. 一种基于特征融合的恶意代码快速检测方法[J]. 电子学报, 2023, 51(1): 57-66.
[5]	丁琪, 田萱, 孙国栋. 基于注意力增强的热点感知新闻推荐模型[J]. 电子学报, 2023, 51(1): 93-104.
[6]	金天虎, 陶砚蕴, 李佐勇. 基于超像素图像分割的暗通道先验去雾改进算法[J]. 电子学报, 2023, 51(1): 146-159.
[7]	梁顺攀, 刘伟, 郑智中, 原福永. 基于方面和胶囊网络的跨域评分预测模型[J]. 电子学报, 2022, 50(9): 2102-2109.
[8]	刘群, 谭洪胜, 张优敏, 王国胤. 基于元路径的动态异质网络表示学习[J]. 电子学报, 2022, 50(8): 1830-1839.
[9]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[10]	任莎莎, 刘琼. 小目标特征增强图像分割算法[J]. 电子学报, 2022, 50(8): 1894-1904.
[11]	赵俊男, 佘青山, 孟明, 陈云. 基于多流空间注意力图卷积SRU网络的骨架动作识别[J]. 电子学报, 2022, 50(7): 1579-1585.
[12]	麻文刚, 张亚东, 郭进. 基于景深先验引导与环境光优化的图像去雾[J]. 电子学报, 2022, 50(7): 1708-1721.
[13]	李政伟, 李佳树, 尤著宏, 聂茹, 赵欢, 钟堂波. 基于异质图注意力网络的miRNA与疾病关联预测算法[J]. 电子学报, 2022, 50(6): 1428-1435.
[14]	严莉, 张凯, 徐浩, 韩圣亚, 刘珅岐, 史玉良. 基于图注意力机制和Transformer的异常检测[J]. 电子学报, 2022, 50(4): 900-908.
[15]	王延达, 陈炜通, 皮德常, 岳琳. 一种自适应记忆神经网络多跳读取与覆盖度机制结合的药物推荐模型[J]. 电子学报, 2022, 50(4): 943-953.

双注意力引导的细节和结构信息融合图像去雾网络

Dual Attention-Guided Detail and Structure Information Fusion Network for Image Dehazing

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 44

相关文章 15

编辑推荐

Metrics