小目标特征增强图像分割算法

doi:10.12263/DZXB.20211123

电子学报 ›› 2022, Vol. 50 ›› Issue (8): 1894-1904.DOI: 10.12263/DZXB.20211123

小目标特征增强图像分割算法

任莎莎, 刘琼()

华南理工大学校软件学院，广东广州 511436

收稿日期:2021-08-18 修回日期:2022-06-22 出版日期:2022-08-25
- 通讯作者:
- 刘琼
- 作者简介:
- 任莎莎　女，1992年3月出生，安徽淮北人.现为华南理工大学软件学院在读博士研究生.主要研究方向为信号处理、图像理解与分割等方向.E-mail: 201910107240@mail.scut.edu.cn
  刘琼　女，1959年3月出生，云南昆明人.现为华南理工大学软件学院教授、博士生导师.承担、参加国家自然科学基金项目、国家863、973及地方政府项目10余项，国内外学术刊物和会议发表论文50余篇，中国专利5件.研究方向为计算机网络、计算机视觉、模式识别等.
- 基金资助:
- 国家自然科学基金(61976094);广东省自然科学基金(2021A1515011349)

A Tiny Target Feature Enhancement Algorithm for Semantic Segmentation

REN Sha-sha, LIU Qiong()

School of Software Engineering, South China University of Technology, Guangzhou, Guangdong 511436, China

Received:2021-08-18 Revised:2022-06-22 Online:2022-08-25 Published:2022-09-08
- Corresponding author:
- LIU Qiong
- Supported by:
- National Natural Science Foundation of China(61976094);National Natural Science Foundation of Guangdong Province(2021A1515011349)

摘要/Abstract

摘要：

在图像场景分割中存在小目标易丢失，边缘轮廓噪声大等问题.在目前的增强特征表征能力与优化空间细节的语义分割算法中，由于边缘和小目标特征的丢失，导致小目标和边缘很难被准确分割.为此，本文研究了一种小目标特征增强的图像分割算法.首先设计一种像素空间注意力模块（Pixel spatial Attention Module，PAM），来获得空间像素具有较强语义信息的特征图像.然后通过对PAM的输出进行建模提取，分别获得含有语义类别信息的边缘特征和小目标特征.最后，将特定的损失函数应用到语义分割训练中，并将多种特征进行融合，经过反复的监督学习和训练校正，可以在不影响其他类别性能的情况下提高边缘和小目标分割的性能.在Cityscapes，VOC2012，ADE20K和Camvid基线数据集上的实验表明，该算法与先进的图像分割算法相比，在小目标分割、边缘特征增强和内轮廓噪声减少等方面，其性能和效果都有明显提高，分割精度提高了2个百分点.

关键词: 场景分割, 小目标特征增强, 注意力模块, 建模

Abstract:

We have to face the challenge of missing small targets and severe edge noise in semantic segmentation. The existing semantic segmentation algorithms that enhance feature representation and optimize spatial details have difficulty to accurately segment the small targets and edges as the algorithms insufficiently gain detail information from tiny targets and semantic edges. This paper presents a tiny target feature enhancement algorithm for semantic segmentation. Specifically, a pixel spatial attention module(PAM) is designed to obtain strong semantic information from low-level pixel space. Semantic category information including edge features and tiny target features are obtained by modeling mask, respectively. A special loss function is designed for model training and the features gained by the model are fused with the features obtained from above way. Through edge feature enhancement, inner contour noise reduction, the segmentation performance of tiny target is improved while other segmentation categories are not degraded. Experimental results on Cityscapes, VOC2012, ADE20K and Camvid show that the proposed algorithm performance has been significantly improved by 2% in comparison with other state-of-the-art algorithms in the same scene.

Key words: scene segmentation, tiny target feature enhancement, attention module, modeling

中图分类号:

TP751

任莎莎, 刘琼. 小目标特征增强图像分割算法[J]. 电子学报, 2022, 50(8): 1894-1904.

Sha-sha REN, Qiong LIU . A Tiny Target Feature Enhancement Algorithm for Semantic Segmentation[J]. Acta Electronica Sinica, 2022, 50(8): 1894-1904.

图/表 17

参考文献 44

1	FARABETC, COUPRIEC, NAJMANL, et al. Learning hierarchical features for scene labeling[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(8): 1915-1929.
2	SIMONYANK, ZISSERMANA. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-09-04)[2022-06-15]. .
3	Al-QIZWINIM, BARJASTEHI, Al-QASSABH, et al. Deep learning algorithm for autonomous driving using googlenet[C]//IEEE Intelligent Vehicles Symposium. Los Angeles: IEEE, 2017: 89-96.
4	HEK, ZHANGX, RENS, et al. Deep residual learning for image recognition[C]//Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
5	HUANGG, LIUZ, L VAN DERMAATEN, et al. Densely connected convolutional networks[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4700-4708.
6	LIUS, De MELLOS, GUJ, et al. Learning affinity via spatial propagation networks[C]//Neural Information Processing Systems. Long Beach: MIT Press,2017: 1520-1530.
7	CHENL C, PAPANDREOUG, KOKKINOSI, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.
8	KRAHENBUHLP, KOLTUNV. Efficient inference in fully connected crfs with Gaussian edge potentials[J]. Advances in Neural Information Processing Systems, 2011, 24: 109-117.
9	POHLENT, HERMANSA, MATHIASM, et al. Full-resolution residual networks for semantic segmentation in street scenes[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4151-4160.
10	GUOD, ZHUL, LUY, et al. Tiny object sensitive segmentation of urban street scene with spatial adjacency between object classes[J]. IEEE Transactions on Image Processing, 2018, 28(6): 2643-2653.
11	YANGZ, YUH, FENGM, et al. Tiny object augmentation of urban scenes for real-time semantic segmentation[J]. IEEE Transactions on Image Processing, 2020, 29: 5175-5190.
12	CHANDRAS, KOKKINOSI. Fast exact and multi-scale inference for semantic image segmentation with deep Gaussian CRFs[C]//European Conference on Computer Vision. Netherlands: Springer, 2016: 402-418.
13	JAMPANIV, KIEFELM, GEHLERP V. Learning sparse high dimensional filters: Image filtering, dense crfs and bilateral neural networks[C]//Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 4452-4461.
14	ZHAOH, SHIJ, QIX, et al. Pyramid scene parsing network[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2881-2890.
15	YUF, KOLTUNV. Multi-scale context aggregation by dilated convolutions[C]//International Conference on Learning Representations. San Diego: OpenReview.net, 2015: 1-14.
16	CHENL C, ZHUY, PAPANDREOUG, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//European Conference on Computer Vision. Munich: Springer, 2018: 801-818.
17	BERTASIUSG, SHIJ, TORRESANIL. Semantic segmentation with boundary neural fields[C]//Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 3602-3610.
18	CHENGD, MENGG, XIANGS, et al. Fusionnet: Edge aware deep convolutional networks for semantic segmentation of remote sensing harbor images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017, 10(12): 5769-5783.
19	LING, MILANA, SHENC, et al. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1925-1934.
20	PENGC, ZHANGX, YUG, et al. Large kernel matters - improve semantic segmentation by global convolutional network[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4353-4361.
21	TAKIKAWAT, ACUNAD, JAMPANIV, et al. Gated-SCNN: Gated shape cnns for semantic segmentation[C]//International Conference on Computer Vision. South Korea: IEEE, 2019: 5229-5238.
22	DINGH, JIANGX, LIUA, et al. Boundary-aware feature propagation for scene segmentation[C]//International Conference on Computer Vision. South Korea: IEEE, 2019: 6819-6829.
23	HOLSCHNEIDERM. A real-time algorithm for signal analysis with the help of the wavelet transform[J]. Wavelets, 1988, 1: 286-297.
24	VAIDYANATHANP P. Multirate digital filters, filter banks, polyphase networks, and applications: A tutorial[J]. Proc IEEE, 1990, 78(1): 56-93.
25	HUANGZ, WANGX, HUANGL, et al. Ccnet: Criss-cross attention for semantic segmentation[C]//International Conference on Computer Vision. South Korea: IEEE, 2019: 603-612.
26	LIX, ZHONGZ, WUJ, et al. Expectation-maximization attention networks for semantic segmentation[C]//International Conference on Computer Vision. South Korea: IEEE, 2019: 9167-9176.
27	ZHONGZ, LINZ Q, BIDARTR, et al. Squeeze-and-attention networks for semantic segmentation[C]//Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 13065-13074.
28	HUJ, SHENL, SUNG. Squeeze-and-excitation networks[C]//Computer Vision and Pattern Recognition. Washington: IEEE, 2018: 7132-7141.
29	CHENL, ZHANGH, XIAOJ, et al. SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5659-5667.
30	ZHIDINGY, CHENF, LIUM, et al. Casenet: Deep category-aware semantic edge detection[C]//Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1761-1770.
31	ACUNAD, KARA, FidlerS. Devil is in the edges: Learning semantic boundaries from noisy annotations[C]//Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 11075-11083.
32	PERAZZIF, PONT-TUSETJ, McWilliamsB, et al. A benchmark dataset and evaluation methodology for video object segmentation[C]//Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 724-732.
33	梁新宇, 林洗坤, 权冀川, 肖铠鸿. 基于深度学习的图像实例分割技术研究进展[J]. 电子学报, 2020, 48(12): 2476-2486.
	LIANGX, LINX, QUANY, et al. Research on the progress of image instance segmentation based on deep learning[J].Acta Electronica Sinica, 2020, 48(12): 2476-2486. (in Chinese)
34	蔡超丽, 李纯纯, 黄琳, 杨铁军. ED-NAS: 基于神经网络架构搜索的陶瓷晶粒SEM图像分割方法[J]. 电子学报,2022, 50(2): 461-469.
	CAIC, LIC, HUANGL, et al. ED-NAS: Ceramic grain segmentation based on neural architecture search using SEM images[J].Acta Electronica Sinica, 2022, 50(2): 461-469. (in Chinese)
35	MISHRAP, SARAWADEKARK. Polynomial learning rate policy with warm restart for deep neural network[C]//IEEE Region 10 Conference. India: IEEE, 2019: 2087-2092.
36	IOFFES, SZEGEDYC. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning. Lille: PMLR, 2015: 448-456.
37	LONGJ, SHELHAMERE, DARRELLT. Fully convolutional networks for semantic segmentation[C]//Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3431-3440.
38	LIX, LIX, ZHANGL, et al. Improving semantic segmentation via decoupled body and edge supervision[C]//European Conference on Computer Vision. Glasgow: Springer, 2020:1-14.
39	ZHAOH, ZHANGY, LIUS, et al. PSANET: Point-wise spatial attention network for scene parsing[C]//European Conference on Computer Vision. Munich: Springer, 2018: 267-283.
40	FUJ, LIUJ, TIANH, et al. Dual attention network for scene segmentation[C]//Computer Vision and Pattern Recognition. New York: IEEE, 2019: 3146-3154.
41	BOWENC, ALEXS, ALEXANDERK. Per-pixel classification is not all you need for semantic segmentation[C]//Neural Information Processing Systems. Virtual Conference: MIT, 2021:1-12.
42	ZAGORUYKOS, KOMODAKISN. Wide residual networks(EB/OL).(2016-03-23)[2022-06-15]..
43	DAIJ, QIH, XIONGY, et al. Deformable convolutional networks[C]//International Conference on Computer Vision. Venice: IEEE, 2017: 764-773.
44	ZHUL, JID, ZHUS, et al. Learning statistical texture for semantic segmentation[C]//Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 12537-12546.

Method	IoU/%																			mIoU/%
Method	road	swal	build	wall	fenc	pole	tlight	sign	veg	terra	sky	person	rider	car	truch	bus	train	mobi	bike	mIoU/%
FCN	97.8	84.1	91.9	41.3	56.5	64.3	71.3	79.4	92.1	63.6	94.3	82.3	61.4	93.8	47.9	75.1	42.7	56.4	77.5	72.3
PSPNet	98.0	84.5	92.9	54.9	61.9	66.5	72.2	80.9	92.6	65.6	94.8	83.1	63.5	95.4	83.9	90.6	84.0	67.6	78.5	79.6
EAMANet	98.3	86.9	92.8	45.8	62.3	67.7	74.8	81.9	92.7	63.3	95.0	84.2	65.6	95.9	84.3	85.0	58.6	68.8	79.9	78.1
PSANet	98.1	84.8	92.4	46.2	58.9	66.1	73.6	80.8	92.5	64.3	94.7	83.0	62.0	95.5	78.6	88.2	80.7	67.0	79.5	78.2
DANet	98.5	87.2	93.1	55.4	61.6	67.7	74.2	80.4	92.4	64.2	95.0	83.8	64.9	95.9	87.9	91.4	80.9	70.1	79.5	80.2
Deeplabv3+	98.2	86.6	93.1	55.2	63.1	70.0	75.1	82.1	93.0	64.1	95.1	84.1	65.7	95.6	84.3	89.4	77.1	71.2	80.1	80.0
Gscnn	98.1	87.4	92.3	54.8	79.8	75.5	75.8	84.7	93.3	68.0	95.7	82.4	51.7	95.5	66.3	95.9	92.5	32.4	73.3	78.4
Maskformer	97.8	82.6	90.8	43.5	49.3	62.3	66.8	76.4	91.4	60.8	93.7	79.3	59.1	94.4	65.8	77.0	64.3	57.8	76.0	73.1
DSNet	98.3	86.5	93.6	60.7	66.8	70.7	73.9	81.9	931	66.1	95.2	843	67.5	95.8	86.1	92.3	85.5	72.1	80.1	81.5
Ours	98.5	87.7	93.5	54.9	65.0	72.0	77.0	83.9	93.2	67.0	95.5	85.9	69.6	96.2	86.4	92.5	83.7	74.5	81.8	82.8

Method	IoU/%																			mIoU/%
Method	road	swal	build	wall	fenc	pole	tlight	sign	veg	terra	sky	person	rider	car	truch	bus	train	mobi	bike	mIoU/%
FCN	97.8	84.1	91.9	41.3	56.5	64.3	71.3	79.4	92.1	63.6	94.3	82.3	61.4	93.8	47.9	75.1	42.7	56.4	77.5	72.3
PSPNet	98.0	84.5	92.9	54.9	61.9	66.5	72.2	80.9	92.6	65.6	94.8	83.1	63.5	95.4	83.9	90.6	84.0	67.6	78.5	79.6
EAMANet	98.3	86.9	92.8	45.8	62.3	67.7	74.8	81.9	92.7	63.3	95.0	84.2	65.6	95.9	84.3	85.0	58.6	68.8	79.9	78.1
PSANet	98.1	84.8	92.4	46.2	58.9	66.1	73.6	80.8	92.5	64.3	94.7	83.0	62.0	95.5	78.6	88.2	80.7	67.0	79.5	78.2
DANet	98.5	87.2	93.1	55.4	61.6	67.7	74.2	80.4	92.4	64.2	95.0	83.8	64.9	95.9	87.9	91.4	80.9	70.1	79.5	80.2
Deeplabv3+	98.2	86.6	93.1	55.2	63.1	70.0	75.1	82.1	93.0	64.1	95.1	84.1	65.7	95.6	84.3	89.4	77.1	71.2	80.1	80.0
Gscnn	98.1	87.4	92.3	54.8	79.8	75.5	75.8	84.7	93.3	68.0	95.7	82.4	51.7	95.5	66.3	95.9	92.5	32.4	73.3	78.4
Maskformer	97.8	82.6	90.8	43.5	49.3	62.3	66.8	76.4	91.4	60.8	93.7	79.3	59.1	94.4	65.8	77.0	64.3	57.8	76.0	73.1
DSNet	98.3	86.5	93.6	60.7	66.8	70.7	73.9	81.9	931	66.1	95.2	843	67.5	95.8	86.1	92.3	85.5	72.1	80.1	81.5
Ours	98.5	87.7	93.5	54.9	65.0	72.0	77.0	83.9	93.2	67.0	95.5	85.9	69.6	96.2	86.4	92.5	83.7	74.5	81.8	82.8

Backbone	Method	mIoU/%	Δ/%
ResNet50	FCN	71.4	—
	ASPP	76.6	—
	Ours	80.1	3.5↑
ResNet101	ASPP	77.5	—
ResNet101	Ours	80.8	3.3↑

Backbone	Method	mIoU/%	Δ/%
ResNet50	FCN	71.4	—
	ASPP	76.6	—
	Ours	80.1	3.5↑
ResNet101	ASPP	77.5	—
ResNet101	Ours	80.8	3.3↑

Method	mIoU/%	Δ/%
FCN+ASPP	77.5	—
DCN	78.2	0.7↑
GSCNN	77.8	0.3↑
DSNet	80.1	2.6↑
STLNet	80.9	3.4↑
Ours	80.8	3.3↑

小目标特征增强图像分割算法

A Tiny Target Feature Enhancement Algorithm for Semantic Segmentation

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 44

相关文章 15

编辑推荐

Metrics

Method	L_seg	L_edge-bce	L_edge-ohem	mIoU/%	Δ/%
FCN+ASPP	√	—	—	76.6	—
	—	√	—	76.8	0.2↑
	√	√	—	77.6	1.0↑
	√	—	√	80.1	3.5↑

Method	mIoU/%	Δ/%
FCN+ASPP	77.5	—
+ EEM	79.0	1.5↑
+ TTE	79.8	2.3↑
+ EEM+TTE	80.8	3.3↑

BackBone	Method	mIoU/%	Δ/%	#GFLOPs
ResNet50	PSPnet	73.5	—	178.4
	Deeplabv3+	74.5	—	179.5
	Ours	75.7	1.2↑	177.1
ResNet101	PSPnet	74.6	—	256.1
	Deeplabv3+	74.9	—	275.2
	Ours	76.4	1.9↑	256.8

Backbone	Method	mIoU/%	∆/%	#GFLOPs
ResNet101	FCN	47.4	—	275.4
	PSPnet	48.9	—	256.1
	Deeplabv3+	67.0	—	254.1
	Ours	69.1	2.0 ↑	254.8

[1]	邓海刚, 王传旭, 李成伟, 林晓萌. 深度学习框架下群组行为识别算法综述[J]. 电子学报, 2022, 50(8): 2018-2036.
[2]	赵雄文, 张钰, 秦鹏, 王晓晴, 耿绥燕, 宋俊元, 刘瑶, 李思峰. 空天地一体化无线光通信网络关键技术及其发展趋势[J]. 电子学报, 2022, 50(1): 1-17.
[3]	王相海, 宋若曦, 曲思洁, 穆振华, 宋传鸣. 图像多尺度几何分析域隐马尔可夫树模型研究进展[J]. 电子学报, 2022, 50(1): 238-249.
[4]	陶凌, 陈安庆, 邓贞宙, 韩春雷, 王玉皞, 王平. 应用分形几何与混沌理论进行植物形态建模[J]. 电子学报, 2021, 49(9): 1776-1782.
[5]	王伟, 于磊, 任国恒, 陈立勇, 董秋雷, 胡占义. 城市建筑立面三维“线-面”结构快速重建[J]. 电子学报, 2021, 49(8): 1551-1560.
[6]	魏光辉, 杜雪, 王雅平. 用频设备带外多频电磁辐射三阶互调阻塞效应测试与建模评估方法[J]. 电子学报, 2021, 49(6): 1094-1100.
[7]	乔思波, 庞善臣, 王敏, 翟雪, 于世行, 丁桐. 基于残差混合注意力机制的脑部CT图像分类卷积神经网络模型[J]. 电子学报, 2021, 49(5): 984-991.
[8]	高敏娟, 党宏社, 魏立力, 刘国军, 张选德. 全参考图像质量评价回顾与展望[J]. 电子学报, 2021, 49(11): 2261-2272.
[9]	张文茹, 汪莞乔, 官科, 艾渤, 曾毅, 赖峥嵘. 城市交通环境下23GHz信道仿真建模[J]. 电子学报, 2021, 49(10): 2069-2080.
[10]	王传旭, 薛豪. 基于GFU和分层LSTM的组群行为识别研究方法[J]. 电子学报, 2020, 48(8): 1465-1471.
[11]	陆兴鹏, 王明羽, 曹扬, 赵瑞莲, 周武, 李兆麟, 魏少军. 一种基于帧图像的动态视觉传感器样本集建模方法[J]. 电子学报, 2020, 48(8): 1457-1464.
[12]	丰艳, 张甜甜, 王传旭. 基于伪3D残差网络与交互关系建模的群组行为识别方法[J]. 电子学报, 2020, 48(7): 1269-1275.
[13]	黄子杰, 陈军华, 高建华. Code Smell视角下分层Web应用失血及充血现象的量化分析[J]. 电子学报, 2020, 48(4): 772-780.
[14]	王传旭, 胡小悦, 孟唯佳, 闫春娟. 基于多流架构与长短时记忆网络的组群行为识别方法研究[J]. 电子学报, 2020, 48(4): 800-807.
[15]	任仪, 罗伟, 胡浩. 基于改进型双正切变换的高阶近奇异性精确积分方法研究[J]. 电子学报, 2020, 48(10): 1873-1882.