基于实景数据增强和双路径融合网络的实时街景语义分割算法

doi:10.12263/DZXB.20210611

摘要/Abstract

摘要：

街景图像的分割在工业运用中具有十分重要的作用，但是街景图像具有种类繁多、光照多变等特点，此外，街景分割任务在追求准确性的同时要兼顾实时性，以上特点使得该任务具有很大的挑战性.本文针对这一挑战性任务提出了一个由空间路径和细节路径组成的双路径网络（Dual-path Fusion Network，DFNet），其中细节路径利用高分辨率的输入得到丰富的边界信息，空间路径利用细节路径产生的高质量特征图获得足够多的语义信息；网络的开始嵌入了一个可训练的图像预处理模块（Image Preprocessing Module，IPM），该模块可以使光照不同的图像进入网络正式训练之前在RGB通道上具有方差和均值的一致性；经过预处理模块之后的特征图会分别输入到细节路径和空间路径；本文提出了一个条状注意力细化模块（Attention Refinement Module，ARM），并将其放到空间路径的最后，可以将通道级信息和局部条状信息有效结合起来；在网络的最后，利用图像融合模块（Feature Fusion Module，FFM）对两条路径的特征信息进行融合，得到最后的分割结果.同时，本文还提出了一种基于小目标重组的“复制粘贴”数据增强方法，减弱了小目标样本数据不均衡的问题，同时扩充了数据集，该算法可以提升单个网络近2%的平均交并比（mIoU）.本文利用所提算法在CityScapes和CamVid数据集上进行了实验验证，对于CityScapes数据集来说，输入大小为1 024×2 048，其每秒处理帧数（FPS）和mIoU分别达到了98和70.1%；对于CamVid数据集来说，输入大小为720×960，其FPS和mIoU分别达到了208和65.7%.与已有算法相比，本文算法的推理速度要优于最先进的实时街景语义分割算法，同时保持了较高的分割结果准确性，本文算法在街景图像语义分割速度和分割性能之间取得了良好的平衡.

关键词: 街景图像, 语义分割, 数据增强, 深度卷积神经网络

Abstract:

Semantic segmentation of road scene image plays a crucial role in industrial applications. However, challenges such as the great variety of target objects, high illumination variability in different scenes, and especially the increased requirement in speed and accuracy, make the segmentation of road scene images become difficult. To solve the above challenges, we propose an efficient convolutional neural network named dual-path fusion network(DFNet), consisting of spatial-path and detail-path. The spatial-path learns global information through low-resolution feature maps. Meanwhile, the detail-path can extract local details through high-resolution feature maps. DFNet starts with a trainable image preprocessing module(IPM), which is applied to unify the input images to have the same consistency of variance mean value on the RGB channel. Attention refinement module(ARM), which includes global pooling and strip pooling, is utilized in spatial-path to guide the feature learning while extracting the global features. After spatial-path and detail-path, a feature fusion module(FFM) is employed to effectively fuse the global and local detail features to achieve the final segmenting result. Besides the novel network DFNet, we propose a data augmentation strategy to enrich the training dataset and further solve the data imbalance issue of small objects. This straightforward “copy and paste” strategy can improve the performance of the same network by 2% in mIoU. We test our method on two public datasets, where it reaches FPS of 98 and mIoU of 70.1% on the CityScapes dataset(image size of 1 024×2 048), and FPS of 208 meantime mIoU of 65.7% on CamVid dataset (image size of 720×960). The experimental results show that our method achieves outperformance on speed as well as a competitive accuracy, compare to state-of-the-art methods. It also demonstrates that our approach can reach a good balance between speed and accuracy.

Key words: road scene image, semantic segmentation, data augmentation, deep convolutional neural network

中图分类号:

TP391

张志文, 刘天歌, 聂鹏举. 基于实景数据增强和双路径融合网络的实时街景语义分割算法[J]. 电子学报, 2022, 50(7): 1609-1620.

ZHANG Zhi-wen, LIU Tian-ge, NIE Peng-ju. Real-Time Semantic Segmentation for Road Scene Based on Data Enhancement and Dual-Path Fusion Network[J]. Acta Electronica Sinica, 2022, 50(7): 1609-1620.

图/表 17

参考文献 43

1	罗会兰, 张云. 基于深度网络的图像语义分割综述[J]. 电子学报, 2019, 47(10): 2211-2220.
	LUOH L, ZHANGY. A survey of image semantic segmentation based on deep network[J]. Acta Electronica Sinica, 2019, 47(10): 2211-2220. (in Chinese)
2	徐频捷, 陈逸杰, 李之南. 基于事件驱动的车道线识别算法研究[J]. 电子学报, 2021, 49(7): 1379-1385.
	XUP J, CHENY J, LIZ N, et al. Research on event-driven lane recognition algorithms[J]. Acta Electronica Sinica, 2021, 49(7): 1379-1385. (in Chinese)
3	GEIGERA, LENZP, URTASUNR. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition. Providence, RI: IEEE, 2012: 3354-3361.
4	LONGJ, SHELHAMERE, DARRELLT. Fully Convolutional Networks for Semantic Segmentation[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE, 2015: 3431-3440.
5	CHENL C, PAPANDREOUG, KOKKINOSI, et al. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4):834-848.
6	BADRINARAYANANV, KENDALLA, CIPOLLAR. SegNet: A deep convolutional encoder-decoder architecture for Image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
7	NOH, H, HONG, S, HANB. Learning deconvolution network for semantic segmentation[C]//2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 1520-1528.
8	WUZ F, SHENC H, A VAN DENHENGEL. Wider or deeper: Revisiting the ResNet model for visual recognition[J]. Pattern Recognition, 2019, 90: 119-133.
9	HEK M, ZHANGX Y, RENS Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE, 2016: 770-778.
10	ZHAOH S, SHIJ P, QIX J, et al. Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 6230-6239.
11	ZHAOH S, QIX J, SHENX Y, et al. ICNet for real-time semantic segmentation on high-resolution images[C].//European Conference on Computer Vision. Munich: Springer, 2018: 418-434.
12	ORSICM, KRESOI, BEVANDICP, et al. In defense of pre-trained ImageNet architectures for real-time semantic segmentation of road-driving images[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach, CA: IEEE, 2019: 12599-12608.
13	WUZ F, SHENC H, HENGELA V. Real-time semantic image segmentation via spatial sparsity[EB/OL]. (2017-10-01)[2021]. .
14	孟琭, 徐磊, 郭嘉阳. 一种基于改进的MobileNetV2网络语义分割算法[J]. 电子学报, 2020, 48(9): 1769-1776.
	MENGL, XUL, GUOJ Y. Semantic segmentation algorithm based on improved MobileNetV2[J]. Acta Electronica Sinica, 2020, 48(9): 1769-1776. (in Chinese)
15	PASZKEA, CHAURASIAA, KIMS, et al. ENet: A deep neural network architecture for real-time semantic segmentation[EB/OL]. (2016-06-07)[2021]. .
16	CHENL C, PAPANDREOUG, SCHROFFF, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. (2017-10-05)[2021]. .
17	SZEGEDYC, LIUW, JIAY Q, et al. Going deeper with convolutions[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Boston, MA: IEEE, 2015:1-9.
18	IOFFES, SZEGEDYC. Batch normalization: Accelerating deep network training by reducing internal covariate shift[EB/OL]. (2015-03-02)[2021]. .
19	SZEGEDYC, IOFFES, VANHOUCKEV, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[C]//AAAI'17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. San Francisco: AAAI Press, 2017: 4278-4284.
20	LIH C, XIONGP F, FANH Q, et al. DFANet: Deep feature aggregation for real-time semantic segmentation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach, CA: IEEE, 2019: 9514-9523.
21	YUC Q, GAOC X, WANGJ B, et al. BiSeNet V2: Bilateral network with guided aggregation for real-time semantic segmentation[J]. International Journal of Computer Vision, 2021, 129(11): 3051-3068.
22	NIRKINY, WOLFL, HASSNE R T. HyperSeg: Patch-wise hypernetwork for real-time semantic segmentation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Nashville, TN: IEEE, 2021: 4061-4070.
23	HUJ, SHENL, ALBANIES, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Recognition and Machine Intelligence, 2020, 42(8): 2011-2023.
24	HOUQ B, ZHANGL, CHENGM M, et al. Strip pooling: Rethinking spatial pooling for scene parsing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Seattle, WA: IEEE, 2020: 4002-4011.
25	DWIBEDID, MISRAI, Cut HEBERTM., paste and learn: Surprisingly easy synthesis for instance detection[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 1310-1319.
26	DVORNIKN, MAIRALJ, SCHMIDC. Modeling visual context is key to augmenting object detection datasets[M]//Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 375-391.
27	YANGZ G, YUH S, FENGM T, et al. Small object augmentation of urban scenes for real-time semantic segmentation[J]. IEEE Transactions on Image Processing, 2020, 29: 5175-5190.
28	KRIZHEVSKYA, SUTSKEVERI, HINTONG E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
29	KESKARN S, SOCHERR. Improving generalization performance by switching from Adam to SGD[EB/OL]. (2017-10-20)[2021]. .
30	CORDTSM, OMRANM, RAMOSS, et al. The cityscapes dataset for semantic urban scene understanding[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, NV: IEEE, 2016: 3213-3223.
31	BROSTOWG J, FAUQUEURJ, CIPOLLAR. Semantic object classes in video: A high-definition ground truth database[J]. Pattern Recognition Letters, 2009, 30(2): 88-97.
32	MAZZINID. Guided upsampling network for real-time semantic segmentation[EB/OL]. (2018-07-19)[2021]. .
33	MEHTAS, RASTEGARIM, CASPIA, et al. ESPNet: Efficient spatial pyramid of dilated convolutions for semantic segmentation[C]//European Conference on Computer Vision. Munich: Springer, 2018: 3567-3578.
34	MEHTAS, RASTEGARIM, SHAPIROL, et al. ESPNetv2: A light-weight, power efficient, and general purpose convolutional neural network[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Long Beach, CA: IEEE, 2019: 9182-9192.
35	ROMERAE, ALVAREZJ M, BERGASAL M, et al. ERFNet: Efficient residual factorized ConvNet for real-time semantic segmentation[J]. IEEE Transactions on Intelligent Transportation Systems, 2018,19(1): 263-272.
36	YUC Q, WANGJ B, PENGC, et al. BiSeNet: Bilateral segmentation network for real-time semantic segmentation[C]//European Conference on Computer Vision. Munich: Springer, 2018: 334-349.
37	YUF, KOLTUNV. Multi-scale context aggregation by dilated convolutions[EB/OL]. (2016-04-30)[2021]. .
38	CHENL C, PAPANDREOUSG, KOKKINOSI, et al. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
39	GHIASIG, FOWLKESC C. Laplacian Pyramid Reconstruction and Refinement for Semantic Segmentation[C]//European Conference on Computer Vision. Amsterdam: Springer, 2016: 519-534.
40	LING S, MILANA, SHENC H, et al. RefineNet: Multi-path refinement networks for high-resolution semantic segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI: IEEE, 2017: 5168-5177.
41	TREMLM, ARJONA-MEDINAJ, UNTERTHINERT, et al. Speeding up semantic segmentation for autonomous driving[C]//NIPS 2016-29th Conference on Neural Information Processing Systems. Barcelona: NIPS Workshop-mlits, 2016,123:312-331.
42	LIG, YUNI, KIMJ, et al. DABNet: depth-wise asymmetric bottleneck for real-time semantic segmentation[J]. CoRR, 2019, abs/1907.11357. .
43	WANGP Q, CHENP F, YUANY, et al. Understanding convolution for semantic segmentation[J]//2018 IEEE Winter Conference on Applications of Computer Vision(WACV). Lake Tahoe, NV: IEEE, 2018: 1451-1460.

阶段	细节路径
阶段	操作	步长	输入通道数	输出通道数
细节卷积块1	Conv 3×3	2	3	16
细节卷积块1	Conv 3×3	2	16	32
细节卷积块2	Conv 3×3	2	32	64
	Conv 3×3	1	64	64
	Conv 3×3	1	64	64
细节卷积块3	Conv 3×3	2	64	128
	Conv 3×3	1	128	128
	Conv 3×3	1	128	128
细节卷积块4	Conv 3×3	2	128	128
	Conv 3×3	1	128	128
	Conv 3×3	1	128	128

阶段	细节路径
阶段	操作	步长	输入通道数	输出通道数
细节卷积块1	Conv 3×3	2	3	16
细节卷积块1	Conv 3×3	2	16	32
细节卷积块2	Conv 3×3	2	32	64
	Conv 3×3	1	64	64
	Conv 3×3	1	64	64
细节卷积块3	Conv 3×3	2	64	128
	Conv 3×3	1	128	128
	Conv 3×3	1	128	128
细节卷积块4	Conv 3×3	2	128	128
	Conv 3×3	1	128	128
	Conv 3×3	1	128	128

阶段	空间路径
阶段	操作	步长	输入通道数	输出通道数
空间卷积块1	SMM	2	64	32
空间卷积块1	SMM	1	32	32
C	Concat	1	160	160
空间卷积块2	SMM	2	160	64
	SMM	1	64	64
	SMM	1	64	64
C	Concat	1	192	192
空间卷积块3	SMM	2	192	128
	SMM	1	128	128
	SMM	1	128	128
	SMM	1	128	128

阶段	空间路径
阶段	操作	步长	输入通道数	输出通道数
空间卷积块1	SMM	2	64	32
空间卷积块1	SMM	1	32	32
C	Concat	1	160	160
空间卷积块2	SMM	2	160	64
	SMM	1	64	64
	SMM	1	64	64
C	Concat	1	192	192
空间卷积块3	SMM	2	192	128
	SMM	1	128	128
	SMM	1	128	128
	SMM	1	128	128

类别	小目标	IoU/%
类别	小目标	原始数据	复制粘贴	本文算法
车道	否	97.3	97.4	97.5
人行道	否	79.1	80.0	80.1
建筑	否	89.0	89.7	89.5
墙	否	56.6	55.1	54.3
栅栏	否	53.4	53.8	52.8
支撑杆	是	47.2	48.5	48.0
交通灯	是	57.2	58.6	59.6
标志	是	65.9	67.3	67.5
植物	否	89.7	90.1	89.8
地势	否	59.3	59.7	56.0
天空	否	92.6	92.5	92.5
行人	是	71.8	72.4	72.7
骑行者	是	56.0	53.4	56.5
小汽车	否	91.5	91.7	92.1
卡车	否	68.7	61.9	69.6
公交车	否	74.6	73.1	79.0
列车	否	50.2	60.9	70.0
自行车	是	50.2	60.2	58.2
摩托车	是	68.5	69.2	69.9
mIoU/%	—	69.4	70.3	71.4