基于剪边策略的图残差卷积深层网络模型

doi:10.12263/DZXB.20210152

摘要/Abstract

摘要：

图神经网络自2005年以来已经逐步成为图学习中的一个重要的研究分支，其中最为活跃的是图卷积神经网络.由于图数据在现实世界中广泛存在，因此有效地完成图结构数据的学习具有很大的应用前景.目前出现的大多数图卷积神经网络模型基本都是浅层结构，过平滑问题成为制约该领域发展的瓶颈问题.本文提出了一种称为dri-GCN（Graph Convolutional Network via dropedge， residual and identity mapping）的图残差卷积深层网络模型，该模型集成了图剪边、初始残差和恒等映射技术.主要思想包括：利用图剪边技术增加学习数据的多样性，以防止学习过程中的过拟合现象；构建恒等映射下的初始残差网络，来扩展残差单元的学习路径，以削弱学习过程中的过平滑问题.实验结果表明，本文提出的dri-GCN模型可以帮助构建深层图卷积神经网络，通过网络层次的加深可以获得优于浅层网络的学习准确率.

关键词: 图神经网络, 图卷积神经网络, 剪边, 残差

Abstract:

Graph neural network (GNN) has become an important research field since 2005, and the most active branch is the graph convolutional network(GCN). In the real world, many applications are directly related to graphic data, so learning from graphic structure data is becoming more and more important, and can has a huge application prospect. However, most of the existing graph convolution neural network models can only support very few convoluted layers. This is mainly because of the over smoothing problem in GCNs, which has become one of the main bottlenecks in GCNs. In this paper, a novel GCN network: dri-GCN is proposed, which integrates the technologies of the Initial Residual, Identity Mapping and DropEdge into the traditional GCN. The main contributions include: using DropEdge to increase data diversity and prevent over-fitting; constructing the residual convolutional network under the constant mapping to extend the learning paths, which can effectively weaken the over smooth problem in GCNs. Experimental results show that dri-GCN model can help building the deeper graph convolution neural networks. There is no doubt that deeper GCNs can achieve better learning accuracy than shallower networks.

Key words: graph neural network, graph convolutional neural networks, DropEdge, residual network

中图分类号:

TP391

毛国君, 王者浩, 黄山, 王翔. 基于剪边策略的图残差卷积深层网络模型[J]. 电子学报, 2022, 50(9): 2205-2214.

MAO Guo-jun, WANG Zhe-hao, HUANG Shan, WANG Xiang. A Deep Residual Graph Convolution Network Based on Dropedge Method[J]. Acta Electronica Sinica, 2022, 50(9): 2205-2214.

图/表 15

图1 CNN和GNN在图像处理上的输入数据

图2 一个残差块的连接结构示意

图3 残差和恒等映射在GCN中工作原理示意

图4 dri_GCN运行框架

图5 dri_GCN网络的学习过程示意

表1 三个数据集的基本信息

数据集	节点数	边数	类别数	特征数
Cora	2708	5429	7	1433
Citeseer	3327	4732	6	3703
Pubmed	19717	44338	3	500

图6 数据集对应的部分无向连通图

表2 实验模型所用的参数

数据集	学习率lr	剪输出系数dropout	剪边系数p	残差系数α	权重控制系数 $λ$
Cora	0.01	0.6	0.9	0.1	0.5
Citeseer	0.01	0.6	0.9	0.1	0.5
Pubmed	0.01	0.6	0.98	0.1	0.5

表2 实验模型所用的参数

数据集	学习率lr	剪输出系数dropout	剪边系数p	残差系数α	权重控制系数 $λ$
Cora	0.01	0.6	0.9	0.1	0.5
Citeseer	0.01	0.6	0.9	0.1	0.5
Pubmed	0.01	0.6	0.98	0.1	0.5

表3 不同方法在Cora数据集上的分类准确率(%)

层数	模型
层数	GCN (G)	d_GCN (d+G)	GCNII (r+G)	dri_GCN (d+r+G)
2	81.1	82.8	80.4	80.6
4	80.3	82.0	81.3	81.4
8	69.5	75.8	82.6	82.8
16	64.9	75.7	84.0	84.2
32	60.3	62.5	84.5	84.5
64	28.7	49.5	84.5	84.5
128	—	—	84.3	84.6

表4 不同方法在Citeseer数据集上的分类准确率(%)

层数	模型
层数	GCN (G)	d_GCN (d+G)	GCNII (r+G)	dri_GCN (d+r+G)
2	70.8	72.3	65.9	66.2
4	67.6	70.6	66.1	66.5
8	30.2	61.4	69.3	70.0
16	18.3	57.2	71.7	72.0
32	25.0	41.6	72.2	72.5
64	20.0	34.4	72.1	72.2
128	—	—	72.0	72.3

表5 不同方法在Pubmed数据集上的分类准确率(%)

层数	模型
层数	GCN (G)	d_GCN (d+G)	GCNII (r+G)	dri_GCN (d+r+G)
2	79.0	79.6	78.1	77.8
4	76.5	79.4	78.3	78.4
8	61.2	78.1	79.1	79.1
16	40.9	78.5	79.2	79.3
32	22.4	77.0	79.3	79.7
64	35.3	61.5	79.2	79.3
128	—	—	79.1	79.3

图7 dri_GCN 在Cora数据集的精度对比

表6 Cora数据集p参数实验(%)

p值	层数
p值	2	4	8	16	32	64	128
p=0.9	80.7	81.4	83.0	84.4	84.9	84.5	84.4
p=0.8	81.0	81.1	82.6	84.2	84.2	84.0	83.5
p=0.7	80.4	81.6	82.2	83.7	83.9	83.1	83.1
p=0.6	81.5	79.5	80.9	82.2	82.7	81.9	82.4
p=0.5	81.3	79.4	80.8	81.9	82.1	81.9	82.1
p=0.4	80.7	76.5	79.7	81.5	81.4	81.9	81.7
p=0.3	80.1	75.4	79.3	79.9	80.9	81.3	80.9
p=0.2	80.0	72.4	79.1	79.9	80.0	80.6	80.1

图8 dri_GCN 损失函数对比

图9 dri_GCN 验证集准确率对比

参考文献 27

1	白铂, 刘玉婷, 马驰骋, 等. 图神经网络[J]. 中国科学: 数学, 2020, 50(3): 367-384.
	BAI B, LIU Y T, MA C C, et al. Graph neural network[J]. Scientia Sinica (Mathematica), 2020, 50(3): 367-384. (in Chinese)
2	钱榕, 张茹, 张克君, 等. 融合全局和局部特征的胶囊图神经网络[J]. 吉林大学学报(工学版), 2021, 51(3): 1048-1054.
	QIAN R, ZHANG R, ZHANG K J, et al. Capsule graph neural network based on global and local features fusion[J]. Journal of Jilin University (Engineering and Technology Edition), 2021, 51(3): 1048-1054. (in Chinese)
3	高留杰, 赵文, 张君福, 等. G2S: 基于语义块的知识图谱问答语义解析[J]. 电子学报, 2021, 49(6): 1132-1141.
	GAO L J, ZHAO W, ZHANG J F, et al. G2S: semantic segment based semantic parsing for question answering over knowledge graph[J]. Acta Electronica Sinica, 2021, 49(6): 1132-1141. (in Chinese)
4	HAMILTON W L, YING R, LESKOVEC J. Inductive representation learning on large graphs[C]//NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2017: 1025-1035.
5	WANG X L, YE Y F, GUPTA A. Zero-shot recognition via semantic embeddings and knowledge graphs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 6857-6866.
6	LIN W Y, JI S X, LI B C. Adversarial attacks on link prediction algorithms based on graph neural networks[C]//ASIA CCS'20: Proceedings of the 15th ACM Asia Conference on Computer and Communications Security. New York: ACM, 2020: 370-380.
7	车向北, 康文倩, 邓彬, 等. 一种基于图神经网络的SDN路由性能预测模型[J]. 电子学报, 2021, 49(3): 484-491.
	CHE X B, KANG W Q, DENG B, et al. A prediction model of SDN routing performance based on graph neural network[J]. Acta Electronica Sinica, 2021, 49(3): 484-491. (in Chinese)
8	ZITNIK M, AGRAWAL M, LESKOVEC J. Modeling polypharmacy side effects with graph convolutional networks[J]. Bioinformatics(Oxford, England), 2018, 34(13): i457-i466.
9	PEROZZI B, AL-RFOU R, SKIENA S. DeepWalk: online learning of social representations[C]//KDD'14: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2014: 701-710.
10	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2016-09-09). .
11	KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]//CVPR'14: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus:IEEE, 2014: 1725-1732.
12	SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. Piscataway: IEEE, 2016, 39(4): 640-651.
13	CHEN M, WEI Z, HUANG Z, et al. Simple and deep graph convolutional networks[C]//International Conference on Machine Learning. Vienna: ICML, 2020: 1725-1735.
14	XU K, LI C, TIAN Y, et al. Representation learning on graphs with jumping knowledge networks[C]//International Conference on Machine Learning. Stockholm: ICML, 2018: 5453-5462.
15	LI G H, MÜLLER M, THABET A, et al. DeepGCNs: can GCNs go as deep as CNNs?[C]//2019 IEEE/CVF International Conference on Computer Vision(ICCV). Piscataway: IEEE, 2019: 9266-9275.
16	WU H B, GU X D. Towards dropout training for convolutional neural networks[J]. Neural Networks, 2015, 71: 1-10.
17	DO T H, NGUYEN D M, BEKOULIS G, et al. Graph convolutional neural networks with node transition probability-based message passing and DropNode regularization[J]. Expert Systems With Applications, 2021, 174: 114711.
18	RONG Y, HUANG W B, XU T Y, et al. DropEdge: towards deep graph convolutional networks on node classification[EB/OL]. (2019-07-25). .
19	SRIVASTAVA N, HINTON G E, KRIZHEVSKY A, et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
20	SCARSELLI F, GORI M, TSOI A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80.
21	BRUNA J, ZAREMBA W, SZLAM A, et al. Spectral networks and locally connected networks on graphs[EB/OL].(2013-12-21). .
22	郭嘉琰, 李荣华, 张岩, 等. 基于图神经网络的动态网络异常检测算法[J]. 软件学报, 2020, 31(3): 748-762.
	GUO J Y, LI R H, ZHANG Y, et al. Graph neural network based anomaly detection in dynamic networks[J]. Journal of Software, 2020, 31(3): 748-762. (in Chinese)
23	WANG R, HUANG C Y, WANG X Y. Global relation reasoning graph convolutional networks for human pose estimation[J]. IEEE Access, 2020, 8: 38472-38480.
24	YU K, JIANG H, LI T, et al. Data fusion oriented graph convolution network model for rumor detection[J]. IEEE Transactions on Network and Service Management, 2020, 17(4): 2171-2181.
25	HE K M, ZHANG X Y, REN S Q, et al. Identity Mappings in Deep Residual Networks[C]//European Conference on Computer Vision. Cham: Springer, 2016: 630-645.
26	OONO K, SUZUKI T. Graph neural networks exponentially lose expressive power for node classification[EB/OL]. (2019-05-27). .
27	SEN P, NAMATA G, BILGIC M, et al. Collective classification in network data[J]. AI Magazine, 2008, 29(3): 93.

[1]	王相海, 赵晓阳, 王鑫莹, 赵克云, 宋传鸣. 非抽取小波边缘学习深度残差网络的单幅图像超分辨率重建[J]. 电子学报, 2022, 50(7): 1753-1765.
[2]	胡向东, 李之涵. 基于胶囊网络的工业互联网入侵检测方法[J]. 电子学报, 2022, 50(6): 1457-1465.
[3]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.
[4]	袁冠, 邴睿, 刘肖, 代伟, 张艳梅, 蔡卓. 基于时空图神经网络的手势识别[J]. 电子学报, 2022, 50(4): 921-931.
[5]	刘小洋, 刘加苗, 刘超, 张宜浩. 融合字符级滑动窗口和深度残差网络的僵尸网络DGA域名检测方法[J]. 电子学报, 2022, 50(1): 250-256.
[6]	张聿远, 张立民, 闫文君. 基于深度多级残差网络的低信噪比下空频分组码识别方法[J]. 电子学报, 2022, 50(1): 79-88.
[7]	苏志刚, 武瑞, 郝敬堂. 面向无源定位的直达波传感器识别算法[J]. 电子学报, 2021, 49(6): 1178-1186.
[8]	乔思波, 庞善臣, 王敏, 翟雪, 于世行, 丁桐. 基于残差混合注意力机制的脑部CT图像分类卷积神经网络模型[J]. 电子学报, 2021, 49(5): 984-991.
[9]	车向北, 康文倩, 邓彬, 杨柯涵, 李剑. 一种基于图神经网络的SDN路由性能预测模型[J]. 电子学报, 2021, 49(3): 484-491.
[10]	江泽涛, 钱艺, 伍旭, 张少钦. 一种基于ARD‑GAN的低照度图像增强方法[J]. 电子学报, 2021, 49(11): 2160-2165.
[11]	周涛, 霍兵强, 陆惠玲, 任海玲. 残差神经网络及其在医学图像处理中的应用研究[J]. 电子学报, 2020, 48(7): 1436-1447.
[12]	秦鑫, 黄洁, 查雄, 骆丽萍, 胡德秀. 基于扩张残差网络的雷达辐射源信号识别[J]. 电子学报, 2020, 48(3): 456-462.
[13]	罗会兰, 敖阳, 袁璞. 一种生成对抗网络用于图像修复的方法[J]. 电子学报, 2020, 48(10): 1891-1898.
[14]	曾伟辉, 李淼, 李增, 熊焰. 基于高阶残差和参数共享反馈卷积神经网络的农作物病害识别[J]. 电子学报, 2019, 47(9): 1979-1986.
[15]	郑兴华, 孙喜庆, 吕嘉欣, 鲜征征, 李磊. 基于深度学习和智能规划的行为识别[J]. 电子学报, 2019, 47(8): 1661-1668.