基于深度强化学习的云边协同计算迁移研究

doi:10.12263/DZXB.20200131

电子学报 ›› 2021, Vol. 49 ›› Issue (1): 157-166.DOI: 10.12263/DZXB.20200131

基于深度强化学习的云边协同计算迁移研究

陈思光^1,2, 陈佳民^1,3, 赵传信²

1. 南京邮电大学江苏省通信与网络技术工程研究中心, 江苏南京 210003;
2. 安徽师范大学网络与信息安全安徽省重点实验室, 安徽芜湖 241002;
3. 南京邮电大学江苏省宽带无线通信和物联网重点实验室, 江苏南京 210003

收稿日期:2020-01-20 修回日期:2020-03-19 出版日期:2021-01-25
- 通讯作者:
- 陈思光
- 作者简介:
- 陈佳民 男,1995年12月出生于江苏省南通市.南京邮电大学在读硕士研究生.主要研究方向为边缘计算和机器学习.E-mail:18252010292@163.com;赵传信 男,1977年生于安徽凤阳.现为安徽师范大学教授、博士生导师.主要研究方向为物联网、边缘计算及网络资源分配优化等.E-mail:zhaocx@ahnu.edu.cn
- 基金资助:
- 国家自然科学基金 (No.61971235，No.61771258，No.61871412）; 江苏省"六大人才高峰"高层次人才项目 (No.XYDXXJS-044）; 江苏省"333高层次人才培养工程"; 南京邮电大学"1311"人才计划; 中国博士后科学基金 (面上一等） (No.2018M630590）; 南京邮电大学国家自然科学基金孵化项目 (No.NY217057，No.NY218058）; 网络与信息安全安徽省重点实验室开放课题 (No.AHNIS2020001）; 江苏省通信与网络技术工程研究中心开放课题重点项目 (No.JSGCZX17011）; 赛尔网络下一代互联网技术创新项目 (No.NGII20190702）

Deep Reinforcement Learning Based Cloud-Edge Collaborative Computation Offloading Mechanism

CHEN Si-guang^1,2, CHEN Jia-min^1,3, ZHAO Chuan-xin²

1. Jiangsu Engineering Research Center of Communications and Network Technology, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu 210003, China;
2. Anhui Provincial Key Laboratory of Network and Information Security, Anhui Normal University, Wuhu, Anhui 241002, China;
3. Jiangsu Key Lab of Broadband Wireless Communication and Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu 210003, China

Received:2020-01-20 Revised:2020-03-19 Online:2021-01-25 Published:2021-01-25

摘要/Abstract

摘要： 基于单一边缘节点计算、存储资源的有限性及大数据场景对高效计算服务的需求，本文提出了一种基于深度强化学习的云边协同计算迁移机制.具体地，基于计算资源、带宽和迁移决策的综合性考量，构建了一个最小化所有用户任务执行延迟与能耗权重和的优化问题.基于该优化问题提出了一个异步云边协同的深度强化学习算法，该算法充分利用了云边双方的计算能力，可有效满足大数据场景对高效计算服务的需求；同时，面向边缘云中边缘节点所处环境的多样及动态变化性，该算法能自适应地调整迁移策略以实现系统总成本的最小化.最后，大量的仿真结果表明本文所提出的算法具有收敛速度快、鲁棒性高等特点，并能够以最低的计算成本获得近似贪心算法的最优迁移决策.

关键词: 深度强化学习, 边缘计算, 计算迁移, 资源分配, 能量消耗

Abstract: Based on the computation and storage resources limitation of single edge node and the demand for efficient computing services in big data scenarios,this paper proposes a deep reinforcement learning based cloud-edge collaborative computation offloading mechanism.Specifically,based on a comprehensive consideration of computing resources,bandwidth and offloading policy,an optimization problem is formulated to minimize the weight sum of execution delay and energy consumption of all user tasks.An asynchronous cloud-edge collaborative deep reinforcement learning (ACEC-DRL) algorithm is proposed to solve such optimization problem.This algorithm can effectively satisfy the demand of efficient computing services in big data scenario by jointly leveraging the computation capabilities of cloud and edge nodes.Meanwhile,under the various and dynamic environments of edge nodes in the edge cloud,this algorithm can adaptively adjust offloading policy to achieve the minimization of system cost.Finally,the extensive simulation results show that the proposed ACEC-DRL algorithm has the characteristics of fast convergence rate and high robustness,and its optimal offloading policy closely approximates to the solution of greedy algorithm with the lowest computation cost.

Key words: deep reinforcement learning, edge computing, computation offloading, resource allocation, energy consumption

中图分类号:

TP393

陈思光, 陈佳民, 赵传信. 基于深度强化学习的云边协同计算迁移研究[J]. 电子学报, 2021, 49(1): 157-166.

CHEN Si-guang, CHEN Jia-min, ZHAO Chuan-xin. Deep Reinforcement Learning Based Cloud-Edge Collaborative Computation Offloading Mechanism[J]. Acta Electronica Sinica, 2021, 49(1): 157-166.

参考文献

[1] Wang C,Liang C,Yu F R,et al.Computation offloading and resource allocation in wireless cellular networks with mobile edge computing[J].IEEE Transactions on Wireless Communications,2017,16(8):4924-4938.
[2] Haber E E,Nguyen T M,Assi C.Joint optimization of computational cost and devices energy for task offloading in multi-tier edge-clouds[J].IEEE Transactions on Communications,2019,67(5):3407-3421.
[3] Chen S,Zheng Y,Lu W,et al.Energy-optimal dynamic computation offloading for industrial IoT in fog computing[J].IEEE Transactions on Green Communications and Networking,2019,DOI:10.1109/TGCN.2019.2960767.
[4] Chen S,Zheng Y,Wang K,et al.Delay guaranteed energy-efficient computation offloading for industrial IoT in fog computing[A].Proceedings of IEEE International Conference on Communications (ICC)[C].Shanghai,China:IEEE,2019.1-6.
[5] Lei L,Xu H,Xiong X,et al.Joint computation offloading and multiuser scheduling using approximate dynamic programming in NB-IoT edge computing system[J].IEEE Internet of Things Journal,2019,6(3):5345-5362.
[6] Chen S,Zhu X,Zhang H,et al.Efficient privacy preserving data collection and computation offloading for fog-assisted IoT[J].IEEE Transactions on Sustainable Computing,2020,DOI:10.1109/TSUSC.2020.2968589.
[7] Yu S,Wang X,Langar R.Computation offloading for mobile edge computing:a deep learning approach[A].Proceedings of IEEE International Symposium on Personal,Indoor and Mobile Radio Communications (PIMRC)[C].Montreal,Canada:IEEE,2017.1-6.
[8] Li L,Ota K,Dong M.Deep learning for smart industry:efficient manufacture inspection system with fog computing[J].IEEE Transactions on Industrial Informatics,2018,14(10):4665-4673.
[9] Zhu X,Chen S,Chen S,et al.Energy and delay co-aware computation offloading with deep learning in fog computing networks[A].Proceedings of IEEE International Performance Computing and Communications Conference (IPCCC)[C].London,UK:IEEE,2019.1-6.
[10] Dab B,Aitsaadi N,Langar R.Q-learning algorithm for joint computation offloading and resource allocation in edge cloud[A].Proceedings of IFIP/IEEE Symposium on Integrated Network and Service Management (IM)[C].Arlington,USA:IEEE,2019.8-12.
[11] Liu X,Qin Z,Gao Y.Resource allocation for edge computing in IoT networks via reinforcement learning[A].Proceedings of IEEE International Conference on Communications (ICC)[C].Shanghai,China:IEEE,2019.20-24.
[12] Huang L,Feng X,Qian L,et al.Deep reinforcement learning-based task offloading and resource allocation for mobile edge computing[A].Proceedings of International Conference on Machine Learning and Intelligent Communications (MLICOM)[C].Nanjing,China:Springer Verlag,2018.33-42.
[13] Meng H,Chao D,Huo R,et al.Deep reinforcement learning based delay-sensitive task scheduling and resource management algorithm for multi-user mobile-edge computing systems[A].Proceedings of International Conference on Mathematics and Artificial Intelligence (ICMAI)[C].Chengdu,China:ACM,2019.66-70.
[14] Wei Y,Yu F R,Song M,et al.Joint optimization of caching,computing,and radio resources for fog-enabled IoT using natural actor-critic deep reinforcement learning[J].IEEE Internet of Things Journal,2019,6(2):2061-2073.
[15] Chen X,Zhang H,Wu C,et al.Performance optimization in mobile-edge computing via deep reinforcement learning[A].Proceedings of IEEE Vehicular Technology Conference (VTC-Fall)[C].Chicago,USA:IEEE,2018.27-30.
[16] Huang L,Bi S,Zhang Y J.Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[J].IEEE Transactions on Mobile Computing,2019,DOI:10.1109/TMC.2019.2928811.
[17] Mnih V,Badia A P,Mirza L,et al.Asynchronous methods for deep reinforcement learning[A].Proceedings of the 33^rd International Conference on Machine Learning (ICML)[C].New York,USA:IMLS,2016.1928-1937.
[18] Zeinali Y,Story B A.Competitive probabilistic neural network[J].Integrated Computer-Aided Engineering,2017,24(2):105-118.
[19] Kaing D,Medsker L.Competitive hybrid ensemble using neural network and decision tree[J].Advances in Intelligent Systems and Computing,2018,648(28):147-155.

基于深度强化学习的云边协同计算迁移研究

Deep Reinforcement Learning Based Cloud-Edge Collaborative Computation Offloading Mechanism

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	熊小峰, 黄淳岚, 乐光学, 戴亚盛, 杨晓慧, 杨忠明. 边缘计算中基于综合信任评价的任务卸载策略[J]. 电子学报, 2022, 50(9): 2134-2145.
[2]	段世红, 何昊, 徐诚, 殷楠, 王然. 未知环境下基于深度序列蒙特卡罗树搜索的信源导航方法[J]. 电子学报, 2022, 50(7): 1744-1752.
[3]	宋佰霖, 许华, 齐子森, 饶宁, 彭翔. 一种基于深度强化学习的协同通信干扰决策算法[J]. 电子学报, 2022, 50(6): 1301-1309.
[4]	饶宁, 许华, 蒋磊, 宋佰霖, 史蕴豪. 基于多智能体深度强化学习的分布式协同干扰功率分配算法[J]. 电子学报, 2022, 50(6): 1319-1330.
[5]	易令, 李泽平. 基于深度强化学习的码率自适应算法研究[J]. 电子学报, 2022, 50(5): 1192-1200.
[6]	李世银, 鲁姗妹, 马帅, 张凡, 徐子涵, 王洪梅, 李宗艳, 熊海良. 有限字符输入DCO-OFDMA系统的资源分配算法研究[J]. 电子学报, 2022, 50(3): 574-584.
[7]	宋政育, 郝媛媛, 孙昕. 低轨卫星协作边缘计算任务迁移和资源分配算法[J]. 电子学报, 2022, 50(3): 567-573.
[8]	刘焕淋, 胡俊岭, 任杰, 胡会霞, 唐畅, 陈浩楠. 基于光路负载均衡和邻域匹配的串扰感知资源分配方法[J]. 电子学报, 2022, 50(11): 2746-2753.
[9]	赵雄文, 张钰, 秦鹏, 王晓晴, 耿绥燕, 宋俊元, 刘瑶, 李思峰. 空天地一体化无线光通信网络关键技术及其发展趋势[J]. 电子学报, 2022, 50(1): 1-17.
[10]	肖硕, 黄珍珍, 张国鹏, 杨树松, 江海峰, 李天旭. 基于SAC的多智能体深度强化学习算法[J]. 电子学报, 2021, 49(9): 1675-1681.
[11]	廖勇, 田肖懿, 蔡志镕, 花远肖, 韩庆文. 面向C-V2I的基于边缘计算的智能信道估计[J]. 电子学报, 2021, 49(5): 833-842.
[12]	许新操, 刘凯, 刘春晖, 蒋豪, 郭松涛, 吴巍炜. 基于势博弈的车载边缘计算信道分配方法[J]. 电子学报, 2021, 49(5): 851-860.
[13]	刘雷, 陈晨, 冯杰, 肖婷婷, 裴庆祺. 车载边缘计算卸载技术研究综述[J]. 电子学报, 2021, 49(5): 861-871.
[14]	樊凌雁, 燕武, 陈晞涵. 一种低传输能耗的无人机数据传输设计[J]. 电子学报, 2021, 49(3): 573-577.
[15]	姚友杰, 钱斌, 董钰明, 胡蓉, 王凌. 基于EDA的绿色零等待作业车间调度问题求解[J]. 电子学报, 2021, 49(2): 225-232.