一种Hadoop Yarn的资源调度方法研究

doi:10.3969/j.issn.0372-2112.2016.05.002

电子学报 ›› 2016, Vol. 44 ›› Issue (5): 1017-1024.DOI: 10.3969/j.issn.0372-2112.2016.05.002

一种Hadoop Yarn的资源调度方法研究

李媛祯¹, 杨群¹, 赖尚琦², 李博涵¹

1. 南京航空航天大学计算机科学与技术学院, 江苏南京 210016;
2. 香港大学计算机科学与技术系, 香港

收稿日期:2014-11-15 修回日期:2015-04-09 出版日期:2016-05-25
- 通讯作者:
- 杨群
- 作者简介:
- 李媛祯 女,1990年出生,硕士研究生,研究方向为资源调度、并行计算.E-mail:liyuanzhen0724@163.com
- 基金资助:
- 国家自然科学基金 (No.41301407); 江苏省自然科学基金 (No.BK20130819)

A Study on Scheduling Method of Hadoop Yarn

LI Yuan-zhen¹, YANG Qun¹, LAI Shang-qi², LI Bo-han¹

1. Computer Science and Technology College, Nanjing University of Aeronautics and Astronautics, Nanjing, Jiangsu 210016, China;
2. Department of Computer Science, The University of Hong Kong, Hong Kong, China

Received:2014-11-15 Revised:2015-04-09 Online:2016-05-25 Published:2016-05-25

摘要/Abstract

摘要：

针对Hadoop Yarn资源调度问题,为提高集群作业执行效率,提出一种基于蚁群算法与粒子群算法的自适应Hadoop资源调度算法SRSAPH.SRSAPH中,通过Hadoop Yarn跳通信机制获取负载、内存、CPU速度等属性信息初始化信息素矩阵;同时,将粒子群算法的自我认知能力与社会认知能力引入到蚁群算法,提高算法的收敛速度;此外,根据蚁群算法全局最优解的波动趋势动态调整信息素挥发系数,提高解的精度.实验表明,采用SRSAPH进行资源调度,集群的作业执行时间缩短至少10%.

关键词: 资源调度, 蚁群算法, 粒子群算法, Hadoop Yarn

Abstract:

In view of the resource scheduling problem of Hadoop Yarn, to improve the execution efficiency of the cluster job, we propose a Self-adapt Resource Scheduling algorithm based on Ant Colony Algorithm and Particle Swarm Algorithm in Hadoop (SRSAPH).In SRSAPH, we initialize the pheromone matrix of SRSAPH by using the attribute information of load, memory, and CPU speed obtained through the heartbeat message transfer mechanism.Meanwhile, we introduce the self-cognitive ability and social cognition ability of particle swarm algorithm into the ant colony algorithm to speed up the rate of convergence of the algorithm.Moreover, we dynamically adjust the pheromone evaporation rate based on the fluctuation trends of global optimal solution to enhance the accuracy of the solutions.Experimental result shows that by using SRSAPH in resource scheduling, the execution time of cluster job is shorten by 10%.

Key words: resource scheduling, ant colony algorithm, particle swarm algorithm, Hadoop Yarn

中图分类号:

TP301.6

李媛祯, 杨群, 赖尚琦, 李博涵. 一种Hadoop Yarn的资源调度方法研究[J]. 电子学报, 2016, 44(5): 1017-1024.

LI Yuan-zhen, YANG Qun, LAI Shang-qi, LI Bo-han. A Study on Scheduling Method of Hadoop Yarn[J]. Acta Electronica Sinica, 2016, 44(5): 1017-1024.

参考文献

[1] Elghoneimy E,Bouhali O,Alnuweiri H.Resource allocation and scheduling in cloud computing[A].2012 International Conference on Computing,Networking and Communications[C].Hawaii:IEEE,2012.309-314.
[2] Gokilavani M,SelviS,Udhayakumar C.A survey on resource allocation and task scheduling algorithms in cloud environment[J].International Journal of Engineering and Innovative Technology,2013,3(4):173-179.
[3] Chauhan J.Simulation and performance evaluation of hadoop capacity scheduler[D].Saskatoon:University of Saskatchewan,2013:93-96.
[4] Fair Scheduler Guide[R/OL].http://hadoop.apache.org/common/docs/r0.20.2/fair_shcehduler.html,2013-08-04.
[5] Rasooli A,Down D G.A hybrid scheduling approach for scalable heterogeneous hadoop systems[A].2012 High Performance Computing,Networking,Storage and Analysis[C].Salt Lake City:IEEE,2012.1284-1291.
[6] Rasooli A,Down D G.COSHH:A classification and optimization based scheduler for heterogeneous Hadoop systems[J].Future Generation Computer Systems,2014,36(7):1-15.
[7] Yazir Y O,Matthews C,Farahbod R,et al.Dynamic resource allocation in computing clouds using distributed multiple criteria decision analysis[A].3th IEEE 2010 International Conference on Cloud Computing[C].Florida:IEEE,2010.91-98.
[8] Yue Z,Xu Q.Resource allocation and scheduling theory based on distributed environment[A].16^th International Conference on Advanced Communication Technology[C].PyeongChang Korea:IEEE,2014.1124-1128.
[9] Ergu D,Kou G,Peng Y,et al.The analytic hierarchy process:task scheduling and resource allocation in cloud computing environment[J].The Journal of Supercomputing,2013,64(3):835-848.
[10] Senouci A B,Eldin N N.Use of genetic algorithms in resource scheduling of construction projects[J].Journal of Construction Engineering and Management,2004,130(6):869-877.
[11] Gon?alves J F,Mendes J J M,Resende M G C.A genetic algorithm for the resource constrained multi-project scheduling problem[J].European Journal of Operational Research,2008,189(3):1171-1190.
[12] Merkle D,Middendorf M,Schmeck H.Ant colony optimization for resource-constrained project scheduling[J].IEEE Transactions on Evolutionary Computation,2002,6(4):333-346.
[13] Yin P Y,Wang J Y.Ant colony optimization for the nonlinear resource allocation problem[J].Applied mathematics and computation,2006,174(2):1438-1453.
[14] Chang R S,Chang J S,Lin P S.An ant algorithm for balanced job scheduling in grids[J].Future Generation Computer Systems,2009,25(1):20-27.
[15] Gandomi A H,Yun G J,Yang X S,et al.Chaos-enhanced accelerated particle swarm optimization[J].Communications in Nonlinear Science and Numerical Simulation,2013,18(2):327-340.
[16] Chaharsooghi S K,Meimand Kermani A H.An effective ant colony optimization algorithm (ACO) for multi-objective resource allocation problem (MORAP)[J].Applied Mathematics and Computation,2008,200(1):167-177.
[17] Achary R,Vityanathan V,Raj P,et al.Dynamic Job Scheduling Using Ant Colony Optimization for Mobile Cloud Computing[M].Switzerland:Springer International Publishing,2015:71-82.
[18] Zhang Z,Zhang N,Feng Z.Multi-satellite control resource scheduling based on ant colony optimization[J].Expert Systems with Applications,2014,41(6):2816-2823.
[19] Dorigo M,Blum C.Ant colony optimization theory:A survey[J].Theoretical Computer Science,2005,344(1):243-278.
[20] Hengliang S,Guanyi B,Zhenmin T.Aco algorithm-based parallel job scheduling investigation on hadoop[J].International Journal of Digital Content Technology and Its Applications,2011,5(7):283-288.
[21] 数据堂.2013年11月中国新闻汇总[DB/OL].http://www.datatang.com/data/45718,2013-12-23.

一种Hadoop Yarn的资源调度方法研究

A Study on Scheduling Method of Hadoop Yarn

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	熊小峰, 黄淳岚, 乐光学, 戴亚盛, 杨晓慧, 杨忠明. 边缘计算中基于综合信任评价的任务卸载策略[J]. 电子学报, 2022, 50(9): 2134-2145.
[2]	李纪三, 纪彦星, 曹鼎, 刘溶, 任渊. 基于广义时间窗的旋转相控阵雷达资源调度算法[J]. 电子学报, 2022, 50(5): 1050-1057.
[3]	周泓岑, 白恒, 才振功, 蔡亮, 顾静, 汤志敏. 基于LSTM和GRNN的容器配额优化算法[J]. 电子学报, 2022, 50(2): 366-373.
[4]	孙全明, 曲志坚, 任崇广. 基于粒子群优化和LightGBM的情景感知多式联运推荐[J]. 电子学报, 2021, 49(5): 894-903.
[5]	周熙炜, 汪贵平, 王会峰, 尚宵. 车-路协同电动汽车动态无线充电的路权调度控制[J]. 电子学报, 2021, 49(5): 904-911.
[6]	郝诗雅, 杨媛媛, 董怡靖, 赵晓旭, 陈嘉兴. 水声传感器网络中粒子群与蒙特卡罗优化的移动定位算法[J]. 电子学报, 2021, 49(2): 292-299.
[7]	王毅, 李晓梦, 耿国华, 周琳, 段焱中. 基于直觉模糊熵的混合粒子群优化算法[J]. 电子学报, 2021, 49(12): 2381-2389.
[8]	陈健瑞, 王景璟, 侯向往, 方政儒, 杜军, 任勇. 挺进深蓝：从单体仿生到群体智能[J]. 电子学报, 2021, 49(12): 2458-2467.
[9]	刘润滋, 吴伟华, 张文柱, 周笛, 张琰. 基于图学习的密集空间网络传输资源调度方法[J]. 电子学报, 2021, 49(11): 2133-2137.
[10]	孙鑫威, 纪爱敏, 陈曦晖, 林新海, 许行. 强噪声背景下动车组轴承微弱故障信号检测[J]. 电子学报, 2021, 49(11): 2217-2224.
[11]	廖伟志, 夏小云, 贾小军. 基于蚁群算法的多路径覆盖测试数据生成[J]. 电子学报, 2020, 48(7): 1330-1342.
[12]	郝少伟, 李勇军, 赵尚弘, 王蔚龙, 王星宇. 基于改进粒子群算法的多载波NOMA功率分配策略[J]. 电子学报, 2020, 48(10): 2009-2016.
[13]	周平, 殷波, 邱雪松, 郭少勇, 孟洛明. 面向服务可靠性的云资源调度方法[J]. 电子学报, 2019, 47(5): 1036-1043.
[14]	许凯波, 鲁海燕, 黄洋, 胡士娟. 基于双层蚁群算法和动态环境的机器人路径规划方法[J]. 电子学报, 2019, 47(10): 2166-2176.
[15]	邓先礼, 魏波, 曾辉, 桂凌, 夏学文. 基于多种群的自适应迁移PSO算法[J]. 电子学报, 2018, 46(8): 1858-1865.