电子学报 ›› 2016, Vol. 44 ›› Issue (5): 1017-1024.DOI: 10.3969/j.issn.0372-2112.2016.05.002

• 学术论文 • 上一篇    下一篇

一种Hadoop Yarn的资源调度方法研究

李媛祯1, 杨群1, 赖尚琦2, 李博涵1   

  1. 1. 南京航空航天大学计算机科学与技术学院, 江苏南京 210016;
    2. 香港大学计算机科学与技术系, 香港
  • 收稿日期:2014-11-15 修回日期:2015-04-09 出版日期:2016-05-25 发布日期:2016-05-25
  • 通讯作者: 杨群
  • 作者简介:李媛祯 女,1990年出生,硕士研究生,研究方向为资源调度、并行计算.E-mail:liyuanzhen0724@163.com
  • 基金资助:

    国家自然科学基金(No.41301407);江苏省自然科学基金(No.BK20130819)

A Study on Scheduling Method of Hadoop Yarn

LI Yuan-zhen1, YANG Qun1, LAI Shang-qi2, LI Bo-han1   

  1. 1. Computer Science and Technology College, Nanjing University of Aeronautics and Astronautics, Nanjing, Jiangsu 210016, China;
    2. Department of Computer Science, The University of Hong Kong, Hong Kong, China
  • Received:2014-11-15 Revised:2015-04-09 Online:2016-05-25 Published:2016-05-25

摘要:

针对Hadoop Yarn资源调度问题,为提高集群作业执行效率,提出一种基于蚁群算法与粒子群算法的自适应Hadoop资源调度算法SRSAPH.SRSAPH中,通过Hadoop Yarn跳通信机制获取负载、内存、CPU速度等属性信息初始化信息素矩阵;同时,将粒子群算法的自我认知能力与社会认知能力引入到蚁群算法,提高算法的收敛速度;此外,根据蚁群算法全局最优解的波动趋势动态调整信息素挥发系数,提高解的精度.实验表明,采用SRSAPH进行资源调度,集群的作业执行时间缩短至少10%.

关键词: 资源调度, 蚁群算法, 粒子群算法, Hadoop Yarn

Abstract:

In view of the resource scheduling problem of Hadoop Yarn, to improve the execution efficiency of the cluster job, we propose a Self-adapt Resource Scheduling algorithm based on Ant Colony Algorithm and Particle Swarm Algorithm in Hadoop (SRSAPH).In SRSAPH, we initialize the pheromone matrix of SRSAPH by using the attribute information of load, memory, and CPU speed obtained through the heartbeat message transfer mechanism.Meanwhile, we introduce the self-cognitive ability and social cognition ability of particle swarm algorithm into the ant colony algorithm to speed up the rate of convergence of the algorithm.Moreover, we dynamically adjust the pheromone evaporation rate based on the fluctuation trends of global optimal solution to enhance the accuracy of the solutions.Experimental result shows that by using SRSAPH in resource scheduling, the execution time of cluster job is shorten by 10%.

Key words: resource scheduling, ant colony algorithm, particle swarm algorithm, Hadoop Yarn

中图分类号: