一种基于动态拓扑的流计算性能优化方法及其在Storm中的实现

doi:10.3969/j.issn.0372-2112.2020.05.007

电子学报 ›› 2020, Vol. 48 ›› Issue (5): 878-890.DOI: 10.3969/j.issn.0372-2112.2020.05.007

一种基于动态拓扑的流计算性能优化方法及其在Storm中的实现

陆佳炜¹, 吴涵¹, 陈烘², 张元鸣¹, 梁倩卉³, 肖刚¹

1. 浙江工业大学计算机科学与技术学院, 浙江杭州 310023;
2. 阿里巴巴基础架构事业部大数据计算与服务团队, 浙江杭州 310011;
3. 南洋理工大学计算机科学与工程学院, 新加坡 637457

收稿日期:2019-07-02 修回日期:2019-11-13 出版日期:2020-05-25 发布日期:2020-05-25
通讯作者: 肖刚
作者简介:陆佳炜 男,1981年9月出生于浙江湖州,讲师,研究方向为大数据,云计算、服务计算.E-mail:viivan@zjut.edu.cn;吴涵男,1995年2月出生于福建顺昌,硕士研究生,研究方向为大数据,云计算.E-mail:wuhan@zjut.edu.cn
基金资助:
国家自然科学基金（No.61976193）；浙江省自然科学基金（No.LY19F020034）；浙江省重点研发计划项目（No.2018C01064）

A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm

LU Jia-wei¹, WU Han¹, CHEN Hong², ZHANG Yuan-ming¹, LIANG Qian-hui³, XIAO Gang¹

1. Department of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, Zhejiang 310023, China;
2. Team of Big Data Computing and Service, Department of Infrastructure Business, Alibaba, Hangzhou, Zhejiang 310011, China;
3. School of Computer Science and Engineering, Nanyang Technological University, Singapore 637457, Singapore

Received:2019-07-02 Revised:2019-11-13 Online:2020-05-25 Published:2020-05-25

摘要/Abstract

摘要： 响应性和稳定性一直是流式计算中两个至关重要的问题，而流计算系统在过载时常常表现出数据计算延迟增加和拓扑不稳定的现象，无法适应数据负载的动态变化.针对这一问题本文研究提出了一种基于动态拓扑的流计算性能优化方法，主要包括：（1）动态逐级反压：拓扑中的任务可以根据当前自身负载情况，动态调整上游向其发送数据的速率.（2）无状态拓扑数据重放：拓扑不维持数据的计算状态，尽可能地实现数据容错.（3）自适应拓扑替换：在拓扑不暂停的情况下对任务并发度进行自发调整.（4）延迟持久化队列：拓扑中对磁盘的IO读写被延迟到数据处理之外，减缓IO高频阻塞对流计算系统的影响.本文在Apache Storm中实现了以上四种方案，性能测试结果表明优化后的流计算系统与Storm默认实现相比，不仅增强了大数据动态匹配能力，而且在最优情况下改善了17%的吞吐量，并提升了约20%的数据处理速度.

关键词: 数据流拓扑, 流计算, 大数据, 流计算系统, 性能优化

Abstract: Responsiveness and stability have always been two important problems in stream computing.However,as the scale of data being processed in real-time has increased,along with an increase in the data processing latency and topology instability of stream computing,many limitations of stream processing system have become apparent.Aiming at these problems,we present a performance optimization method based on dynamic topology for stream computing:(1) Dynamic step-by-step backpressure:the task in the topology can dynamically adjust the rate of upstream data transmission according to the current load.(2) Stateless topology data replay:topology can achieve data fault tolerance autonomously without maintaining the calculation of data state.(3) Adaptive topology replacement:no need for topology to suspend,the system can adjust the task concurrency spontaneously.(4) Delayed persistent queue:it delays the IO reading and writing in the disk out of the data processing,which mitigates the impact of IO high-frequency blocking in stream computing system.In this paper,the four methods are implemented in Apache Storm.The experimental results show that the optimized system not only enhances the dynamic matching capability of big data,but also achieves 17% higher throughput and 20% better data processing speed in the best case.

Key words: data stream topology, stream computing, big data, stream computing system, performance optimization

中图分类号:

TP311.13

陆佳炜, 吴涵, 陈烘, 张元鸣, 梁倩卉, 肖刚. 一种基于动态拓扑的流计算性能优化方法及其在Storm中的实现[J]. 电子学报, 2020, 48(5): 878-890.

LU Jia-wei, WU Han, CHEN Hong, ZHANG Yuan-ming, LIANG Qian-hui, XIAO Gang. A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm[J]. Acta Electronica Sinica, 2020, 48(5): 878-890.

参考文献

[1] Shieh CK,Huang SW,Sun LD,et al.A topology-based scaling mechanism for Apache Storm[J].International Journal of Network Management,2016,27(3):63-68.
[2] Dean J,Ghemawat S.MapReduce:A flexible data processing tool[J].Communications of the ACM,2010,53(1):72-77.
[3] 孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.
[4] Akidau T,Balikov A,Bekiro,et al.MillWheel:fault-tolerant stream processing at internet scale[J].Proceedings of the Vldb Endowment,2013,6(11):1033-1044.
[5] Gulisano V,Jimenez-Peris R,Patino-Martinez M,et al.StreamCloud:An elastic and scalable data streaming system[J].IEEE Transactions on Parallel & Distributed Systems,2012,23(12):2351-2365.
[6] Shkapsky A,Yang M,Interlandi M,et al.Big data analytics with datalog queries on spark[A].Proceedings of the 2016 International Conference on Management of Data(SIGMOD2016)[C].USA:2016.1135-1149.
[7] Nair LR,Shetty SD,Shetty SD.Applying spark based machine learning model on streaming big data for health status prediction[J].Computers & Electrical Engineering,2018,65(1):393-399.
[8] Barber R,Garcia-Arellano C,Grosman R,et al.Evolving databases for new-gen big data applications[A].The Eighth Biennial Conference on Innovative Data Systems Research(CIDR 2017)[C].USA:2017.
[9] 欧阳建权,周勇,唐欢容.基于Storm的在线序列极限学习机的气象预测模型[J].计算机研究与发展,2017,54(8):1736-1743.
[10] Heinze T,Aniello L,Querzoni L,et al.Tutorial:Cloud-based data stream processing[A].Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems[C].Mumbai,India:2014.238-245.
[11] Iqbal MH,Soomro TR.Big data analysis:apache storm perspective[J].International Journal of Computer Trends & Technology,2015,19(1):9-14.
[12] Xhafa F,Naranjo V,Caballé S.Processing and analytics of big data streams with yahoo!S4[A].2015 IEEE 29th International Conference on Advanced Information Networking and Applications(AINA-2015)[C].South Korea:2015.263-270.
[13] Borthakur D,Gray J,Sarma JS,et al.Apache hadoop goes realtime at Facebook[A].Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data(SIGMOD 2011)[C].Greece:2011.1071-1080.
[14] Qian Z,He Y,Su C,et al.TimeStream:reliable stream computation in the cloud[A].Proceedings of the 8th ACM European Conference on Computer Systems(EuroSys 2013)[C].Czech Republic:2013.1-14.
[15] Dean J,Barroso LA.The tail at scale[J].Communications of the ACM,2013,56(2):74-80.
[16] Li j,Sharma NK,Ports DR,et al.Tales of the tail:Hardware,OS,and application-level sources of tail latency[A].Proceedings of the ACM Symposium on Cloud Computing(SoCC 2014)[C].USA:2014.1-14.
[17] Gulisano V,Nikolakopoulos Y,Papatriantafilou M,et al.Scalejoin:A deterministic,disjoint-parallel and skew-resilient stream join[A].2015 IEEE International Conference on Big Data (Big Data 2015)[C].USA:2015.144-153.
[18] Roy P,Teubner J,Gemulla R.Low-latency handshake join[J].Proceedings of the Vldb Endowment,2014,7(9):709-720.
[19] Du G,Gupta I.New techniques to curtail the tail latency in stream processing systems[A].Proceedings of the 4th Workshop on Distributed Cloud Computing(DCC 2016)[C].Chicago,Ulinois:ACM,2016.7.
[20] Dawei Sun,Shang Gao,Xunyun Liu,et al.State and runtime-aware scheduling in elastic stream computing systems[J].Future Generation Computer Systems,2019,97:194-209.
[21] Guang Sun,Yingjie Song,Ziqin Gong,et al.Survey on streaming data computing system[A].ACMTURC Conference on Artificial Intelligence and Security(ACM TUR-C)[C].China:ACM,2019.159:1-159:8.
[22] Georgios Chantzialexiou,André Luckow,Shantenu Jha.Pilot-Streaming:A stream processing framework for high-performance computing[A].2018 IEEE 14th International Conference on e-Science (e-Science)[C].Netherlands:IEEE,2018.177-188.
[23] 鲁亮,于炯,卞琛,等.大数据流式计算框架Storm的任务迁移策略[J].计算机研究与发展,2018,55(1):71-92.
[24] Xu J,Chen Z,Tang J,et al.T-Storm:Traffic-aware online scheduling in storm[A].2014 IEEE 34th International Conference on Distributed Computing Systems(ICDCS 2014)[C].Spain:IEEE,2014.535-544.
[25] Peng B,Hosseini M,Hong Z,et al.R-Storm:Resource-aware scheduling in storm[A].Proceedings of the 16th Annual MIDDLEWARE Conference(MIDDLEWARE 2015)[C].USA:ACM,2015.149-161.
[26] Li T,Tang J,Xu J.Performance modeling and predictive scheduling for distributed stream data processing[J].IEEE Transactions on Big Data,2016,2(4):353-364.
[27] Kulkarni S,Bhagat N,Fu M,et al.Twitter heron:Stream processing at scale[A].Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data(SIGMOD 2015)[C].Australia:ACM,2015.239-250.
[28] Goetz P T,O'Neill B.Storm Blueprints:Patterns for Distributed Realtime Computation[M].U.K:Packt Publishing Ltd,2014.
[29] Carroll L.Alice's Adventures in Wonderland[M].Canada:Broadview Press,2011.

一种基于动态拓扑的流计算性能优化方法及其在Storm中的实现

A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	廖彬, 黄静莱, 王鑫, 孙瑞娜, 葛晓燕, 国冰磊. SCEA:一种适应高维海量数据的并行聚类集成算法[J]. 电子学报, 2021, 49(6): 1077-1087.
[2]	侯秀聪, 赖晓平, 曹九稳. 正则化超限学习机的最大分划广义交替方向乘子法[J]. 电子学报, 2021, 49(4): 625-630.
[3]	张聪炫, 周仲凯, 陈震, 葛利跃, 黎明, 江少锋, 陈昊. 深度学习光流计算技术研究进展[J]. 电子学报, 2020, 48(9): 1841-1849.
[4]	杨洋. 改进帕累托算法求解超大规模多选择背包问题[J]. 电子学报, 2020, 48(6): 1205-1212.
[5]	徐超, 陈勇, 葛红美, 何炎祥. 基于大数据的审计技术研究[J]. 电子学报, 2020, 48(5): 1003-1017.
[6]	江小玉, 李贵洋, 周悦, 胡金平, 李慧. 基于负载均衡的纠删码修复流水线[J]. 电子学报, 2020, 48(5): 930-936.
[7]	张鹏, 叶剑, 张鹏. 基于概念漂移检测的大数据交易过程模型优化方法[J]. 电子学报, 2019, 47(7): 1465-1474.
[8]	葛利跃, 张聪炫, 陈震, 黎明, 陈昊. 相互结构引导滤波TV-L1变分光流估计[J]. 电子学报, 2019, 47(3): 707-713.
[9]	魏波, 吴瑞峰, 张文生, 吕敬钦, 王莹莹, 夏学文. 具有双重稀疏机制的在线学习算法[J]. 电子学报, 2019, 47(10): 2202-2210.
[10]	金宏桥, 董一鸿, 陈华辉, 钱江波. 分布式环境下基于马尔科夫链的图流三角近似计算[J]. 电子学报, 2018, 46(9): 2139-2148.
[11]	郭艺, 叶剑, 张鹏. 基于偏差约减的大数据交易模型分析与修复方法[J]. 电子学报, 2018, 46(7): 1754-1761.
[12]	尹鑫, 田有亮, 王海龙. 面向大数据定价的委托拍卖方案[J]. 电子学报, 2018, 46(5): 1113-1120.
[13]	丁琳琳, 崔子强, 尹显坤, 王俊陆, 宋宝燕. 基于MapReduce的增广动态Skyline查询处理方法[J]. 电子学报, 2018, 46(5): 1062-1070.
[14]	朱皖宁, 刘志昊. 基于量子计算的用户识别算法[J]. 电子学报, 2018, 46(1): 24-30.
[15]	王玮, 陈晋, 余林峰, 薛群虎, 杨春利. 两类阻变机理及性能改善方法的研究[J]. 电子学报, 2017, 45(4): 989-999.