LU Jia-wei, WU Han, CHEN Hong, et al. A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm[J]. Acta Electronica Sinica, 2020, 48(5): 878-890.
DOI:
LU Jia-wei, WU Han, CHEN Hong, et al. A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm[J]. Acta Electronica Sinica, 2020, 48(5): 878-890. DOI: 10.3969/j.issn.0372-2112.2020.05.007.
A Performance Optimization Method Based on Dynamic Topology for Stream Computing and Its Implementation in Storm
Responsiveness and stability have always been two important problems in stream computing. However
as the scale of data being processed in real-time has increased
along with an increase in the data processing latency and topology instability of stream computing
many limitations of stream processing system have become apparent. Aiming at these problems
we present a performance optimization method based on dynamic topology for stream computing: (1) Dynamic step-by-step backpressure: the task in the topology can dynamically adjust the rate of upstream data transmission according to the current load. (2) Stateless topology data replay: topology can achieve data fault tolerance autonomously without maintaining the calculation of data state. (3) Adaptive topology replacement: no need for topology to suspend
the system can adjust the task concurrency spontaneously. (4) Delayed persistent queue: it delays the IO reading and writing in the disk out of the data processing
which mitigates the impact of IO high-frequency blocking in stream computing system. In this paper
the four methods are implemented in Apache Storm. The experimental results show that the optimized system not only enhances the dynamic matching capability of big data
but also achieves 17% higher throughput and 20% better data processing speed in the best case.