Multipath Transmission Control Mechanism Based on Deep Reinforcement Learning and Sub-flow Coupling Perception

QIN Jiu-ren; XU Chang-qiao; YANG Shu-jie; GAO Kai; ZHANG Hong-ke

doi:10.12263/DZXB.20201414

您当前的位置：

首页 >

文章列表页 >

Multipath Transmission Control Mechanism Based on Deep Reinforcement Learning and Sub-flow Coupling Perception

PAPERS | 更新时间：2025-12-08

- Multipath Transmission Control Mechanism Based on Deep Reinforcement Learning and Sub-flow Coupling Perception
- ACTA ELECTRONICA SINICA Vol. 50, Issue 2, Pages: 346-357(2022)
- 作者机构：
  
  1.北京邮电大学网络与交换技术国家重点实验室，北京 100876
  2.北京交通大学电子信息工程学院，北京 100044
- 作者简介：
- 基金信息：
- DOI：10.12263/DZXB.20201414
  CLC： TP393;
- Received：09 December 2020，
  
  Revised：2021-03-04，
  
  Published：25 February 2022
- 稿件说明：
移动端阅览
秦久人,许长桥,杨树杰等.基于深度增强学习与子流耦合感知的多路传输控制机制[J].电子学报,2022,50(02):346-357.

QIN Jiu-ren,XU Chang-qiao,YANG Shu-jie,et al.Multipath Transmission Control Mechanism Based on Deep Reinforcement Learning and Sub-flow Coupling Perception[J].ACTA ELECTRONICA SINICA,2022,50(02):346-357.
秦久人,许长桥,杨树杰等.基于深度增强学习与子流耦合感知的多路传输控制机制[J].电子学报,2022,50(02):346-357. DOI： 10.12263/DZXB.20201414.

QIN Jiu-ren,XU Chang-qiao,YANG Shu-jie,et al.Multipath Transmission Control Mechanism Based on Deep Reinforcement Learning and Sub-flow Coupling Perception[J].ACTA ELECTRONICA SINICA,2022,50(02):346-357. DOI： 10.12263/DZXB.20201414.

摘要

为了解决多路传输中子流耦合感知缺乏与传输控制效率低下等的问题，针对未来异构、动态的网络环境，提出了一种基于耦合感知与深度Q网络的多路传输控制机制（WaveLet and Deep Q Network based multipath transmission control mechanism，WL-DQN）.利用小波去噪技术，消除子流单向传输时延中由非耦合路段及系统随机产生的噪声，并基于子流互相关系数对子流耦合特性进行提取；在此基础上，依据深度增强学习理论对多路传输控制进行建模，并提出多路DQN拥塞控制算法，实现了异构、动态网络环境下的智能多路拥塞控制.仿真结果表明，所提算法在传输吞吐量、传输时延、数据包重传避免等方面均优于标准及相似的代表性解决方案.

Abstract

To solve the problems of lacking coupled sub-flow perception and low transmission control efficiency for multipath transmission

a multipath transmission control mechanism based on wavelet and deep Q network(WL-DQN) is proposed for future heterogeneous and dynamic networks. Wavelet techniques are adopted to eliminate the noise caused by uncoupled queueing and system random delay in the one-way transmission delay

and the coupling characteristics are extracted based on the path correlation factors. On this basis

the deep reinforcement learning theories are applied to model the multipath transmission control

and the multipath DQN control algorithm is developed to realize the intelligent multipath congestion control under the heterogeneous and dynamic networks. Simulation results demonstrate that the proposed algorithms outperform the standard and similar representative solutions in terms of transmission throughput

transmission delay

packet retransmission avoidance

etc.

关键词

Keywords

references

LIU Y , YUAN X , XIONG Z , et al . Federated learning for 6G communications: Challenges, methods, and future directions [J]. China Communications , 2020 , 17 ( 9 ): 105 - 118 .

Ford A . TCP extensions for multipath operation with multiple addresses [EB/OL]. ( 2020-03 )[ 2020-08 ]. https://datatracker.ietf.org/doc/html/rfc8684.html https://datatracker.ietf.org/doc/html/rfc8684.html .

CAO Y , GUO H , LIU J , et al . Optimal satellite gateway placement in space-ground integrated networks [J]. IEEE Network , 2018 , 32 ( 5 ): 32 - 37 .

CHOO K R , GRITZALIS S , PARK J H . Cryptographic solutions for industrial internet-of-things: Research challenges and opportunities [J]. IEEE Transactions on Industrial Informatics , 2018 , 14 ( 8 ): 3567 - 3569 .

KHABBAZ M , ANTOUN J , ASSI C . Modeling and performance analysis of UAV-assisted vehicular networks [J]. IEEE Transactions on Vehicular Technology , 2019 , 68 ( 9 ): 8384 - 8396 .

FERLIN S , ALAY Ö , DREIBHOLZ T , et al . Revisiting congestion control for Multipath TCP with shared bottleneck Detection [C]// IEEE International Conference on Computer Communications(INFOCOM) . San Francisco, CA, USA : IEEE , 2016 : 1 - 9 .

Raiciu C . Coupled Congestion Control for Multipath Transport Protocols [EB/OL]. ( 2011-10 )[ 2020-08 ]. https://datatracker.ietf.org/doc/html/rfc6356.html https://datatracker.ietf.org/doc/html/rfc6356.html .

OH B H , LEE J . Feedback-based path failure detection and buffer blocking protection for MPTCP [J]. IEEE/ACM Transactions on Networking , 2016 , 24 ( 6 ): 3450 - 3461 .

ZHAO J , LIU J , WANG H , et al . Measurement, analysis, and enhancement of multipath TCP energy efficiency for datacenters [J]. IEEE/ACM Transactions on Networking , 2020 , 28 ( 1 ): 57 - 70 .

XU C , LI Z , LI J , et al . Cross-layer fairness-driven concurrent multipath video delivery over heterogeneous wireless networks [J]. IEEE Transactions on Circuits & Systems for Video Technology , 2015 , 25 ( 7 ): 1175 - 1189 .

XU C , WANG P , XIONG C , et al . Pipeline network coding-based multipath data transfer in heterogeneous wireless networks [J]. IEEE Transactions on Broadcasting , 2017 , 63 ( 2 ): 376 - 390 .

XU C , JIA S , WANG M , et al . Performance-aware mobile community-based VoD streaming over vehicular Ad Hoc networks [J]. IEEE Transactions on Vehicular Technology , 2015 , 64 ( 3 ): 1201 - 1217 .

DONOHO D L , JOHNSTONE I M . Ideal spatial adaptation by wavelet shrinkage [J]. Biometrika , 1994 , 81 ( 3 ): 425 - 455 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J]. Nature , 2015 , 518 ( 7540 ): 529 - 533 .

SADIO O , NGOM I , LISHOU C . Design and prototyping of a software defined vehicular networking [J]. IEEE Transactions on Vehicular Technology , 2020 , 69 ( 1 ): 842 - 850 .

GE X . Ultra-reliable low-latency communications in autonomous vehicular networks [J]. IEEE Transactions on Vehicular Technology , 2019 , 68 ( 5 ): 5005 - 5016 .

李红艳 , 张焘 , 张靖乾 , 等 . 基于时变图的天地一体化网络时间确定性路由算法与协议 [J]. 通信学报 , 2020 , 41 ( 10 ): 116 - 129 .

LI H Y , ZHANG T , ZHANG J Q , et al . Time deterministic routing algorithm and protocol based on time-varying graph over the space-ground integrated network [J]. Journal on Communications , 2020 , 41 ( 10 ): 116 - 129 . (in Chinese)

HUI H , ZHOU C , XU S , et al . A novel secure data transmission scheme in industrial internet of things [J]. China Communications , 2020 , 17 ( 1 ): 73 - 88 .

LI X , LI D , WAN J , et al . Adaptive transmission optimization in SDN-based industrial internet of things with edge computing [J]. IEEE Internet of Things Journal , 2018 , 5 ( 3 ): 1351 - 1360 .

孙鹏浩 , 兰巨龙 , 申涓 , 等 . 一种基于深度增强学习的智能路由技术 [J]. 电子学报 , 2020 , 48 ( 11 ): 92 - 99 .

SUN P H , LAN J L , SHEN J , et al . An intelligent routing technology based on deep reinforcement learning [J]. Acta Electronica Sinica , 2020 , 48 ( 11 ): 92 - 99 . (in Chinese)

POKHREL S R , CHOI J . Low-delay scheduling for internet of vehicles: Load-balanced multipath communication with FEC [J]. IEEE Transactions on Communications , 2019 , 67 ( 12 ): 8489 - 8501 .

SINGH P K , SHARMA S , NANDI S K , et al . Multipath TCP for V2I communication in SDN controlled small cell deployment of smart city [J]. Vehicular Communications , 2019 , 15 ( 1 ): 1 - 15 .

NING Z , FENG Y , COLLOTTA M , et al . Deep learning in edge of vehicles: Exploring trirelationship for data transmission [J]. IEEE Transactions on Industrial Informatics , 2019 , 15 ( 10 ): 5737 - 5746 .

LI W , ZHANG H , GAO S , et al . SmartCC: A reinforcement learning approach for multipath TCP congestion control in heterogeneous networks [J]. IEEE Journal on Selected Areas in Communications , 2019 , 37 ( 11 ): 2621 - 2633 .

XU Z , TANG J , YIN C , et al . Experience-driven congestion control: When multi-path TCP meets deep reinforcement learning [J]. IEEE Journal on Selected Areas in Communications , 2019 , 37 ( 6 ): 1325 - 1336 .

WINSTEIN K , BALAKRISHNAN H . TCP ex machina: Computer-generated congestion control [C]// Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM . New York, NY, USA : ACM , 2013 : 123 - 134 .

KASHIF N . Mptcp implementation in ns3 [EB/OL]. ( 2019-04-05 )[ 2021-12-30 ]. https://github.com/Kashif-Nadeem/ns-3-dev-git https://github.com/Kashif-Nadeem/ns-3-dev-git .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cross-Layer Congestion Control for Wireless Sensor Networks

An Active Queue Management Algorithm Based on Predictable PI Controller in Self-Similar Network

Related Author

Jiu-ren QIN

Chang-qiao XU

Shu-jie YANG

Kai GAO

Hong-ke ZHANG

ZHANG Yu-peng

LIU Kai

WANG Guang-xue

Related Institution

School of Electronic and Information Engineering， Beijing Jiaotong University

State Key Laboratory of Networking and Switching Technology， Beijing University of Posts and Telecommunication

China Unicom Beijing BranchBeijing 100038China

School of Electronics and Information EngineeringBeihang UniversityBeijing 100191China

China Unicom Beijing Branch

⁰