基于多智能体深度强化学习的分布式协同干扰功率分配算法

doi:10.12263/DZXB.20210818

摘要/Abstract

摘要：

针对战场通信对抗协同干扰中的干扰功率分配难题，本文基于多智能体深度强化学习设计了一种分布式协同干扰功率分配算法.具体地，将通信干扰功率分配问题构建为完全协作的多智能体任务，采用集中式训练、分布式决策的方式缓解多智能体系统环境非平稳、决策维度高的问题，减少智能体之间的通信开销，并加入最大策略熵准则控制各智能体的探索效率，以最大化累积干扰奖励和最大化干扰策略熵为优化目标，加速各智能体间协同策略的学习.仿真结果表明，所提出的分布式算法能有效解决高维协同干扰功率分配难题，相比于已有的集中式分配算法具有学习速度更快、波动性更小等优点，且相同条件下干扰效率可高出集中式算法16.8%.

长摘要
针对战场通信对抗场景下多设备协同干扰中的干扰功率分配难题，本文基于多智能体深度强化学习设计了一种分布式协同干扰功率分配算法。具体地，将通信干扰功率分配问题构建为完全协作的多智能体任务，融合了多智能体系统中集中式学习方法和独立式学习方法各自的优点，采用集中式训练、分布式决策的算法结构缓解多智能体系统环境非平稳、决策维度高、训练收敛难的问题，减少智能体之间的通信开销，并加入最大策略熵准则控制各智能体的探索效率，以最大化累积干扰奖励和最大化干扰策略熵为优化目标，加速各智能体间协同策略的学习。在奖励函数中，综合考虑了整体干扰压制任务的实现以及干扰功率利用的最优化，可以在不同干扰压制系数条件下，自适应地调整合理的干扰功率分配方案。仿真结果表明，所提出的分布式算法能有效解决高维协同干扰功率分配难题，相比于已有的集中式分配算法具有学习速度更快，波动性更小等优点，且相同条件下干扰效率可高出集中式算法16.8%；消融实验表明最大策略熵可以进一步提升探索效率，更快地找到最优方案。

关键词: 通信对抗, 协同功率分配, 多智能体深度强化学习, 分布式策略, 最大策略熵

Abstract:

In order to solve the problem of jamming power allocation in battlefield cooperative communication countermeasures, this paper designs a distributed cooperative jamming power allocation method based on multi-agent deep reinforcement learning. Specifically, modeling the communication jamming power allocation as a fully cooperative multi-agent task, then the framework of centralized training and distributed decision-making is adopted to alleviate the characteristic of non-stationary environment and high dimensions in multi-agent system, reducing the communication overhead between agents as well, and introducing the maximum policy entropy criterion to control the exploration efficiency of each agent. Regarding maximizing the cumulative jamming reward and maximizing the entropy of the jamming policy as the optimization goal, then accelerates the learning of cooperative strategies. Simulation results indicate the proposed distributed method can effectively solve the high-dimensional cooperative jamming power allocation problem. Compared with the existing centralized allocation method, it has faster learning speed and less volatility, and the jamming efficiency is 16.8% higher than that of the centralized method under the same conditions.

Extended Abstract
In order to solve the problem of jamming power allocation in multi device cooperative jamming in battlefield cooperative communication countermeasure scenario, this paper designs a distributed cooperative jamming power allocation method based on multi-agent deep reinforcement learning. Specifically, modeling the communication jamming power allocation as a fully cooperative multi-agent task, combining the advantages of centralized learning method and independent learning method in multi-agent system, then the framework of centralized training and distributed decision-making is adopted to alleviate the characteristic of non-stationary environment, high decision-making dimensions and difficult training convergence in multi-agent system, reducing the communication overhead between agents as well, and introducing the maximum policy entropy criterion to control the exploration efficiency of each agent. Regarding maximizing the cumulative jamming reward and maximizing the entropy of the jamming policy as the optimization goal, then accelerates the learning of cooperative strategies. In the reward function, the realization of the overall jamming suppression task and the optimization of jamming power utilization are comprehensively considered, and the reasonable jamming power allocation scheme can be adaptively adjusted under different jamming suppression coefficients. Simulation results indicate the proposed distributed method can effectively solve the high-dimensional cooperative jamming power allocation problem, compared with the existing centralized allocation method, it has faster learning speed and less volatility, and the jamming efficiency is 16.8% higher than that of the centralized method under the same conditions. The ablation experiment shows that the maximum strategy entropy can further improve the exploration efficiency and find the optimal scheme faster.

Key words: communication countermeasures, cooperative resource allocation, multi-agent deep reinforcement learning, distributed strategy, maximum policy entropy

中图分类号:

TN975

饶宁, 许华, 蒋磊, 宋佰霖, 史蕴豪. 基于多智能体深度强化学习的分布式协同干扰功率分配算法[J]. 电子学报, 2022, 50(6): 1319-1330.

RAO Ning, XU Hua, JIANG Lei, SONG Bai-lin, SHI Yun-hao. Allocation Algorithm of Distributed Cooperative Jamming Power Based on Multi-Agent Deep Reinforcement Learning[J]. Acta Electronica Sinica, 2022, 50(6): 1319-1330.

图/表 11

图1 对抗模型示意图

图2 集中式学习

图3 集中式训练、分布式决策结构

图4 MADJPA算法示意图

图5 分布式策略

表1 各通信链路信息

编号	类型	中心频率/MHz	跳频点数	重要性指数
链路1	定频	230.25	—	0.2
链路2	定频	275.50	—	0.3
链路3	跳频	—	160	0.6
链路4	定频	366.00	—	0.5
链路5	跳频	—	200	0.8

表2 实验及网络模型参数

参数	取值
干扰设备数量N	3
通信链路数量M	5
单设备最多可同时干扰目标数U	2
总干扰带宽B_j	2 MHz
定频链路带宽B_d	50 kHz
跳频链路频率间隔f_i	25 kHz·n(n=1,2,3,4)
干扰设备最大辐射功率P_max	77 dBm(约50 kW)
通信电台辐射功率P_c	55 dBm(约300 W)
噪声功率 $σ 2$	-85 dBm
通信链路增益G_c	8 dB
干扰链路增益G_j	3 dB
基站与电台最近距离R_c	110 km
干扰设备与电台最远距离R_j	300 km
压制系数K_i	2
柔性更新系数τ	0.01
训练回合数E	5 000
每回合交互次数T	500
经验回放池容量CRB	2¹⁷
批次样本大小B	256
折扣因子γ	0.98
熵系数初始值α	1

表2 实验及网络模型参数

参数	取值
干扰设备数量N	3
通信链路数量M	5
单设备最多可同时干扰目标数U	2
总干扰带宽B_j	2 MHz
定频链路带宽B_d	50 kHz
跳频链路频率间隔f_i	25 kHz·n(n=1,2,3,4)
干扰设备最大辐射功率P_max	77 dBm(约50 kW)
通信电台辐射功率P_c	55 dBm(约300 W)
噪声功率 $σ 2$	-85 dBm
通信链路增益G_c	8 dB
干扰链路增益G_j	3 dB
基站与电台最近距离R_c	110 km
干扰设备与电台最远距离R_j	300 km
压制系数K_i	2
柔性更新系数τ	0.01
训练回合数E	5 000
每回合交互次数T	500
经验回放池容量CRB	2¹⁷
批次样本大小B	256
折扣因子γ	0.98
熵系数初始值α	1

图6 整体完全压制效果

图7 3种算法对各链路的压制成功率曲线

图8 分配的总干扰功率对比

图9 不同压制系数的整体完全压制成功率

参考文献 31

1	王沙飞, 鲍雁飞, 李岩. 认知电子战体系结构与技术[J]. 中国科学: 信息科学, 2018, 48(12): 1603-1613, 1709.
	WANG S F, BAO Y F, LI Y. The architecture and technology of cognitive electronic warfare[J]. Scientia Sinica(Informationis), 2018, 48(12): 1603-1613, 1709. (in Chinese)
2	BAYRAM S, VANLI N D, DULEK B, et al. Optimum power allocation for average power constrained jammers in the presence of non-Gaussian noise[J]. IEEE Communications Letters, 2012, 16(8): 1153-1156.
3	XU C, SHENG M, WANG X J, et al. Distributed subchannel allocation for interference mitigation in OFDMA femtocells: A utility-based learning approach[J]. IEEE Transactions on Vehicular Technology, 2015, 64(6): 2463-2475.
4	GOMADAM K, CADAMBE V R, JAFAR S A. Approaching the capacity of wireless networks through distributed interference alignment[C]//2008 IEEE Global Telecommunications Conference. New Orleans: IEEE, 2008: 1-6.
5	AMURU S, TEKIN C, SCHAAR M VAN DER, et al. Jamming bandits—A novel learning method for optimal jamming[J]. IEEE Transactions on Wireless Communications, 2016, 15(4): 2792-2808.
6	颛孙少帅, 杨俊安, 刘辉, 等. 基于正强化学习和正交分解的干扰策略选择算法[J]. 系统工程与电子技术, 2018, 40(3): 518-525.
	ZHUANSUN S S, YANG J N, LIU H, et al. Jamming strategy learning based on positive reinforcement learning and orthogonal decomposition[J]. Systems Engineering and Electronics, 2018, 40(3): 518-525. (in Chinese)
7	AMURU S, BUEHRER R M. Optimal jamming using delayed learning[C]//2014 IEEE Military Communications Conference. Baltimore: IEEE, 2014: 1528-1533.
8	黄志清, 曲志伟, 张吉, 等. 基于深度强化学习的端到端无人驾驶决策[J]. 电子学报, 2020, 48(9): 1711-1719.
	HUANG Z Q, QU Z W, ZHANG J, et al. End-to-end autonomous driving decision based on deep reinforcement learning[J]. Acta Electronica Sinica, 2020, 48(9): 1711-1719. (in Chinese)
9	SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016, 529(7587): 484-489.
10	VINYALS O, BABUSCHKIN I, CZARNECKI W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. Nature, 2019, 575(7782): 350-354.
11	陈思光, 陈佳民, 赵传信. 基于深度强化学习的云边协同计算迁移研究[J]. 电子学报, 2021, 49(1): 157-166.
	CHEN S G, CHEN J M, ZHAO C X. Deep reinforcement learning based cloud-edge collaborative computation offloading mechanism[J]. Acta Electronica Sinica, 2021, 49(1): 157-166. (in Chinese)
12	LI S, YAN Y H, REN J, et al. A sample-efficient actor-critic algorithm for recommendation diversification[J]. Chinese Journal of Electronics, 2020, 29(1): 89-96.
13	杨启萌, 禹龙, 田生伟, 等. 基于深度强化学习的维吾尔语人称代词指代消解[J]. 电子学报, 2020, 48(6): 1077-1083.
	YANG Q M, YU L, TIAN S W, et al. Anaphora resolution of uyghur personal pronouns based on deep reinforcement learning[J]. Acta Electronica Sinica, 2020, 48(6): 1077-1083. (in Chinese)
14	LUONG N C, HOANG D T, GONG S M, et al. Applications of deep reinforcement learning in communications and networking: A survey[J]. IEEE Communications Surveys & Tutorials, 2019, 21(4): 3133-3174.
15	ZHAO D, QIN H, SONG B, et al. A graph convolutional network-based deep reinforcement learning approach for resource allocation in a cognitive radio network[J]. Sensors(Basel, Switzerland), 2020, 20(18): 5216-5239.
16	WANG S X, LIU H P, GOMES P H, et al. Deep reinforcement learning for dynamic multichannel access in wireless networks[J]. IEEE Transactions on Cognitive Communications and Networking, 2018, 4(2): 257-265.
17	XU Z Y, WANG Y Z, TANG J, et al. A deep reinforcement learning based framework for power-efficient resource allocation in cloud RANs[C]//2017 IEEE International Conference on Communications. Paris: IEEE, 2017: 1-6.
18	GUO D L, TANG L, ZHANG X G, et al. Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2020, 69(11): 13124-13138.
19	刘婷婷, 罗义南, 杨晨阳. 基于多智能体深度强化学习的分布式干扰协调[J]. 通信学报, 2020, 41(7): 38-48.
	LIU T T, LUO Y N, YANG C Y. Distributed interference coordination based on multi-agent deep reinforcement learning[J]. Journal on Communications, 2020, 41(7): 38-48. (in Chinese)
20	NASIR Y S, GUO D N. Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(10): 2239-2250.
21	ZHAO N, LIANG Y C, NIYATO D, et al. Deep reinforcement learning for user association and resource allocation in heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2019, 18(11): 5141-5152.
22	MENG F, CHEN P, WU L N, et al. Power allocation in multi-user cellular networks: Deep reinforcement learning approaches[J]. IEEE Transactions on Wireless Communications, 2020, 19(10): 6255-6267.
23	ZHANG K Q, YANG Z R, BAŞAR T. Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms[M/OL]. [2021]. .
24	NGUYEN T T, NGUYEN N D, NAHAVANDI S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications[J]. IEEE Transactions on Cybernetics, 2020, 50(9): 3826-3839.
25	冯小平, 李鹏, 杨绍全. 通信对抗原理[M]. 西安: 西安电子科技大学出版社, 2009.
26	FOERSTER J, FARQUHAR G, AFOURAS T, et al. Counterfactual multi-agent policy gradients[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence(AAAI). New Orleans:ACM, 2018: 2974-2983.
27	LOWE R, WU Y, TAMAR A, et al. Multiagent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems(NIPS). Long Beach: MIT Press, 2017: 6379-6390.
28	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of the 35th International Conference on Machine Learning(ICML). Stockholm: IMLS, 2018: 1861-1870.
29	HAARNOJA T, TANG H, ABBEEL P, et al. Reinforcement learning with deep energy-based policies[C]//Proceedings of the 34th International Conference on Machine Learning(ICML). Sydney: IMLS, 2017: 1352-1361.
30	MNIHL V, KAVUKCUOGLU K, SLIVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540):529-533.
31	FUJIMOTO S, HOOF H, MEGER M. Addressing function approximation error in actor-critic methods[C]//Proceedings of the 35th International Conference on Machine Learning(ICML). Stockholm: IMLS, 2018: 1587-1596.

[1]	肖硕, 黄珍珍, 张国鹏, 杨树松, 江海峰, 李天旭. 基于SAC的多智能体深度强化学习算法[J]. 电子学报, 2021, 49(9): 1675-1681.
[2]	谢岸宏, 朱立东, 翟继强, 李雄飞. 一种抗盲检测的直扩隐蔽信号设计方法[J]. 电子学报, 2018, 46(12): 2817-2823.
[3]	田新广;高立志;张尔扬. 改善直接数字合成周期信号频率分辨率的两种波形序列[J]. 电子学报, 2004, 32(3): 388-390.