Energy-Efficient Sleep-Mode Based on Deep Reinforcement Learning for Cell-Free mmWave Massive MIMO Systems

HE Yun; SHEN Min; WANG Rui; ZHANG Meng

doi:10.12263/DZXB.20220247

您当前的位置：

首页 >

文章列表页 >

Energy-Efficient Sleep-Mode Based on Deep Reinforcement Learning for Cell-Free mmWave Massive MIMO Systems

PAPERS | 更新时间：2025-12-08

- Energy-Efficient Sleep-Mode Based on Deep Reinforcement Learning for Cell-Free mmWave Massive MIMO Systems
- ACTA ELECTRONICA SINICA Vol. 51, Issue 10, Pages: 2831-2843(2023)
- 作者机构：
  
  1.重庆邮电大学通信与信息工程学院，重庆 400065
  2.重庆邮电大学通信核心芯片、协议及系统应用团队，重庆 400065
- 作者简介：
- 基金信息：
  
  National Science and Technology Major Project of China(2018ZX03001026-002)
- DOI：10.12263/DZXB.20220247
  CLC： TN929.5;
- Received：08 October 2021，
  
  Revised：2023-09-01，
  
  Published：25 October 2023
- 稿件说明：
移动端阅览
何云,申敏,王蕊等.无蜂窝毫米波大规模MIMO系统基于深度强化学习的节能睡眠策略[J].电子学报,2023,51(10):2831-2843.

HE Yun,SHEN Min,WANG Rui,et al.Energy-Efficient Sleep-Mode Based on Deep Reinforcement Learning for Cell-Free mmWave Massive MIMO Systems[J].ACTA ELECTRONICA SINICA,2023,51(10):2831-2843.
何云,申敏,王蕊等.无蜂窝毫米波大规模MIMO系统基于深度强化学习的节能睡眠策略[J].电子学报,2023,51(10):2831-2843. DOI： 10.12263/DZXB.20220247.

HE Yun,SHEN Min,WANG Rui,et al.Energy-Efficient Sleep-Mode Based on Deep Reinforcement Learning for Cell-Free mmWave Massive MIMO Systems[J].ACTA ELECTRONICA SINICA,2023,51(10):2831-2843. DOI： 10.12263/DZXB.20220247.

摘要

为了提升无蜂窝毫米波大规模MIMO （Cell-Free millimeter-Wave massive MIMO，CF mmWave mMIMO）系统总能量效率，本文研究时变信道环境中接入点（Access Point，AP）睡眠节能机制.将AP开关切换（AP Switch ON-OFF，ASO）策略看作一个马尔可夫决策过程，使用深度强化学习（Deep Reinforcement Learning，DRL）工具解决AP开关问题.引入干扰感知技术和局部敏感哈希检索方法减少代理与复杂环境的交互以及样本偏差，构造了一个新的效用函数，在严格用户服务质量（Quality of Service，QoS）约束下更好地权衡总能效和可达速率性能.通过对效用函数离散化分级处理，将状态空间映射为更小的分级状态空间，以加快决斗深度Q网络（Dueling Deep Q-Network，Dueling DQN）的收敛速度.仿真结果证明了该方案的稳定性、收敛性和严格QoS约束下的总能效性能优势.

Abstract

To improve the global energy-efficiency (GEE) performance in cell-free millimeter-wave massive MIMO (CF mmWave mMIMO) systems

the access points (APs) sleep-mode techniques in dynamic time-varying channels are investigated. The AP switch ON-OFF (ASO) strategy is formulated as a Markov decision process. Thus

a deep reinforcement learning (DRL) model can be used to solve the AP activation problem. The interference-aware method and the locality-sensitive hashing method are introduced to reduce sample bias and interaction between agents and complex environments. A novel cost function is constructed to achieve a better balance between GEE and achievable rate under the strict quality of service (QoS) constraints. In order to accelerate the convergence of the dueling deep Q-Network (DQN)

the state space is mapped to the smaller hierarchical state space by discretizing the cost function. Simulation results have demonstrated the performance advantage of the convergence of deep reinforcement learning and GEE under the strict QoS constraint.

关键词

Keywords

references

KIM S , SHIM B . Energy-efficient millimeter-wave cell-free systems under limited feedback [J]. IEEE Transactions on Communications , 2021 , 69 ( 6 ): 4067 - 4082 .

VAN CHIEN T , BJöRNSON E , LARSSON E G . Joint power allocation and load balancing optimization for energy-efficient cell-free massive MIMO networks [J]. IEEE Transactions on Wireless Communications , 2020 , 19 ( 10 ): 6798 - 6812 .

FEMENIAS G , LASSOUED N , RIERA-PALOU F . Access point switch ON/OFF strategies for green cell-free massive MIMO networking [J]. IEEE Access , 2020 , 8 : 21788 - 21803 .

USAMA M , EROL-KANTARCI M . A survey on recent trends and open issues in energy efficiency of 5G [J]. Sensors , 2019 , 19 ( 14 ): 3126 .

ZHUANG B , GUO D , HONIG M L . Energy-efficient cell activation, user association, and spectrum allocation in heterogeneous networks [J]. IEEE Journal on Selected Areas in Communications , 2016 , 34 ( 4 ): 823 - 831 .

GARCíA-MORALES J , FEMENIAS G , RIERA-PALOU F . Energy-efficient access-point sleep-mode techniques for cell-free mmWave massive MIMO networks with non-uniform spatial traffic density [J]. IEEE Access , 2020 , 8 : 137587 - 137605 .

HE H , JIN S , WEN C-K , et al . Model-driven deep learning for physical layer communications [J]. IEEE Wireless Communications , 2019 , 26 ( 5 ): 77 - 83 .

PHAM Q V , MIRJALILI S , KUMAR N , et al . Whale optimization algorithm with applications to resource allocation in wireless networks [J]. IEEE Transactions on Vehicular Technology , 2020 , 69 ( 4 ): 4285 - 4297 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J]. Nature , 2015 , 518 ( 7540 ): 529 - 533 .

HASHMI U S , ZAIDI S A R , IMRAN A , et al . Enhancing downlink QoS and energy efficiency through a user-centric Stienen cell architecture for mmWave networks [J]. IEEE Transactions on Green Communications and Networking , 2020 , 4 ( 2 ): 387 - 403 .

SUN G L , ZHAN T , OWUSU B G , et al . Revised reinforcement learning based on anchor graph hashing for autonomous cell activation in cloud-RANs [J]. Future Generation Computer Systems , 2020 , 104 : 60 - 73 .

SUN G L , AYEPAH-MENSAH D , XU R , et al . Transfer learning for autonomous cell activation based on relational reinforcement learning with adaptive reward [J]. IEEE Systems Journal , 2021 , 16 ( 1 ): 1044 - 1055 .

VAN HUYNH N , HOANG D T , NGUYEN D N , et al . DeepFake: Deep dueling-based deception strategy to defeat reactive jammers [J]. IEEE Transactions on Wireless Communications , 2021 , 20 ( 10 ): 6898 - 6914 .

DATAR M , IMMORLICA N , INDYK P , et al . Locality-sensitive hashing scheme based on p-stable distributions [C]// Proceedings of the twentieth annual symposium on Computational geometry . New York : ACM , 2004 : 253 - 262 .

CHAFIK S , YACOUBI M A EL , DAOUDI I , et al . Unsupervised deep neuron-per-neuron hashing [J]. Applied Intelligence , 2019 , 49 ( 6 ): 2218 - 2232 .

ALONZO M , BUZZI S , ZAPPONE A , et al . Energy-efficient power control in cell-free and user-centric massive MIMO at millimeter wave [J]. IEEE Transactions on Green Communications and Networking , 2019 , 3 ( 3 ): 651 - 663 .

RIBEIRO L N , SCHWARZ S , RUPP M , et al . Energy efficiency of mmWave massive MIMO precoding with low-resolution DACs [J]. IEEE Journal of Selected Topics in Signal Processing , 2018 , 12 ( 2 ): 298 - 312 .

WANG Z , SCHAUL T , HESSEL M , et al . Dueling network architectures for deep reinforcement learning [C]// International Conference on Machine Learning . New York : ICML , 2016 : 1995 - 2003 .

SARWAR S S , SRINIVASAN G , HAN B , et al . Energy efficient neural computing: A study of cross-layer approximations [J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems , 2018 , 8 ( 4 ): 796 - 809 .

ZHANG T , ZHU K , WANG J . Energy-efficient mode selection and resource allocation for D2D-enabled heterogeneous networks: A deep reinforcement learning approach [J]. IEEE Transactions on Wireless Communications , 2020 , 20 ( 2 ): 1175 - 1187 .

ZHANG J , HUANG Y , WANG J , et al . Intelligent interactive beam training for millimeter wave communications [J]. IEEE Transactions on Wireless Communications , 2020 , 20 ( 3 ): 2034 - 2048 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Design and Testing Method of A Miniaturized LTCC Filter with High-Selectivity and Low-Loss for 6G (B5G) Millimeter-Wave Communication

A Causal Tree-of-Thought-Based Model for Battery State-of-Charge Prediction in Electric Vehicles

Efficient Task Offloading Based on Traffic Prediction in IoV-Enabled Edge Computing

Research on UAV Path Planning Algorithm for Fairness Data Collection and Energy Supplement

Related Author

HE Yun

SHEN Min

WANG Rui

ZHANG Meng

LIU Huan

LI Chen

CAI Jie-ding

LUO Xin-jiang

Related Institution

School of Communication and Information Engineering， Chongqing University of Posts and；Telecommunications

Zhejiang University

College of Electronics Information, Hangzhou Dianzi University

Jiaxing Glead Electronics Co., LTD.

School of Transportation and Electrical Engineering, Hunan University of Technology

⁰