基于分层深度强化学习的RIS辅助车载边缘计算中信息年龄与能效优化
Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning
- 电子学报 2025年 页码：1-16
- 作者机构：
西北师范大学计算机科学与工程学院，甘肃兰州 730071
- 作者简介：
[ "兰军 男，2001年4月出生于甘肃省平凉市.现为西北师范大学计算机科学与工程学院硕士研究生.主要研究方向为智能反射面、信息年龄、车联网与资源分配.E-mail: 202421162081@nwnu.edu.cn" ]
[ "贾向东 男，1971年8月出生于甘肃省定西市.教授，博士.发表学术论文200余篇.主要研究方向为移动与无线通信关键理论与技术.E-mail: jiaxd@nwnu.edu.cn" ]
[ "寇志龙 男，2000年5月出生于山西省太原市.现为西北师范大学计算机科学与工程学院硕士研究生.主要研究方向为通感一体化网络无线通信、相对论及其应用.E-mail: 202421162193@nwnu.edu.cn" ]
[ "包红丽 女，2004年2月出生于甘肃省陇南市.现为西北师范大学计算机科学与工程学院研究生.主要研究方向为通感一体化、无人机通信.E-mail: 202421162044@nwnu.edu.cn" ]
[ "梁文艳 女，2001年10月出生于山西省太原市.现为西北师范大学计算机科学与工程学院硕士研究生.主要研究方向为智能反射面辅助无线通信.E-mail: 202421162027@nwnu.edu.cn" ]
[ "武婧婧 女，2001年12月出生于甘肃省武威市.2024年毕业于西北师范大学计算机科学与工程学院.现为西北师范大学计算机科学与工程学院硕士研究生.主要研究方向为通感一体化、智能反射面辅助无线通信等.E-mail: 202421162038@nwnu.edu.cn" ]
- 基金信息：国家自然科学基金(62261048;61861039);甘肃省高校产业支持计划(2025CYZC-014)
DOI：10.12263/DZXB.20250415
中图分类号： TN929.5;收稿：2025-05-25，
修回：2025-10-09，
网络出版：2025-11-12，
- 稿件说明：
移动端阅览
兰军, 贾向东, 寇志龙, 等. 基于分层深度强化学习的RIS辅助车载边缘计算中信息年龄与能效优化[J/OL]. 电子学报, 2025,1-16.
LAN Jun, JIA Xiang-dong, KOU Zhi-long, et al. Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning[J/OL]. ACTA ELECTRONICA SINICA, 2025, 1-16.
兰军, 贾向东, 寇志龙, 等. 基于分层深度强化学习的RIS辅助车载边缘计算中信息年龄与能效优化[J/OL]. 电子学报, 2025,1-16. DOI： 10.12263/DZXB.20250415.
LAN Jun, JIA Xiang-dong, KOU Zhi-long, et al. Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning[J/OL]. ACTA ELECTRONICA SINICA, 2025, 1-16. DOI： 10.12263/DZXB.20250415.
摘要
随着第五代（5G）和第六代（6G）移动通信技术的发展以及智能交通系统（Intelligent Transportation Systems， ITS）的成熟，车联网（Internet of Vehicles，IoV）逐渐成为智慧交通的重要支撑.车载边缘计算（Vehicular Edge Computing， VEC）通过在基站（Base Station， BS）或路侧单元（Roadside Unit，RSU）部署边缘服务器，为车载终端提供低时延计算服务.然而，车辆高速移动导致的信道衰落、能量受限及任务动态变化，使系统难以兼顾信息时效性与能量效率.智能反射面（Reconfigurable Intelligent Surface，RIS）能够通过相位可控反射重构传播环境，为VEC系统提供提升链路可靠性和能效的新途径.本文针对RIS辅助VEC系统中信息年龄（Age of Information，AoI）与能量消耗的协同优化问题，提出一种基于分层深度强化学习（Hierarchical Deep Reinforcement Learning，HDRL）的多目标优化框架.首先，本文构建了一个考虑车辆运动特性、三维几何信道和任务动态的系统模型，并建立最小化AoI与能量消耗加权和的非凸优化问题.其次，本文设计了具有“集中控制—分布协同”特性的分层混合强化学习架构：上层采用双延迟确定性策略梯度算法（Twin Delayed Deep Deterministic Policy Gradient，TD3）实现RIS相位连续优化，下层采用联邦多智能体深度确定性策略梯度算法（Federated Multi-Agent Deep Deterministic Policy Gradient，FMADDPG）实现功率与计算频率的分布式资源分配.为增强两层间的协同学习，本文提出联合预训练与轨迹嵌入机制：上层TD3控制器预生成RIS相位轨迹供下层FMADDPG策略初始化使用，从而实现跨层感知与加速收敛.此外，本文从理论上证明了FMADDPG算法在有界状态空间与Lipschitz连续奖励条件下的稳定收敛性.仿真结果表明，所提HDRL框架在信息新鲜度与能耗权衡方面显著优于软演员评论家算法（Soft Actor-Critic，SAC）、Q值混合网络（Q-value MIXing，QMIX）和块坐标下降（Block Coordinate Descent，BCD）等基准方法.与SAC算法相比，平均信息年龄降低约15%，系统能量效率提升约29%，在信道估计误差与遮挡概率较高的环境下仍保持稳定性能.本文的主要创新包括：（1）构建了RIS辅助VEC系统中AoI与能耗的多目标优化模型；（2）提出了结合TD3与FMADDPG的分层强化学习框架，实现集中控制与分布式协同；（3）设计了联合预训练与轨迹嵌入机制，有效提升了算法的收敛速度与策略感知能力.该研究为RIS辅助车联网的低时延与高能效优化提供了新的智能决策范式，对未来智能交通系统的边缘智能化具有重要参考价值.
Abstract
With the advancement of fifth-generation (5G) and sixth-generation (6G) mobile communication technologies and the continuous development of intelligent transportation systems (ITS)
the internet of vehicles (IoV) has gradually become a key foundation for smart transportation. Vehicular edge computing (VEC) provides low-latency computing services for vehicular terminals by deploying edge servers at base station (BS) or roadside unit (RSU). However
the high mobility of vehicles results in severe channel fading
limited energy resources
and dynamic task variations
which make it challenging to jointly guarantee information freshness and energy efficiency. Reconfigurable intelligent surface (RIS) technology
capable of reconfiguring the wireless propagation environment through controllable phase reflections
offers a promising solution to improve link reliability and energy efficiency in VEC systems.This paper proposes a hierarchical deep reinforcement learning (HDRL)-based multi-objective optimization framework to jointly optimize the Age of Information (AoI) and energy consumption in RIS-assisted VEC systems. Firstly
a system model is established that considers vehicular mobility
three-dimensional geometric channels
and dynamic task arrivals
and a non-convex optimization problem is formulated to minimize the weighted sum of AoI and energy consumption. Secondly
a hierarchical hybrid reinforcement learning architecture with “centralized control and distributed coordination” is designed. In the upper layer
the twin delayed deep deterministic policy gradient (TD3) algorithm is employed to continuously optimize RIS phase configurations
while the lower layer adopts the federated multi-agent deep deterministic policy gradient (FMADDPG) algorithm to realize distributed power allocation and computation frequency control.To enhance cross-layer learning coordination
a joint pretraining and trajectory-embedding mechanism is proposed
where the upper-layer TD3 controller generates representative RIS phase trajectories for initializing the policies of lower-layer FMADDPG agents. This mechanism effectively improves cross-layer awareness and accelerates convergence. In addition
theoretical analysis proves the stability and convergence of the FMADDPG algorithm under bounded state spaces and Lipschitz-continuous reward conditions.Simulation results demonstrate that the proposed HDRL framework significantly outperforms benchmark methods such as the soft actor-critic (SAC)
q-value mixing （QMIX） and block coordinate descent (BCD) algorithms in terms of balancing information freshness and energy efficiency. Compared with the SAC algorithm
the proposed approach reduces the average AoI by approximately 15% and improves energy efficiency by about 29%
while maintaining stable convergence under high channel estimation errors and blockage probabilities.The main innovations of this paper are as follows: (1) a multi-objective optimization model is developed for joint AoI and energy efficiency optimization in RIS-assisted VEC systems; (2) a hierarchical reinforcement learning framework combining TD3 and FMADDPG is proposed to achieve centralized RIS control and distributed resource coordination; (3) a joint pretraining and trajectory-embedding mechanism is designed to improve convergence speed and policy adaptability. This study provides a novel intelligent decision-making paradigm for low-latency and energy-efficient vehicular edge computing and offers valuable insights into the edge intelligence development of future intelligent transportation systems.
关键词
Keywords
references
许小龙 , 杨威 , 杨辰翊 , 等 . 车联网边缘计算环境下基于流量预测的高效任务卸载策略研究 [J ] . 电子学报 , 2025 , 53 ( 2 ): 329 - 343 .
XU X L , YANG W , YANG C Y , et al . Efficient task offloading based on traffic prediction in IoV-enabled edge computing [J ] . Acta Electronica Sinica , 2025 , 53 ( 2 ): 329 - 343 . (in Chinese)
王为念 , 苏健 , 陈勇 , 等 . 基于多智能体深度强化学习的车联网频谱共享 [J ] . 电子学报 , 2024 , 52 ( 5 ): 1690 - 1699 .
WANG W N , SU J , CHEN Y , et al . Multi-agent reinforcement learning enabled spectrum sharing for vehicular networks [J ] . Acta Electronica Sinica , 2024 , 52 ( 5 ): 1690 - 1699 . (in Chinese)
李国权 , 胡航 , 王玥涛 , 等 . STAR-RIS辅助的CR-SWIPT系统安全波束成形算法 [J ] . 电子学报 , 2024 , 52 ( 12 ): 4002 - 4008 .
LI G Q , HU H , WANG Y T , et al . Secure beamforming algorithm for STAR-RIS assisted cognitive radio systems with SWIPT [J ] . Acta Electronica Sinica , 2024 , 52 ( 12 ): 4002 - 4008 . (in Chinese)
ZHANG C , ZHANG W J , WU Q , et al . Distributed deep reinforcement learning-based gradient quantization for federated learning enabled vehicle edge computing [J ] . IEEE Internet of Things Journal , 2025 , 12 ( 5 ): 4899 - 4913 .
TANG F X , KAWAMOTO Y , KATO N , et al . Future intelligent and secure vehicular network toward 6G: Machine-learning approaches [J ] . Proceedings of the IEEE , 2020 , 108 ( 2 ): 292 - 307 .
ZHU H B , WU Q , WU X J , et al . Decentralized power allocation for MIMO-NOMA vehicular edge computing based on deep reinforcement learning [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 14 ): 12770 - 12782 .
HE J L , YU K Q , SHI Y M , et al . Reconfigurable intelligent surface assisted massive MIMO with antenna selection [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 7 ): 4769 - 4783 .
DI RENZO M , ZAPPONE A , DEBBAH M , et al . Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and the road ahead [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 11 ): 2450 - 2525 .
AHN J , MEHMOOD MUGHAL D , KIM S H , et al . Computation rate maximization in active RIS-assisted hybrid FDMA-NOMA MEC systems: A deep reinforcement learning approach [J ] . IEEE Wireless Communications Letters , 2025 , 14 ( 5 ): 1346 - 1350 .
LIU Y W , LIU X , MU X D , et al . Reconfigurable intelligent surfaces: Principles and opportunities [J ] . IEEE Communications Surveys&Tutorials , 2021 , 23 ( 3 ): 1546 - 1577 .
JI Z L , QIN Z J , PARINI C G . Reconfigurable intelligent surface aided cellular networks with device-to-device users [J ] . IEEE Transactions on Communications , 2022 , 70 ( 3 ): 1808 - 1819 .
MEI H B , YANG K , LIU Q , et al . 3D-trajectory and phase-shift design for RIS-assisted UAV systems using deep reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 3 ): 3020 - 3029 .
XIE Y B , SHI L , LI Z H , et al . Efficient task offloading in double roadside RIS-assisted vehicular edge computing networks using deep reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 7 ): 11353 - 11365 .
NGUYEN K K , TRAN T X , POMPILI D , et al . Reconfigurable intelligent surface-assisted multi-UAV networks: Efficient resource allocation with deep reinforcement learning [J ] . IEEE Journal of Selected Topics in Signal Processing , 2022 , 16 ( 3 ): 358 - 368 .
ZHAO J J , YU L , CAI K Q , et al . RIS-aided ground-aerial NOMA communications: A distributionally robust DRL approach [J ] . IEEE Journal on Selected Areas in Communications , 2022 , 40 ( 4 ): 1287 - 1301 .
HUANG C , MO R , YUEN C . Reconfigurable intelligent surface for wireless communication: Potential, challenges, and research directions [J ] . IEEE Communications Magazine , 2020 , 28 ( 2 ): 136 - 143 .
ZENG Y , ZHANG R , LIM T J . Wireless communications with unmanned aerial vehicles: Opportunities and challenges [J ] . IEEE Communications Magazine , 2016 , 54 ( 5 ): 36 - 42 .
HAZARIKA B , SINGH K , BISWAS S , et al . Multi-agent DRL-based task offloading in multiple RIS-aided IoV networks [J ] . IEEE Transactions on Vehicular Technology , 2024 , 73 ( 1 ): 1175 - 1190 .
WU Q Q , ZHANG R . Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network [J ] . IEEE Communications Magazine , 2020 , 58 ( 1 ): 106 - 112 .
ZENG L , LIU X , ZHANG W . Joint beamforming design for RIS-assisted multiuser MISO system with discrete phase shifts [J ] . IEEE Wireless Communications Letters , 2021 , 10 ( 5 ): 1052 - 1056 .
WANG X , CHEN M , TANG J . Energy-efficient resource allocation in RIS-assisted VEC networks using DRL [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 3 ): 1800 - 1812 .
LIU-YANG F , OUYANG W , CHEN L . Intelligent resource management for vehicular networks using multi-agent deep reinforcement learning [J ] . IEEE Transactions on Intelligent Transportation Systems , 2021 , 22 ( 7 ): 4123 - 4135 .
ZHENG-LI S , ZHOU P , LIU K . RIS-assisted secure communication in V2X: A game-theoretic approach [J ] . IEEE Transactions on Vehicular Technology , 2023 , 72 ( 2 ): 1230 - 1244 .
XU Z , ZHANG X , LI Y , et al . Federated deep reinforcement learning for computation offloading in VEC with non-IID data [J ] . IEEE Transactions on Vehicular Technology , 2023 , 72 ( 9 ): 11234 - 11248 .
ZHANG K , MA Y , WANG H , et al . Digital twin empowered industrial Internet of Things: A survey [J ] . IEEE Internet of Things Journal , 2021 , 8 ( 8 ): 13789 - 13804 .
PAN Y , WANG H , SHEN X , et al . Federated learning for edge intelligence: A survey [J ] . IEEE Communications Magazine , 2021 , 59 ( 1 ): 46 - 51 .
FENG L , LI W J , LIN Y X , et al . Joint computation offloading and URLLC resource allocation for collaborative MEC assisted cellular-V2X networks [J ] . IEEE Access , 2020 , 8 : 24914 - 24926 .
QI K W , WU Q , FAN P Y , et al . Deep-reinforcement-learning-based AoI-aware resource allocation for RIS-aided IoV networks [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 1 ): 1365 - 1378 .
ZHANG R , WU Q . Optimization for intelligent reflecting surface assisted wireless communication: A survey [J ] . IEEE Communications Magazine , 2020 , 58 ( 1 ): 26 - 32 .
LIANG F , LIU C , DU J , et al . Intelligent reflecting surface meets machine learning: A survey [J ] . IEEE Transactions on Wireless Communications , 2022 , 29 ( 1 ): 114 - 121 .
SUN Q M , NIU J P , ZHOU X W , et al . AoI and data rate optimization in aerial IRS-assisted IoT networks [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 4 ): 6481 - 6493 .
LU B S , FANG J L , HONG X M , et al . Task offloading in dynamic energy splitting STAR-RIS assisted NOMA-MEC systems with decomposition based multi-agent DRL [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 8 ): 13091 - 13103 .
0
浏览量
5
下载量
0
CSCD
- 网络PDF下载
- Meta-XML下载
- BibTeX下载
- Endnote下载
- RefMan 下载
- NoteExpress下载
- NoteFirst下载
关联资源
相关文章
相关作者
相关机构
- 技术支持由北京北大方正电子有限公司提供 京ICP备09064830号-19
京公网安备11010802024621 - 本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
- Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies