Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning

LAN Jun; JIA Xiang-dong; KOU Zhi-long; BAO Hong-li; LIANG Wen-yan; WU Jing-jing

doi:10.12263/DZXB.20250415

您当前的位置：

首页 >

文章列表页 >

Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning

更新时间：2025-11-12

- Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning
- ACTA ELECTRONICA SINICA Pages: 1-16(2025)
- 作者机构：
  
  西北师范大学计算机科学与工程学院，甘肃兰州 730071
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62261048;61861039);Gansu Province University Industry Support Plan(2025CYZC-014)
- DOI：10.12263/DZXB.20250415
  CLC： TN929.5;
- Received：25 May 2025，
  
  Revised：2025-10-09，
  
  Published Online：12 November 2025，
- 稿件说明：
移动端阅览
LAN Jun, JIA Xiang-dong, KOU Zhi-long, et al. Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning[J/OL]. ACTA ELECTRONICA SINICA, 2025, 1-16.
DOI：

LAN Jun, JIA Xiang-dong, KOU Zhi-long, et al. Age of Information and Energy Efficiency Optimization in RIS-Assisted Vehicular Edge Computing Based on Deep Reinforcement Learning[J/OL]. ACTA ELECTRONICA SINICA, 2025, 1-16. DOI： 10.12263/DZXB.20250415.

摘要

随着第五代（5G）和第六代（6G）移动通信技术的发展以及智能交通系统（Intelligent Transportation Systems， ITS）的成熟，车联网（Internet of Vehicles，IoV）逐渐成为智慧交通的重要支撑.车载边缘计算（Vehicular Edge Computing， VEC）通过在基站（Base Station， BS）或路侧单元（Roadside Unit，RSU）部署边缘服务器，为车载终端提供低时延计算服务.然而，车辆高速移动导致的信道衰落、能量受限及任务动态变化，使系统难以兼顾信息时效性与能量效率.智能反射面（Reconfigurable Intelligent Surface，RIS）能够通过相位可控反射重构传播环境，为VEC系统提供提升链路可靠性和能效的新途径.本文针对RIS辅助VEC系统中信息年龄（Age of Information，AoI）与能量消耗的协同优化问题，提出一种基于分层深度强化学习（Hierarchical Deep Reinforcement Learning，HDRL）的多目标优化框架.首先，本文构建了一个考虑车辆运动特性、三维几何信道和任务动态的系统模型，并建立最小化AoI与能量消耗加权和的非凸优化问题.其次，本文设计了具有“集中控制—分布协同”特性的分层混合强化学习架构：上层采用双延迟确定性策略梯度算法（Twin Delayed Deep Deterministic Policy Gradient，TD3）实现RIS相位连续优化，下层采用联邦多智能体深度确定性策略梯度算法（Federated Multi-Agent Deep Deterministic Policy Gradient，FMADDPG）实现功率与计算频率的分布式资源分配.为增强两层间的协同学习，本文提出联合预训练与轨迹嵌入机制：上层TD3控制器预生成RIS相位轨迹供下层FMADDPG策略初始化使用，从而实现跨层感知与加速收敛.此外，本文从理论上证明了FMADDPG算法在有界状态空间与Lipschitz连续奖励条件下的稳定收敛性.仿真结果表明，所提HDRL框架在信息新鲜度与能耗权衡方面显著优于软演员评论家算法（Soft Actor-Critic，SAC）、Q值混合网络（Q-value MIXing，QMIX）和块坐标下降（Block Coordinate Descent，BCD）等基准方法.与SAC算法相比，平均信息年龄降低约15%，系统能量效率提升约29%，在信道估计误差与遮挡概率较高的环境下仍保持稳定性能.本文的主要创新包括：（1）构建了RIS辅助VEC系统中AoI与能耗的多目标优化模型；（2）提出了结合TD3与FMADDPG的分层强化学习框架，实现集中控制与分布式协同；（3）设计了联合预训练与轨迹嵌入机制，有效提升了算法的收敛速度与策略感知能力.该研究为RIS辅助车联网的低时延与高能效优化提供了新的智能决策范式，对未来智能交通系统的边缘智能化具有重要参考价值.

Abstract

With the advancement of fifth-generation (5G) and sixth-generation (6G) mobile communication technologies and the continuous development of intelligent transportation systems (ITS)

the internet of vehicles (IoV) has gradually become a key foundation for smart transportation. Vehicular edge computing (VEC) provides low-latency computing services for vehicular terminals by deploying edge servers at base station (BS) or roadside unit (RSU). However

the high mobility of vehicles results in severe channel fading

limited energy resources

and dynamic task variations

which make it challenging to jointly guarantee information freshness and energy efficiency. Reconfigurable intelligent surface (RIS) technology

capable of reconfiguring the wireless propagation environment through controllable phase reflections

offers a promising solution to improve link reliability and energy efficiency in VEC systems.This paper proposes a hierarchical deep reinforcement learning (HDRL)-based multi-objective optimization framework to jointly optimize the Age of Information (AoI) and energy consumption in RIS-assisted VEC systems. Firstly

a system model is established that considers vehicular mobility

three-dimensional geometric channels

and dynamic task arrivals

and a non-convex optimization problem is formulated to minimize the weighted sum of AoI and energy consumption. Secondly

a hierarchical hybrid reinforcement learning architecture with “centralized control and distributed coordination” is designed. In the upper layer

the twin delayed deep deterministic policy gradient (TD3) algorithm is employed to continuously optimize RIS phase configurations

while the lower layer adopts the federated multi-agent deep deterministic policy gradient (FMADDPG) algorithm to realize distributed power allocation and computation frequency control.To enhance cross-layer learning coordination

a joint pretraining and trajectory-embedding mechanism is proposed

where the upper-layer TD3 controller generates representative RIS phase trajectories for initializing the policies of lower-layer FMADDPG agents. This mechanism effectively improves cross-layer awareness and accelerates convergence. In addition

theoretical analysis proves the stability and convergence of the FMADDPG algorithm under bounded state spaces and Lipschitz-continuous reward conditions.Simulation results demonstrate that the proposed HDRL framework significantly outperforms benchmark methods such as the soft actor-critic (SAC)

q-value mixing （QMIX） and block coordinate descent (BCD) algorithms in terms of balancing information freshness and energy efficiency. Compared with the SAC algorithm

the proposed approach reduces the average AoI by approximately 15% and improves energy efficiency by about 29%

while maintaining stable convergence under high channel estimation errors and blockage probabilities.The main innovations of this paper are as follows: (1) a multi-objective optimization model is developed for joint AoI and energy efficiency optimization in RIS-assisted VEC systems; (2) a hierarchical reinforcement learning framework combining TD3 and FMADDPG is proposed to achieve centralized RIS control and distributed resource coordination; (3) a joint pretraining and trajectory-embedding mechanism is designed to improve convergence speed and policy adaptability. This study provides a novel intelligent decision-making paradigm for low-latency and energy-efficient vehicular edge computing and offers valuable insights into the edge intelligence development of future intelligent transportation systems.

关键词

Keywords

references

许小龙 , 杨威 , 杨辰翊 , 等 . 车联网边缘计算环境下基于流量预测的高效任务卸载策略研究 [J ] . 电子学报 , 2025 , 53 ( 2 ): 329 - 343 .

XU X L , YANG W , YANG C Y , et al . Efficient task offloading based on traffic prediction in IoV-enabled edge computing [J ] . Acta Electronica Sinica , 2025 , 53 ( 2 ): 329 - 343 . (in Chinese)

王为念 , 苏健 , 陈勇 , 等 . 基于多智能体深度强化学习的车联网频谱共享 [J ] . 电子学报 , 2024 , 52 ( 5 ): 1690 - 1699 .

WANG W N , SU J , CHEN Y , et al . Multi-agent reinforcement learning enabled spectrum sharing for vehicular networks [J ] . Acta Electronica Sinica , 2024 , 52 ( 5 ): 1690 - 1699 . (in Chinese)

李国权 , 胡航 , 王玥涛 , 等 . STAR-RIS辅助的CR-SWIPT系统安全波束成形算法 [J ] . 电子学报 , 2024 , 52 ( 12 ): 4002 - 4008 .

LI G Q , HU H , WANG Y T , et al . Secure beamforming algorithm for STAR-RIS assisted cognitive radio systems with SWIPT [J ] . Acta Electronica Sinica , 2024 , 52 ( 12 ): 4002 - 4008 . (in Chinese)

ZHANG C , ZHANG W J , WU Q , et al . Distributed deep reinforcement learning-based gradient quantization for federated learning enabled vehicle edge computing [J ] . IEEE Internet of Things Journal , 2025 , 12 ( 5 ): 4899 - 4913 .

TANG F X , KAWAMOTO Y , KATO N , et al . Future intelligent and secure vehicular network toward 6G: Machine-learning approaches [J ] . Proceedings of the IEEE , 2020 , 108 ( 2 ): 292 - 307 .

ZHU H B , WU Q , WU X J , et al . Decentralized power allocation for MIMO-NOMA vehicular edge computing based on deep reinforcement learning [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 14 ): 12770 - 12782 .

HE J L , YU K Q , SHI Y M , et al . Reconfigurable intelligent surface assisted massive MIMO with antenna selection [J ] . IEEE Transactions on Wireless Communications , 2022 , 21 ( 7 ): 4769 - 4783 .

DI RENZO M , ZAPPONE A , DEBBAH M , et al . Smart radio environments empowered by reconfigurable intelligent surfaces: How it works, state of research, and the road ahead [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 11 ): 2450 - 2525 .

AHN J , MEHMOOD MUGHAL D , KIM S H , et al . Computation rate maximization in active RIS-assisted hybrid FDMA-NOMA MEC systems: A deep reinforcement learning approach [J ] . IEEE Wireless Communications Letters , 2025 , 14 ( 5 ): 1346 - 1350 .

LIU Y W , LIU X , MU X D , et al . Reconfigurable intelligent surfaces: Principles and opportunities [J ] . IEEE Communications Surveys&Tutorials , 2021 , 23 ( 3 ): 1546 - 1577 .

JI Z L , QIN Z J , PARINI C G . Reconfigurable intelligent surface aided cellular networks with device-to-device users [J ] . IEEE Transactions on Communications , 2022 , 70 ( 3 ): 1808 - 1819 .

MEI H B , YANG K , LIU Q , et al . 3D-trajectory and phase-shift design for RIS-assisted UAV systems using deep reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2022 , 71 ( 3 ): 3020 - 3029 .

XIE Y B , SHI L , LI Z H , et al . Efficient task offloading in double roadside RIS-assisted vehicular edge computing networks using deep reinforcement learning [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 7 ): 11353 - 11365 .

NGUYEN K K , TRAN T X , POMPILI D , et al . Reconfigurable intelligent surface-assisted multi-UAV networks: Efficient resource allocation with deep reinforcement learning [J ] . IEEE Journal of Selected Topics in Signal Processing , 2022 , 16 ( 3 ): 358 - 368 .

ZHAO J J , YU L , CAI K Q , et al . RIS-aided ground-aerial NOMA communications: A distributionally robust DRL approach [J ] . IEEE Journal on Selected Areas in Communications , 2022 , 40 ( 4 ): 1287 - 1301 .

HUANG C , MO R , YUEN C . Reconfigurable intelligent surface for wireless communication: Potential, challenges, and research directions [J ] . IEEE Communications Magazine , 2020 , 28 ( 2 ): 136 - 143 .

ZENG Y , ZHANG R , LIM T J . Wireless communications with unmanned aerial vehicles: Opportunities and challenges [J ] . IEEE Communications Magazine , 2016 , 54 ( 5 ): 36 - 42 .

HAZARIKA B , SINGH K , BISWAS S , et al . Multi-agent DRL-based task offloading in multiple RIS-aided IoV networks [J ] . IEEE Transactions on Vehicular Technology , 2024 , 73 ( 1 ): 1175 - 1190 .

WU Q Q , ZHANG R . Towards smart and reconfigurable environment: Intelligent reflecting surface aided wireless network [J ] . IEEE Communications Magazine , 2020 , 58 ( 1 ): 106 - 112 .

ZENG L , LIU X , ZHANG W . Joint beamforming design for RIS-assisted multiuser MISO system with discrete phase shifts [J ] . IEEE Wireless Communications Letters , 2021 , 10 ( 5 ): 1052 - 1056 .

WANG X , CHEN M , TANG J . Energy-efficient resource allocation in RIS-assisted VEC networks using DRL [J ] . IEEE Internet of Things Journal , 2022 , 9 ( 3 ): 1800 - 1812 .

LIU-YANG F , OUYANG W , CHEN L . Intelligent resource management for vehicular networks using multi-agent deep reinforcement learning [J ] . IEEE Transactions on Intelligent Transportation Systems , 2021 , 22 ( 7 ): 4123 - 4135 .

ZHENG-LI S , ZHOU P , LIU K . RIS-assisted secure communication in V2X: A game-theoretic approach [J ] . IEEE Transactions on Vehicular Technology , 2023 , 72 ( 2 ): 1230 - 1244 .

XU Z , ZHANG X , LI Y , et al . Federated deep reinforcement learning for computation offloading in VEC with non-IID data [J ] . IEEE Transactions on Vehicular Technology , 2023 , 72 ( 9 ): 11234 - 11248 .

ZHANG K , MA Y , WANG H , et al . Digital twin empowered industrial Internet of Things: A survey [J ] . IEEE Internet of Things Journal , 2021 , 8 ( 8 ): 13789 - 13804 .

PAN Y , WANG H , SHEN X , et al . Federated learning for edge intelligence: A survey [J ] . IEEE Communications Magazine , 2021 , 59 ( 1 ): 46 - 51 .

FENG L , LI W J , LIN Y X , et al . Joint computation offloading and URLLC resource allocation for collaborative MEC assisted cellular-V2X networks [J ] . IEEE Access , 2020 , 8 : 24914 - 24926 .

QI K W , WU Q , FAN P Y , et al . Deep-reinforcement-learning-based AoI-aware resource allocation for RIS-aided IoV networks [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 1 ): 1365 - 1378 .

ZHANG R , WU Q . Optimization for intelligent reflecting surface assisted wireless communication: A survey [J ] . IEEE Communications Magazine , 2020 , 58 ( 1 ): 26 - 32 .

LIANG F , LIU C , DU J , et al . Intelligent reflecting surface meets machine learning: A survey [J ] . IEEE Transactions on Wireless Communications , 2022 , 29 ( 1 ): 114 - 121 .

SUN Q M , NIU J P , ZHOU X W , et al . AoI and data rate optimization in aerial IRS-assisted IoT networks [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 4 ): 6481 - 6493 .

LU B S , FANG J L , HONG X M , et al . Task offloading in dynamic energy splitting STAR-RIS assisted NOMA-MEC systems with decomposition based multi-agent DRL [J ] . IEEE Transactions on Vehicular Technology , 2025 , 74 ( 8 ): 13091 - 13103 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Edge Caching Scheme Based on Value of Information Age in the Internet of Vehicles

A Survey of Computation Offloading in Vehicular Edge Computing Networks

Efficient Task Offloading Based on Traffic Prediction in IoV-Enabled Edge Computing

Intelligent Channel Estimation of SC-FDMA Based on GNN for V2I Scenarios in Internet of Vehicles

Energy Efficiency Research on Cell-Free Massive MIMO-WPT Systems with Mixed-Resolution DACs

Related Author

LIN Zhi-jian

HOU Ying

CAO Xiao-xiao

WU Wen-hao

CHEN Xiao-pei

CHEN Ping-ping

LIU Lei

CHEN Chen

Related Institution

Department of Electronic Information Engineering， Fuzhou University

State Key Laboratory of Integrated Services Networks, Xidian University

School of Software, Nanjing University of Information Science and Technology

Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology

Jiangsu Province Engineering Research Center of Advanced Computing and Intelligent Services

⁰