基于强化学习的离散事件系统最优定向监控

胡瑜洪; 王德光; 杨明; 王玺

doi:10.12263/DZXB.20221267

您当前的位置：

首页 >

文章列表页 >

基于强化学习的离散事件系统最优定向监控

学术论文 | 更新时间：2025-12-08

- 基于强化学习的离散事件系统最优定向监控
- Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning
- 电子学报 2024年52卷第9期页码：3172-3184
- 作者机构：
  
  1.贵州大学电气工程学院，贵州贵阳 550025
  2.西安电子科技大学机电工程学院，陕西西安 710071
- 作者简介：
  
  [ "胡瑜洪男，1999年4月出生于广东省揭阳市.现为贵州大学电子信息硕士研究生.主要研究方向为监督控制理论、机器人路径规划、调度等. E-mail: Yhong@163.com" ]
  [ "王德光男，1991年6月出生于山西省侯马市.2019年毕业于西安电子科技大学机电工程学院.现为贵州大学电气工程学院讲师（特岗教授）.从事监督控制理论、离散事件系统故障诊断、复杂系统建模与分析等方面的研究工作.E-mail: dgwang@gzu.edu.cn" ]
  [ "杨明男，1990年12月出生于贵州省铜仁市.2020年于北京化工大学获工学博士学位.现为贵州大学电气工程学院讲师（特岗教授）.主要研究方向为振动测试与计量、机器视觉检测、激光测量等. E-mail: myang23@gzu.edu.cn" ]
  [ "王玺男，1986年1月出生于河北省邯郸市.现为西安电子科技大学机电工程学院讲师.主要研究方向为离散事件系统监督控制理论、实时系统调度及重构的理论与应用研究.E-mail: wangxi@xidian.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(52265066;62203132);贵州省省级科技计划资助项目（No.黔科合基础-ZK［2022］一般103）;贵州省教育厅青年科技人才成长项目(黔教合KY字［2022］138号);贵州大学科研基金资助项目(贵大特岗合字［2021］04号);贵州省教育厅创新群体(黔科合支撑［2021］012)
- DOI：10.12263/DZXB.20221267
  中图分类号： TP301
- 收稿：2022-11-03，
  
  修回：2023-03-01，
  
  纸质出版：2024-09-25
- 稿件说明：
移动端阅览
胡瑜洪, 王德光, 杨明, 等. 基于强化学习的离散事件系统最优定向监控[J]. 电子学报, 2024, 52(09): 3172-3184.

HU Yu-hong, WANG De-guang, YANG Ming, et al. Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning[J]. Acta Electronica Sinica, 2024, 52(09): 3172-3184.
胡瑜洪, 王德光, 杨明, 等. 基于强化学习的离散事件系统最优定向监控[J]. 电子学报, 2024, 52(09): 3172-3184. DOI：10.12263/DZXB.20221267

HU Yu-hong, WANG De-guang, YANG Ming, et al. Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning[J]. Acta Electronica Sinica, 2024, 52(09): 3172-3184. DOI：10.12263/DZXB.20221267

摘要

对于多个可控事件（控制指令）允许同时执行的情形，离散事件系统的监控器进行随机选择.然而，在实际应用中，如交通调度、机器人路径规划，可控事件的定向选择和数值优化是必须要考虑和解决的两个问题.对此，引入一种优化机制量化控制成本，将监督控制理论与强化学习结合，提出一种基于强化学习的离散事件系统最优定向监控器求解方法，使被控系统实现以下三个目标：（1）遵循安全性和活性控制规范；（2）每个状态下至多允许一个可控事件执行；（3）从初始状态到标记状态事件执行累计成本最小.首先，建立系统和控制规范的自动机模型，做同步积运算后可得到目标模型，通过定义的成本函数为目标模型中每个事件的执行赋予成本.其次，利用监督控制理论求解无阻塞且行为最大许可的监控器.最后，将监控器转化为马尔可夫决策过程，并利用Q学习算法求解出最优定向监控器.使用单向列车导轨控制案例和多轨道列车控制案例验证所提方法的有效性和正确性.仿真结果表明，所提出方法能够实现系统的无阻塞定向控制，并且使得定向监控器的数值成本最小.

Abstract

In the case that several controllable events (control commands) are allowed to execute simultaneously

the supervisor in the framework of discrete event systems (DESs) selects one randomly. However

in practical applications

such as traffic scheduling and robot path planning

the problems of directed control and numerical optimization should be considered. This paper introduces an optimization mechanism to quantify the control cost and combines supervisory control theory (SCT) with reinforcement learning. A systematic procedure is proposed to synthesize the optimal directed supervisor of a DES based on reinforcement learning

which makes the controlled system achieve the following three goals: (1) the control specifications relevant to security and liveness are not violated; (2) at most one controllable event can be executed at each state; (3) the cumulative cost of event execution from the initial state to a mark state is minimal. First

given the automaton models of the plant and specifications

the target automaton model is obtained by the synchronous operation of these two models; a cost function is defined and assigns the execution cost for each event in the target model. Second

the non-blocking and maximally permissive supervisor is synthesized by SCT. Finally

the supervisor is transformed into a Markov decision process and then the Q-learning algorithm is utilized to compute the optimal directed supervisor. Two applications are used to verify the effectiveness and correctness of the proposed method. The simulation results show that the proposed method can realize the directed control of the system

and the numerical cost of the directed supervisor is minimized.

关键词

Keywords

references

RAMADGE P J G , WONHAM W M . The control of discrete event systems [J ] . Proceedings of the IEEE , 1989 , 77 ( 1 ): 81 - 98 .

CAI K , WONHAM W M . Supervisory control of discrete-event systems [M ] // Encyclopedia of Systems and Control . Cham : Springer , 2021 : 2245 - 2253 .

辛宗洋 . 基于RW框架的离散事件系统监督控制理论计算和仿真平台 [J ] . 工业控制计算机 , 2009 , 22 ( 12 ): 39 - 41, 43 .

XIN Z Y . Computation and simulation platform for supervisory control of discrete event system based on RW structure [J ] . Industrial Control Computer , 2009 , 22 ( 12 ): 39 - 41, 43 . (in Chinese)

焦亭 , 刘振国 . 组件多工作模式下的离散事件系统监督控制方法研究 [J ] . 控制理论与应用 , 2020 , 37 ( 3 ): 534 - 539 .

JIAO T , LIU Z G . Study on the supervisory control of discrete-event systems incorporating components with several working modes [J ] . Control Theory & Applications , 2020 , 37 ( 3 ): 534 - 539 . (in Chinese)

史晶锌 , 舒少龙 , 林峰 , 等 . 基于监督控制理论的家庭住宅用电安全性控制研究 [J ] . 现代建筑电气 , 2014 , 5 ( 1 ): 9 - 15 .

SHI J X , SHU S L , LIN F , et al . Control for safety of home electric usage based on supervisory control theory [J ] . Modern Architecture Electric , 2014 , 5 ( 1 ): 9 - 15 . (in Chinese)

荣胜波 , 朱军 , 史勃 , 等 . 监督控制理论在多任务机床控制系统中的应用 [J ] . 机床与液压 , 2011 , 39 ( 22 ): 85 - 87 .

RONG S B , ZHU J , SHI B , et al . The application of supervisory control theory to control system of multiple-task machine tool [J ] . Machine Tool & Hydraulics , 2011 , 39 ( 22 ): 85 - 87 . (in Chinese)

TATSUMOTO Y , SHIRAISHI M , CAI K , et al . Application of online supervisory control of discrete-event systems to multi-robot warehouse automation [J ] . Control Engineering Practice , 2018 , 81 : 97 - 104 .

GONZALEZ A G C , ALVES M V S , VIANA G S , et al . Supervisory control-based navigation architecture: A new framework for autonomous robots in industry 4.0 environments [J ] . IEEE Transactions on Industrial Informatics , 2018 , 14 ( 4 ): 1732 - 1743 .

UMEMOTO H , YAMASAKI T . Optimal LLP supervisor for discrete event systems based on reinforcement learning [C ] // 2015 IEEE International Conference on Systems, Man, and Cybernetics . Piscataway : IEEE , 2015 : 545 - 550 .

KAYMAKCI O , ANIK V G , USTOGLU I . A local modular supervisory controller for a real railway station [C ] // 5th IET International Conference on System Safety 2010 . Manchester : IET , 2010 : 1 - 6 .

PASSINO K M , ANTSAKLIS P J . On the optimal control of discrete event systems [C ] // Proceedings of the 28th IEEE Conference on Decision and Control . Piscataway : IEEE , 1989 : 2713 - 2718 .

BARBEAU M , FRAPPIER M , KABANZA F . A supervisory control synthesis case study: The antenna control system [C ] // Allerton Conference on Communication, Control, and Computing . Nederlanden : Elsevier , 1997 : 533 - 542 .

HUANG J , KUMAR R . An optimal directed control framework for discrete event systems [J ] . IEEE Transactions on Systems, Man, and Cybernetics , Part A: Systems and Humans, 2007 , 37 ( 5 ): 780 - 791 .

HUANG J , KUMAR R . Optimal nonblocking directed control of discrete event systems [C ] // 2007 American Control Conference . Piscataway : IEEE , 2007 : 4285 - 4290 .

王飞跃 , 陈俊龙 . 智能控制方法与应用 [M ] . 北京 : 中国科学技术出版社 , 2020 : 65 - 80 .

WANG F Y , CHEN J L . Intelligent Control Method and Application [M ] . Beijing : China Science and Technology Press , 2020 : 65 - 80 . (in Chinese)

王玺 . 实时系统的基于优先级的实时重构和不基于优先级的条件剥夺调度算法 [D ] . 西安 : 西安电子科技大学 , 2016 .

WANG X . Priority-based Reconfiguration and Priority-free Conditionally-preemptive Scheduling of Real-time Systems [D ] . Xi'an : Xidian University , 2016 . (in Chinese)

张涛 , 张文涛 , 代凌 , 等 . 基于序贯博弈多智能体强化学习的综合模块化航空电子系统重构方法 [J ] . 电子学报 , 2022 , 50 ( 4 ): 954 - 966 .

ZHANG T , ZHANG W T , DAI L , et al . Integrated modular avionics system reconstruction method based on dequential game multi-agent teinforcement learning [J ] . Acta Electronica Sinica , 2022 , 50 ( 4 ): 954 - 966 . (in Chinese)

袁都佳 . 基于Q学习的钢琴指法自动生成方法研究 [D ] . 广州 : 华南理工大学 , 2020 .

YUAN D J . Research on Automatic Generation Method of Piano Fingering Based on Q-learning [D ] . Guangzhou : South China University of Technology , 2020 . (in Chinese)

MADANI O , THORUP M , ZWICK U . Discounted deterministic Markov decision processes and discounted all-pairs shortest paths [J ] . ACM Transactions on Algorithms , 2010 , 6 ( 2 ): 1 - 25 .

SHUAI B , LI Y F , ZHOU Q , et al . Supervisory control of the hybrid off-highway vehicle for fuel economy improvement using predictive double Q-learning with backup models [J ] . Journal of Central South University , 2022 , 29 ( 7 ): 2266 - 2278 .

张德干 , 葛辉 , 刘晓欢 , 等 . 一种基于Q-Learning策略的自适应移动物联网路由新算法 [J ] . 电子学报 , 2018 , 46 ( 10 ): 2325 - 2332 .

ZHANG D G , GE H , LIU X H , et al . A kind of new routing algorithm with adaptivity for mobile IOT based on Q-Learning [J ] . Acta Electronica Sinica , 2018 , 46 ( 10 ): 2325 - 2332 . (in Chinese)

俞凯 . 基于图割/图搜索的医学影像分割算法研究及应用 [D ] . 苏州 : 苏州大学 , 2020 .

YU K . Study of Graph Cut/Search Based Medical Image Segmentation Algorithms and Applications [D ] . Suzhou : Soochow University , 2020 . (in Chinese)

徐超群 , 易忠 , 孟立飞 , 等 . 基于遗传算法的卫星磁模型研究 [J ] . 电子学报 , 2020 , 48 ( 6 ): 1108 - 1112 .

XU C Q , YI Z , MENG L F , et al . Research on satellite magnetic model based on genetic algorithm [J ] . Acta Electronica Sinica , 2020 , 48 ( 6 ): 1108 - 1112 . (in Chinese)

廖伟志 , 夏小云 , 贾小军 . 基于蚁群算法的多路径覆盖测试数据生成 [J ] . 电子学报 , 2020 , 48 ( 7 ): 1330 - 1342 .

LIAO W Z , XIA X Y , JIA X J . Test data generation for multiple paths coverage based on ant colony algorithm [J ] . Acta Electronica Sinica , 2020 , 48 ( 7 ): 1330 - 1342 . (in Chinese)

许凯波 , 鲁海燕 , 黄洋 , 等 . 基于双层蚁群算法和动态环境的机器人路径规划方法 [J ] . 电子学报 , 2019 , 47 ( 10 ): 2166 - 2176 .

XU K B , LU H Y , HUANG Y , et al . Robot path planning based on double-layer ant colony optimization algorithm and dynamic environment [J ] . Acta Electronica Sinica , 2019 , 47 ( 10 ): 2166 - 2176 . (in Chinese)

ZIELINSKI K M C , HENDGES L V , FLORINDO J B , et al . Flexible control of discrete event systems using environment simulation and reinforcement learning [J ] . Applied Soft Computing , 2021 , 111 : 107714 .

SHI H R , LIU G J , ZHANG K W , et al . MARL Sim2real transfer: Merging physical reality with digital virtuality in metaverse [J ] . IEEE Transactions on Systems, Man, and Cybernetics: Systems , 2023 , 53 ( 4 ): 2107 - 2117 .

ZHOU Z Y , LIU G J . RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states [EB/OL ] . ( 2022-05-15 )[ 2022-08-02 ] . http://arxiv.org/abs/2205.07229 http://arxiv.org/abs/2205.07229 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

联合异常检测的卫星安全传输优化算法

基于图组合优化的高效社区搜索

知识数据协同的多对手智能空中博弈策略设计

基于强化学习的免调参即插即用单光子图像重建方法