Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning

HU Yu-hong; WANG De-guang; YANG Ming; WANG Xi

doi:10.12263/DZXB.20221267

您当前的位置：

首页 >

文章列表页 >

Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning

PAPERS | 更新时间：2025-12-08

- Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning
- ACTA ELECTRONICA SINICA Vol. 52, Issue 9, Pages: 3172-3184(2024)
- 作者机构：
  
  1.贵州大学电气工程学院，贵州贵阳 550025
  2.西安电子科技大学机电工程学院，陕西西安 710071
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(52265066;62203132);Guizhou Provincial Science and Technology Projects (No.QianKeHeJiChu- ZK[2022]YiBan 103);Youth Science and Technology Talents Development Project of Guizhou Education Department (No.QianJiaoHe KY [2022]138)(黔教合KY字［2022］138号);Scientific Research Fund of GuiZhou University(贵大特岗合字［2021］04号);Guizhou Provincial Education Innovation Group Projects (No.QianKe- HeZhiCheng[2021]012.)(黔科合支撑［2021］012)
- DOI：10.12263/DZXB.20221267
  CLC： TP301
- Received：03 November 2022，
  
  Revised：2023-03-01，
  
  Published：25 September 2024
- 稿件说明：
移动端阅览
胡瑜洪, 王德光, 杨明, 等. 基于强化学习的离散事件系统最优定向监控[J]. 电子学报, 2024, 52(09): 3172-3184.

HU Yu-hong, WANG De-guang, YANG Ming, et al. Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning[J]. Acta Electronica Sinica, 2024, 52(09): 3172-3184.
胡瑜洪, 王德光, 杨明, 等. 基于强化学习的离散事件系统最优定向监控[J]. 电子学报, 2024, 52(09): 3172-3184. DOI：10.12263/DZXB.20221267

HU Yu-hong, WANG De-guang, YANG Ming, et al. Optimal Directed Control of Discrete Event Systems Based on Reinforcement Learning[J]. Acta Electronica Sinica, 2024, 52(09): 3172-3184. DOI：10.12263/DZXB.20221267

摘要

对于多个可控事件（控制指令）允许同时执行的情形，离散事件系统的监控器进行随机选择.然而，在实际应用中，如交通调度、机器人路径规划，可控事件的定向选择和数值优化是必须要考虑和解决的两个问题.对此，引入一种优化机制量化控制成本，将监督控制理论与强化学习结合，提出一种基于强化学习的离散事件系统最优定向监控器求解方法，使被控系统实现以下三个目标：（1）遵循安全性和活性控制规范；（2）每个状态下至多允许一个可控事件执行；（3）从初始状态到标记状态事件执行累计成本最小.首先，建立系统和控制规范的自动机模型，做同步积运算后可得到目标模型，通过定义的成本函数为目标模型中每个事件的执行赋予成本.其次，利用监督控制理论求解无阻塞且行为最大许可的监控器.最后，将监控器转化为马尔可夫决策过程，并利用Q学习算法求解出最优定向监控器.使用单向列车导轨控制案例和多轨道列车控制案例验证所提方法的有效性和正确性.仿真结果表明，所提出方法能够实现系统的无阻塞定向控制，并且使得定向监控器的数值成本最小.

Abstract

In the case that several controllable events (control commands) are allowed to execute simultaneously

the supervisor in the framework of discrete event systems (DESs) selects one randomly. However

in practical applications

such as traffic scheduling and robot path planning

the problems of directed control and numerical optimization should be considered. This paper introduces an optimization mechanism to quantify the control cost and combines supervisory control theory (SCT) with reinforcement learning. A systematic procedure is proposed to synthesize the optimal directed supervisor of a DES based on reinforcement learning

which makes the controlled system achieve the following three goals: (1) the control specifications relevant to security and liveness are not violated; (2) at most one controllable event can be executed at each state; (3) the cumulative cost of event execution from the initial state to a mark state is minimal. First

given the automaton models of the plant and specifications

the target automaton model is obtained by the synchronous operation of these two models; a cost function is defined and assigns the execution cost for each event in the target model. Second

the non-blocking and maximally permissive supervisor is synthesized by SCT. Finally

the supervisor is transformed into a Markov decision process and then the Q-learning algorithm is utilized to compute the optimal directed supervisor. Two applications are used to verify the effectiveness and correctness of the proposed method. The simulation results show that the proposed method can realize the directed control of the system

and the numerical cost of the directed supervisor is minimized.

关键词

Keywords

references

RAMADGE P J G , WONHAM W M . The control of discrete event systems [J ] . Proceedings of the IEEE , 1989 , 77 ( 1 ): 81 - 98 .

CAI K , WONHAM W M . Supervisory control of discrete-event systems [M ] // Encyclopedia of Systems and Control . Cham : Springer , 2021 : 2245 - 2253 .

辛宗洋 . 基于RW框架的离散事件系统监督控制理论计算和仿真平台 [J ] . 工业控制计算机 , 2009 , 22 ( 12 ): 39 - 41, 43 .

XIN Z Y . Computation and simulation platform for supervisory control of discrete event system based on RW structure [J ] . Industrial Control Computer , 2009 , 22 ( 12 ): 39 - 41, 43 . (in Chinese)

焦亭 , 刘振国 . 组件多工作模式下的离散事件系统监督控制方法研究 [J ] . 控制理论与应用 , 2020 , 37 ( 3 ): 534 - 539 .

JIAO T , LIU Z G . Study on the supervisory control of discrete-event systems incorporating components with several working modes [J ] . Control Theory & Applications , 2020 , 37 ( 3 ): 534 - 539 . (in Chinese)

史晶锌 , 舒少龙 , 林峰 , 等 . 基于监督控制理论的家庭住宅用电安全性控制研究 [J ] . 现代建筑电气 , 2014 , 5 ( 1 ): 9 - 15 .

SHI J X , SHU S L , LIN F , et al . Control for safety of home electric usage based on supervisory control theory [J ] . Modern Architecture Electric , 2014 , 5 ( 1 ): 9 - 15 . (in Chinese)

荣胜波 , 朱军 , 史勃 , 等 . 监督控制理论在多任务机床控制系统中的应用 [J ] . 机床与液压 , 2011 , 39 ( 22 ): 85 - 87 .

RONG S B , ZHU J , SHI B , et al . The application of supervisory control theory to control system of multiple-task machine tool [J ] . Machine Tool & Hydraulics , 2011 , 39 ( 22 ): 85 - 87 . (in Chinese)

TATSUMOTO Y , SHIRAISHI M , CAI K , et al . Application of online supervisory control of discrete-event systems to multi-robot warehouse automation [J ] . Control Engineering Practice , 2018 , 81 : 97 - 104 .

GONZALEZ A G C , ALVES M V S , VIANA G S , et al . Supervisory control-based navigation architecture: A new framework for autonomous robots in industry 4.0 environments [J ] . IEEE Transactions on Industrial Informatics , 2018 , 14 ( 4 ): 1732 - 1743 .

UMEMOTO H , YAMASAKI T . Optimal LLP supervisor for discrete event systems based on reinforcement learning [C ] // 2015 IEEE International Conference on Systems, Man, and Cybernetics . Piscataway : IEEE , 2015 : 545 - 550 .

KAYMAKCI O , ANIK V G , USTOGLU I . A local modular supervisory controller for a real railway station [C ] // 5th IET International Conference on System Safety 2010 . Manchester : IET , 2010 : 1 - 6 .

PASSINO K M , ANTSAKLIS P J . On the optimal control of discrete event systems [C ] // Proceedings of the 28th IEEE Conference on Decision and Control . Piscataway : IEEE , 1989 : 2713 - 2718 .

BARBEAU M , FRAPPIER M , KABANZA F . A supervisory control synthesis case study: The antenna control system [C ] // Allerton Conference on Communication, Control, and Computing . Nederlanden : Elsevier , 1997 : 533 - 542 .

HUANG J , KUMAR R . An optimal directed control framework for discrete event systems [J ] . IEEE Transactions on Systems, Man, and Cybernetics , Part A: Systems and Humans, 2007 , 37 ( 5 ): 780 - 791 .

HUANG J , KUMAR R . Optimal nonblocking directed control of discrete event systems [C ] // 2007 American Control Conference . Piscataway : IEEE , 2007 : 4285 - 4290 .

王飞跃 , 陈俊龙 . 智能控制方法与应用 [M ] . 北京 : 中国科学技术出版社 , 2020 : 65 - 80 .

WANG F Y , CHEN J L . Intelligent Control Method and Application [M ] . Beijing : China Science and Technology Press , 2020 : 65 - 80 . (in Chinese)

王玺 . 实时系统的基于优先级的实时重构和不基于优先级的条件剥夺调度算法 [D ] . 西安 : 西安电子科技大学 , 2016 .

WANG X . Priority-based Reconfiguration and Priority-free Conditionally-preemptive Scheduling of Real-time Systems [D ] . Xi'an : Xidian University , 2016 . (in Chinese)

张涛 , 张文涛 , 代凌 , 等 . 基于序贯博弈多智能体强化学习的综合模块化航空电子系统重构方法 [J ] . 电子学报 , 2022 , 50 ( 4 ): 954 - 966 .

ZHANG T , ZHANG W T , DAI L , et al . Integrated modular avionics system reconstruction method based on dequential game multi-agent teinforcement learning [J ] . Acta Electronica Sinica , 2022 , 50 ( 4 ): 954 - 966 . (in Chinese)

袁都佳 . 基于Q学习的钢琴指法自动生成方法研究 [D ] . 广州 : 华南理工大学 , 2020 .

YUAN D J . Research on Automatic Generation Method of Piano Fingering Based on Q-learning [D ] . Guangzhou : South China University of Technology , 2020 . (in Chinese)

MADANI O , THORUP M , ZWICK U . Discounted deterministic Markov decision processes and discounted all-pairs shortest paths [J ] . ACM Transactions on Algorithms , 2010 , 6 ( 2 ): 1 - 25 .

SHUAI B , LI Y F , ZHOU Q , et al . Supervisory control of the hybrid off-highway vehicle for fuel economy improvement using predictive double Q-learning with backup models [J ] . Journal of Central South University , 2022 , 29 ( 7 ): 2266 - 2278 .

张德干 , 葛辉 , 刘晓欢 , 等 . 一种基于Q-Learning策略的自适应移动物联网路由新算法 [J ] . 电子学报 , 2018 , 46 ( 10 ): 2325 - 2332 .

ZHANG D G , GE H , LIU X H , et al . A kind of new routing algorithm with adaptivity for mobile IOT based on Q-Learning [J ] . Acta Electronica Sinica , 2018 , 46 ( 10 ): 2325 - 2332 . (in Chinese)

俞凯 . 基于图割/图搜索的医学影像分割算法研究及应用 [D ] . 苏州 : 苏州大学 , 2020 .

YU K . Study of Graph Cut/Search Based Medical Image Segmentation Algorithms and Applications [D ] . Suzhou : Soochow University , 2020 . (in Chinese)

徐超群 , 易忠 , 孟立飞 , 等 . 基于遗传算法的卫星磁模型研究 [J ] . 电子学报 , 2020 , 48 ( 6 ): 1108 - 1112 .

XU C Q , YI Z , MENG L F , et al . Research on satellite magnetic model based on genetic algorithm [J ] . Acta Electronica Sinica , 2020 , 48 ( 6 ): 1108 - 1112 . (in Chinese)

廖伟志 , 夏小云 , 贾小军 . 基于蚁群算法的多路径覆盖测试数据生成 [J ] . 电子学报 , 2020 , 48 ( 7 ): 1330 - 1342 .

LIAO W Z , XIA X Y , JIA X J . Test data generation for multiple paths coverage based on ant colony algorithm [J ] . Acta Electronica Sinica , 2020 , 48 ( 7 ): 1330 - 1342 . (in Chinese)

许凯波 , 鲁海燕 , 黄洋 , 等 . 基于双层蚁群算法和动态环境的机器人路径规划方法 [J ] . 电子学报 , 2019 , 47 ( 10 ): 2166 - 2176 .

XU K B , LU H Y , HUANG Y , et al . Robot path planning based on double-layer ant colony optimization algorithm and dynamic environment [J ] . Acta Electronica Sinica , 2019 , 47 ( 10 ): 2166 - 2176 . (in Chinese)

ZIELINSKI K M C , HENDGES L V , FLORINDO J B , et al . Flexible control of discrete event systems using environment simulation and reinforcement learning [J ] . Applied Soft Computing , 2021 , 111 : 107714 .

SHI H R , LIU G J , ZHANG K W , et al . MARL Sim2real transfer: Merging physical reality with digital virtuality in metaverse [J ] . IEEE Transactions on Systems, Man, and Cybernetics: Systems , 2023 , 53 ( 4 ): 2107 - 2117 .

ZHOU Z Y , LIU G J . RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states [EB/OL ] . ( 2022-05-15 )[ 2022-08-02 ] . http://arxiv.org/abs/2205.07229 http://arxiv.org/abs/2205.07229 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Satellite Security Transmission Optimization Algorithm for Joint Anomaly Detection

CS-ROMF:Efficient Community Search Based on Graph Combinatorial Optimization

Knowledge-Based and Data-Driven Integrating Design Methodology for Air Combat Strategy in Multi-Opponent Adversarial Game

Reinforcement Learning Based Tuning-free Plug-and-Play Image Reconstruction Method for Single Photon Imaging

Related Author

WANG Xi

HUANG Chen

WANG Ding-xuan

HOU Rong-hui

ZHANG An-ran

WANG Xing-fen

ZHAO Yu-han

LI Li-bo

Related Institution

School of Eletro-Mechanical Engineering, Xidian University, , Shaani

School of Cyber Engineering, Xidian University

Beijing Information Science and Technology University

Hong Kong Baptist University, Hongkong

Institute of Automation, Chinese Academy of Sciences

⁰