一种新的基于值函数迁移的快速Sarsa算法

doi:10.3969/j.issn.0372-2112.2014.11.005

电子学报 ›› 2014, Vol. 42 ›› Issue (11): 2157-2161.DOI: 10.3969/j.issn.0372-2112.2014.11.005

一种新的基于值函数迁移的快速Sarsa算法

傅启明¹, 刘全^1,2, 尤树华¹, 黄蔚¹, 章晓芳¹

1. 苏州大学计算机科学与技术学院, 江苏苏州 215006;
2. 吉林大学符号计算与知识工程教育部重点实验室, 吉林长春 130012

收稿日期:2013-09-12 修回日期:2014-03-03 出版日期:2014-11-25
- 通讯作者:
- 刘全
- 作者简介:
- 傅启明 男,1985年生于江苏淮安,博士生.主要研究方向为强化学习、贝叶斯推理及遗传算法.
- 基金资助:
- 国家自然科学基金 (No.61103045,No.61303108); 江苏省自然科学基金 (No.BK2012616); 江苏省高校自然科学研究项目 (No.13KJB520020); 吉林大学符号计算与知识工程教育部重点实验室资助 (No.93K17012K04)

A Novel Fast Sarsa Algorithm Based on Value Function Transfer

FU Qi-ming¹, LIU Quan^1,2, YOU Shu-hua¹, HUANG Wei¹, ZHANG Xiao-fang¹

1. Institute of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006, China;
2. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, Jilin 130012, China

Received:2013-09-12 Revised:2014-03-03 Online:2014-11-25 Published:2014-11-25

摘要/Abstract

摘要：

知识迁移是当前机器学习领域的一个新的研究热点.其基本思想是通过将经验知识从历史任务到目标任务的迁移,达到提高算法收敛速度和收敛精度的目的.针对当前强化学习领域中经典算法收敛速度慢的问题,提出在学习过程中通过迁移值函数信息,减少算法收敛所需要的样本数量,加快算法的收敛速度.基于强化学习中经典的在策略Sarsa算法的学习框架,结合值函数迁移方法,优化算法初始值函数的设置,提出一种新的基于值函数迁移的快速Sarsa算法——VFT-Sarsa.该算法在执行前期,通过引入自模拟度量方法,在状态空间以及动作空间一致的情况下,对目标任务中的状态与历史任务中的状态之间的距离进行度量,对其中相似并满足一定条件的状态进行值函数迁移,而后再通过学习算法进行学习.将VTF-Sarsa算法用于Random Walk问题,并与经典的Sarsa算法、Q学习算法以及具有较好收敛速度的QV算法进行比较,实验结果表明,该算法在保证收敛精度的基础上,具有更快的收敛速度.

关键词: 强化学习, VFT-Sarsa算法, 自模拟度量, 值函数迁移

Abstract:

Knowledge Transfer has gradually became a research hot pot in machine learning,which tries to transfer the knowledge from the historical tasks to the target task in order to speed up the convergence rate and improve the performance of algorithms.With respect to the slow convergence rate of traditional reinforcement learning algorithms,this paper proposed to transfer the value function between different similar learning tasks with the same state space and action space,which tries to reduce the needed samples in the target task and speed up the convergence rate.Based on the framework of on-policy Sarsa algorithm,combined with the value function transfer method,this paper put forward a novel fast Sarsa algorithm based on the value function transfer—VFT-Sarsa.At the beginning,the algorithm uses Bisimulation metric to measure the distance between states in target task and historical task on the condition that these tasks have the same state space and action space,transfers the value function if the distance meets some condition,and finally executes the learning algorithm.At the end,apply the proposed algorithm in Random Walk,compared with Sarsa algorithm,Q-Learning and QV algorithm,the results show that the proposed algorithm can get a better convergence rate with a good performance.

Key words: reinforcement learning, VFT-Sarsa algorithm, bisimulation metric, value function transfer

中图分类号:

TP181

傅启明, 刘全, 尤树华, 黄蔚, 章晓芳. 一种新的基于值函数迁移的快速Sarsa算法[J]. 电子学报, 2014, 42(11): 2157-2161.

FU Qi-ming, LIU Quan, YOU Shu-hua, HUANG Wei, ZHANG Xiao-fang. A Novel Fast Sarsa Algorithm Based on Value Function Transfer[J]. Acta Electronica Sinica, 2014, 42(11): 2157-2161.

参考文献

[1] Taylor J,Precup D,Panangaden P.Bounding performance loss in approximate MDP homomorphisms[A].Proceedings of the 22nd Annual Conference on Neural Information Processing Systems[C].NY:Curran Associates,2008.1660-1667.
[2] 王皓,高阳,陈兴国.强化学习中的迁移:方法和进展[J].电子学报,2008,36(12A):39-43. Wang Hao,Gao Yang,Chen Xinguo.Transfer of reinforcement learning:The state of the art[J].Acta Electronica Sinica,2008,36(12A):39-43.(in Chinese)
[3] Sunmola F T,Wyatt J L.Model transfer for Markov decision tasks via parameter matching[A].Proceedings of the 25th Workshop of the UK Planning and Scheduling Special Interest Group[C].Nottingham,England,2006.17-24.
[4] Konidaris G D,Barto A G.Building portable options:skill transfer in reinforcement learning[A].Proceedings of the 20th International Joint Conference on Artificial Intelligence[C].CA:Morgan Kaufmann Publishers,2007.895-901.
[5] Ferrante E,Lazaric A,Restelli M.Transfer of task representation in reinforcement learning using policy-based proto-value functions[A].Proceedings of the 7th International Conference on Autonomous Agents and Multi-Agent Systems[C].Estoril:,2008.1329-1332.
[6] Lazaric A,Restelli M,Bonarini A.Transfer of samples in batch reinforcement learning[A].Proceedings of the 25th International Conference on Machine Learning[C].NY:ACM Press,2008.544-551.
[7] Sorg J,Singh S.Transfer via soft homomorphisms[A].Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems[C].Hungary:,2009.741-748.
[8] Ammar H B,Taylor M,Tuyls K,Weiss G.Reinforcement learning transfer using a sparse coded inter-task mapping[A].Proceedings of the 9th European Workshop on Multi-agent Systems[C].Berlin:Springer-verlag,2012.1-16.
[9] Konidaris G D,Scheidwasser I and Barto A G.Transfer in Reinforcement Learning via Shared Features[J].Journal of Machine Learning Research,2012,13:1333-1371.
[10] Sutton R S,Barto A G.Reinforcement Learning[M].Cambridge:MIT Press,1998.
[11] Givan R,Dean T,Greig M.Equivalence notions and model minimization in Markov decision processes[J].Artificial Intelligence,2003,147(1-2):163-223.
[12] Ferns N,Panangaden P,Precup D.Metrics for finite markov decision processes[A].Proceeding of the 20th Conference on Uncertainty in Artificial Intelligence[C].Arlington:AUAI Press,2004.162-169.
[13] Wiering M andHasselt H V.The QV family compared to other reinforcement learning algorithms[A].Proceedings of IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning[C].Nashville:IEEE,2009.101-108.

[1]	段世红, 何昊, 徐诚, 殷楠, 王然. 未知环境下基于深度序列蒙特卡罗树搜索的信源导航方法[J]. 电子学报, 2022, 50(7): 1744-1752.
[2]	宋佰霖, 许华, 齐子森, 饶宁, 彭翔. 一种基于深度强化学习的协同通信干扰决策算法[J]. 电子学报, 2022, 50(6): 1301-1309.
[3]	饶宁, 许华, 蒋磊, 宋佰霖, 史蕴豪. 基于多智能体深度强化学习的分布式协同干扰功率分配算法[J]. 电子学报, 2022, 50(6): 1319-1330.
[4]	卓亚琦, 魏家辉, 李志欣. 基于双注意模型的图像描述生成方法研究[J]. 电子学报, 2022, 50(5): 1123-1130.
[5]	易令, 李泽平. 基于深度强化学习的码率自适应算法研究[J]. 电子学报, 2022, 50(5): 1192-1200.
[6]	张涛, 张文涛, 代凌, 陈婧怡, 王丽, 魏倩茹. 基于序贯博弈多智能体强化学习的综合模块化航空电子系统重构方法[J]. 电子学报, 2022, 50(4): 954-966.
[7]	肖硕, 黄珍珍, 张国鹏, 杨树松, 江海峰, 李天旭. 基于SAC的多智能体深度强化学习算法[J]. 电子学报, 2021, 49(9): 1675-1681.
[8]	管永明, 王刚, 骆凯波, 吕梁, 吕晓雯, 史玉良. 基于极端不平衡学习的泛化低压异常箱表关系识别研究与应用[J]. 电子学报, 2021, 49(8): 1507-1514.
[9]	冀大雄, 方文巍, 朱华, 李硕, 唐元贵, 田宇, 姚倩. 基于相对测量的水下机器人主动定位方法研究[J]. 电子学报, 2021, 49(7): 1249-1256.
[10]	周东明, 张灿龙, 李志欣, 王智文. 基于多层级视觉融合的图像描述模型[J]. 电子学报, 2021, 49(7): 1286-1290.
[11]	韩冲, 王俊丽, 吴雨茜, 张超波. 基于神经进化的深度学习模型研究综述[J]. 电子学报, 2021, 49(2): 372-379.
[12]	刘润滋, 吴伟华, 张文柱, 周笛, 张琰. 基于图学习的密集空间网络传输资源调度方法[J]. 电子学报, 2021, 49(11): 2133-2137.
[13]	石义乐, 杨文忠, 杜慧祥, 王丽花, 王婷, 理珊珊. 基于深度学习的图像描述综述[J]. 电子学报, 2021, 49(10): 2048-2060.
[14]	陈思光, 陈佳民, 赵传信. 基于深度强化学习的云边协同计算迁移研究[J]. 电子学报, 2021, 49(1): 157-166.
[15]	黄志清, 曲志伟, 张吉, 张严心, 田锐. 基于深度强化学习的端到端无人驾驶决策[J]. 电子学报, 2020, 48(9): 1711-1719.

一种新的基于值函数迁移的快速Sarsa算法

A Novel Fast Sarsa Algorithm Based on Value Function Transfer

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics