基于Transformer动态场景信息生成对抗网络的行人轨迹预测方法

doi:10.12263/DZXB.20210762

电子学报 ›› 2022, Vol. 50 ›› Issue (7): 1537-1547.DOI: 10.12263/DZXB.20210762

• 学术论文 • 下一篇

基于Transformer动态场景信息生成对抗网络的行人轨迹预测方法

裴炤¹^,², 邱文涛², 王淼³(), 马苗², 张艳宁⁴^,⁵

^1.陕西师范大学现代教学技术教育部重点实验室, 陕西西安 710119
^2.陕西师范大学计算机科学学院, 陕西西安 710119
^3.上海交通大学航空航天学院, 上海 200240
^4.空天地海一体化大数据应用技术国家工程实验室, 陕西西安 710129
^5.西北工业大学计算机学院, 陕西西安 710129

收稿日期:2021-06-16 修回日期:2021-11-07 出版日期:2022-07-25
- 通讯作者:
- 王淼
- 作者简介:
- 裴　炤　男，1983年2月生，陕西西安人，博士、教授、博士生导师.主要从事计算机视觉与人工智能、图像处理与模式识别、机器学习的相关研究.E-mail：zpei@snnu.edu.cn
  邱文涛（1984-）男，1996年8月生，山东枣庄人.陕西师范大学计算机科学学院研究生，主要研究方向为计算机视觉和行人轨迹预测.E-mail:qiuwentao@snnu.edu.cn
  王淼男，1981年7月生，河南义马人，博士，上海交通大学航空航天学院助理研究员，主要研究方向为智能信息处理、数据挖掘、计算机视觉.E-mail:miaowang@sjtu.edu.cn
- 基金资助:
- 国家自然科学基金 (61971273); 陕西省重点研发计划 (2021GY-032); 中央高校基本科研业务 (GK202003077); 上海市自然科学基金 (20ZR1427800)

Pedestrian Trajectory Prediction Method Using Dynamic Scene Information Based Transformer Generative Adversarial Network

PEI Zhao¹^,², QIU Wen-tao², WANG Miao³(), MA Miao², ZHANG Yan-ning⁴^,⁵

^1.Key Laboratory of Modern Teaching Technology（Ministry of Education）, Shaanxi Normal University, Xi’an, Shaanxi 710119, China
^2.School of Computer Science, Shaanxi Normal University, Xi’an, Shaanxi 710119, China
^3.School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai 200240, China
^4.National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, Xi’an, Shaanxi 710129, China
^5.School of Computer Science, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China

Received:2021-06-16 Revised:2021-11-07 Online:2022-07-25 Published:2022-07-30
- Corresponding author:
- WANG Miao
- Supported by:
- National Natural Science Foundation of China (61971273); Key R&D Project of Shaanxi Province (2021GY-032); Fundamental Research Funds for the Central Universities (GK202003077); National Natural Science Foundation of Shanghai Municipality, China (20ZR1427800)

摘要/Abstract

摘要：

行人轨迹预测是视频监控的重要组成部分，因现有方法未充分利用场景特征信息造成其预测轨迹不符合生活常识，导致行人轨迹预测精度较低出现明显偏离真实轨迹的情况.针对上述不足本文提出一种基于Transformer动态场景信息生成对抗网络（Generative Adversarial Network，GAN）的行人轨迹预测方法.该方法利用动态场景特征提取模块的卷积神经网络（Convolutional Neural Networks，CNN）模型对目标行人的动态场景信息进行特征提取，同时生成器网络中的编码器利用Transformer对行人的社会交互信息特征以及轨迹信息特征进行建模.在ETH和UCY数据集上的实验结果表明，与Social GAN模型相比，本文方法在多个场景下的平均位移误差准确率提高了25.61%，最终位移误差准确率提高了38.44%.

关键词: 行人轨迹预测, 生成对抗网络, 转换器, 深度学习, 长短期记忆网络

Abstract:

Pedestrian trajectory prediction is an important part of video surveillance. The current methods are not accurate and sometimes violate common senses because scene information is not fully used. To eliminate the above shortcomings, this paper proposes a transformer generated adversarial network(GAN) algorithm which combines dynamic scene information with pedestrian social interaction information. The convolution neural network model of the dynamic scene extraction module is utilized to extract the dynamic scene information features of the target pedestrian, and the encoder in the generator network uses transformer to model the features of social interaction information and trajectory information of pedestrians. Experimental results on ETH and UCY datasets show that, compared with social GAN model, our method improves the accuracy of average displacement error by 25.61% and the accuracy of average final displacement error by 38.44% in multiple scenarios.

Key words: pedestrian trajectory prediction, generative adversarial networks, transformer, deep learning, long short-term memory

中图分类号:

TP391

裴炤, 邱文涛, 王淼, 马苗, 张艳宁. 基于Transformer动态场景信息生成对抗网络的行人轨迹预测方法[J]. 电子学报, 2022, 50(7): 1537-1547.

PEI Zhao, QIU Wen-tao, WANG Miao, MA Miao, ZHANG Yan-ning. Pedestrian Trajectory Prediction Method Using Dynamic Scene Information Based Transformer Generative Adversarial Network[J]. Acta Electronica Sinica, 2022, 50(7): 1537-1547.

图/表 8

参考文献 32

1	PEIZ, QIX, ZHANGY, et al. Human trajectory prediction in crowded scene using social-affinity long short-term memory[J]. Pattern Recognition, 2019, 93: 273-282.
2	YAMAGUCHIK, BERGA C, ORTIZL E, et al. Who are you with and where are you going?[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs: IEEE, 2011: 1345-1352.
3	DESOUZAG N, KAKA C. Vision for mobile robot navigation: A survey[J]. IEEE Trans on Pattern Analysis and Machine Intelligence, 2002, 24(2): 237-267.
4	RUDENKOA, PALMIERIL, HERMANM, et al. Human motion trajectory prediction: A survey [J]. The International Journal of Robotics Research, 2020. 39(8): 895-935.
5	李康, 李亚敏, 胡学敏, 等. 基于卷积神经网络的鲁棒高精度目标跟踪算法[J]. 电子学报, 2018, 46(9): 2087-2093.
	LIK, LIY M, HUX M, et al. A robust and accurate object tracking algorithm based on convolutional neural network[J]. Acta Electronica Sinica, 2018, 46(9): 2087-2093.(in Chinese)
6	马少雄, 邱实, 唐颖, 等. 基于工地场景的深度学习目标跟踪算法 [J]. 电子学报, 2020, 48(9): 1665-1671.
	MAS X, QIUS, TANGY, et al. Deep learning target tracking algorithm based on construction site scene[J]. Acta Electronica Sinica, 2020, 48(9): 1665-1671. (in Chinese)
7	S-CB LO, CHANH-P, LINJ-S, et al. Artificial convolution neural network for medical image pattern recognition[J]. Neural Networks, 1995, 8(7-8): 1201-1214.
8	PELLEGRINIS, ESS A, VANG L. Improving data association by joint modeling of pedestrian trajectories and groupings[C]//Proceedings of the European Conference on Computer Vision. Crete: Springer, 2010: 452-465.
9	LERNERA, CHRYSANTHOUY, LISCHINSKID. Crowds by example[C]//Proceedings of the Computer Graphics Forum. Oxford: Blackwell Publishing Ltd, 2007: 26(3): 655-664.
10	HELBINGD, MOLNARP. Social force model for pedestrian dynamics[J]. Physical Review E, 1995, 51(5): 4282-4286.
11	KITANIK M, ZIEBARTB D, BAGNELLJ A, et al. Activity forecasting[C]//Proceedings of the European Conference on Computer Vision. Florence, Italy: Springer, 2012: 201-214.
12	LEEN, CHOIW, VERNAZAP, et al. Desire: Distant future prediction in dynamic scenes with interacting agents[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Hawaii, USA: IEEE, 2017: 336-345.
13	PELLEGRINIS, ESS A, SCHINDLERK, et al. You'll never walk alone: Modeling social behavior for multi-target tracking[C]//Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. Kyoto, Japan: IEEE, 2009: 261-268.
14	MOUSSAIDM, PEROZON, GARNIERS, et al. The walking behaviour of pedestrian social groups and its impact on crowd dynamics[J]. Plos One, 2010, 5(3): e10047.
15	XUY, PIAOZ, GAOS. Encoding crowd interaction with deep neural network for pedestrian trajectory prediction[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018: 5275-5284.
16	ZHAOT, XUY, MONFORTM, et al. Multi-agent tensor fusion for contextual trajectory prediction[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 12126-12134.
17	ALAHIA, RAMANATHANV, FEIF L. Socially-aware large-scale crowd forecasting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014: 2203-2210.
18	ALAHIA, GOELK, RAMANATHANV, et al. Social LSTM: Human trajectory prediction in crowded spaces[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 961-971.
19	BALLANL, CASTALDOF, ALAHIA, et al. Knowledge transfer for scene-specific motion prediction[C]//Proceedings of the European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016: 697-713.
20	LIUJ, SHAHROUDYA, XUD, et al. Spatio-temporal LSTM with trust gates for 3d human action recognition[C]//Proceedings of the European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016: 816-833.
21	ROBICQUETA, SADEGHIANA, ALAHIA, et al. Learning social etiquette: Human trajectory understanding in crowded scenes[C]//Proceedings of the European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016: 549-565.
22	ALTCHÉF, DELA. An LSTM network for highway trajectory prediction[C]//Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems. Yokohama, Japan: IEEE, 2017: 353-359.
23	CHENGB, XUX, ZENGY J, et al. Pedestrian trajectory prediction via the social-grid LSTM model[J]. Journal of Engineering-Joe, 2018, 2018(16): 1468-1474.
24	FERNANDOT, DENMANS, SRIDHARANS, et al. Soft+hardwired attention: An LSTM framework for human trajectory prediction and abnormal event detection[J]. Neural Network, 2018, 108(1): 466-478.
25	ZHANGP, OUYANGW, ZHANGP, et al. SR-LSTM: State refinement for LSTM towards pedestrian trajectory prediction[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 12085-12094.
26	金苍宏, 董腾然, 陈天翼, 等. 融合序列分解与时空卷积的时序预测算法[J]. 电子学报, 2021, 49(2): 233-238.
	JINC H, DONGT R, CHENT Y, et al. Spatio-temporal convolutional forecasting based on time-series decomposition strategy [J]. Acta Electronica Sinica, 2021, 49(2): 233-238. (in Chinese)
27	GUPTAA, JOHNSONJ, FEIF L, et al. Social GAN: Socially acceptable trajectories with generative adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018: 2255-2264.
28	SADEGHIANA, KOSARAJUV. et al. Sophie: An attentive GAN for predicting paths compliant to social and physical constraints[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019: 1349-1358.
29	李志欣, 孙亚茹, 唐素勤, 等. 双路注意力引导图卷积网络的关系抽取[J]. 电子学报, 2021, 49(2): 315-323.
	LiZ X, SunY R, TangS Q, et al. Dual attention guided graph convolutional networks for relation extraction[J]. Acta Electronica Sinica, 2021, 49(2): 315-323. (in Chinese)
30	VEMULAA, MUELLINGK, OH J. Social attention: Modeling attention in human crowds[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation. China: IEEE, 2018: 4601-4607.
31	KOSARAJUV, SADEGHIANA, MARTÍNM R, et al. Social-BIGAT: Multimodal trajectory forecasting using bicycle-GAN and graph attention networks[C]//Proceedings of the Advances in Neural Information Processing Systems. Vancouver, Canada: NIPS, 2019: 137-146.
32	DAIZ, YANGZ, YANGY, et al. Transformer-XL: Attentive language models beyond a fixed-length context[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: ACL,2019:2978-2988.

Metric	Dataset	LSTM	Social-LSTM	Social-GAN	Sophie	Social-BiGAT	本文方法
ADE	ETH	1.09	1.09	0.81	0.86	0.69	0.65
	Hotel	0.86	0.79	0.72	0.76	0.49	0.34
	Univ	0.61	0.67	0.60	0.54	0.55	0.53
	Zara1	0.41	0.47	0.34	0.30	0.30	0.32
	Zara2	0.52	0.56	0.42	0.38	0.36	0.31
Average		0.70	0.72	0.58	0.61	0.48	0.44
FDE	ETH	2.41	2.35	1.52	1.65	1.29	1.18
	Hotel	1.91	1.76	1.61	1.67	1.01	0.64
	Univ	1.31	1.40	1.84	1.24	1.32	1.15
	Zara1	1.88	1.00	1.26	0.63	0.62	0.66
	Zara2	1.11	1.17	0.69	0.78	0.75	0.63
Average		1.52	1.54	1.38	1.24	1.00	0.89

Metric	Dataset	LSTM	Social-LSTM	Social-GAN	Sophie	Social-BiGAT	本文方法
ADE	ETH	1.09	1.09	0.81	0.86	0.69	0.65
	Hotel	0.86	0.79	0.72	0.76	0.49	0.34
	Univ	0.61	0.67	0.60	0.54	0.55	0.53
	Zara1	0.41	0.47	0.34	0.30	0.30	0.32
	Zara2	0.52	0.56	0.42	0.38	0.36	0.31
Average		0.70	0.72	0.58	0.61	0.48	0.44
FDE	ETH	2.41	2.35	1.52	1.65	1.29	1.18
	Hotel	1.91	1.76	1.61	1.67	1.01	0.64
	Univ	1.31	1.40	1.84	1.24	1.32	1.15
	Zara1	1.88	1.00	1.26	0.63	0.62	0.66
	Zara2	1.11	1.17	0.69	0.78	0.75	0.63
Average		1.52	1.54	1.38	1.24	1.00	0.89

Metric	Dataset	Baseline	Dynamic scene information	Transformer	Dynamic scene information+ Transformer
ADE	ETH	0.81	0.64	0.68	0.65
	Hotel	0.72	0.38	0.32	0.34
	Univ	0.60	0.7	0.64	0.53
	Zara1	0.34	0.35	0.37	0.32
	Zara2	0.42	0.34	0.35	0.31
Average		0.58	0.48	0.47	0.43
FDE	ETH	1.52	1.1	1.21	1.18
	Hotel	1.61	0.72	0.59	0.64
	Univ	1.84	1.35	1.24	1.15
	Zara1	1.26	0.77	0.76	0.66
	Zara2	0.69	0.65	0.73	0.63
Average		1.39	0.91	0.90	0.85

Metric	Dataset	Baseline	Dynamic scene information	Transformer	Dynamic scene information+ Transformer
ADE	ETH	0.81	0.64	0.68	0.65
	Hotel	0.72	0.38	0.32	0.34
	Univ	0.60	0.7	0.64	0.53
	Zara1	0.34	0.35	0.37	0.32
	Zara2	0.42	0.34	0.35	0.31
Average		0.58	0.48	0.47	0.43
FDE	ETH	1.52	1.1	1.21	1.18
	Hotel	1.61	0.72	0.59	0.64
	Univ	1.84	1.35	1.24	1.15
	Zara1	1.26	0.77	0.76	0.66
	Zara2	0.69	0.65	0.73	0.63
Average		1.39	0.91	0.90	0.85

预测模型	预测耗费时间/ms
LSTM	2.7
Social-LSTM	4.2
Social-GAN	29.4
本文方法	34.3

基于Transformer动态场景信息生成对抗网络的行人轨迹预测方法

Pedestrian Trajectory Prediction Method Using Dynamic Scene Information Based Transformer Generative Adversarial Network

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 32

相关文章 15

编辑推荐

Metrics

[1]	吴靖, 叶晓晶, 黄峰, 陈丽琼, 王志锋, 刘文犀. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
[2]	李雪莹, 王田路, 梁鹏, 王翀. 基于系统模型的用户评论中非功能需求的自动分类[J]. 电子学报, 2022, 50(9): 2079-2089.
[3]	琚长瑞, 秦晓燕, 袁广林, 李豪, 朱虹. 尺度敏感损失与特征融合的快速小目标检测方法[J]. 电子学报, 2022, 50(9): 2119-2126.
[4]	张志昌, 于沛霖, 庞雅丽, 朱林, 曾扬扬. SMGN：用于对话状态跟踪的状态记忆图网络[J]. 电子学报, 2022, 50(8): 1851-1858.
[5]	张亚洲, 俞洋, 朱少林, 陈锐, 戎璐, 梁辉. 一种量子概率启发的对话讽刺识别网络模型[J]. 电子学报, 2022, 50(8): 1885-1893.
[6]	王飞扬, 冀鹏欣, 孙笠, 危倩, 李根, 张忠宝. 一种基于深度学习的动态社交网络用户对齐方法[J]. 电子学报, 2022, 50(8): 1925-1936.
[7]	徐兴荣, 刘聪, 李婷, 郭娜, 任崇广, 曾庆田. 基于双向准循环神经网络和注意力机制的业务流程剩余时间预测方法[J]. 电子学报, 2022, 50(8): 1975-1984.
[8]	欧阳与点, 谢鲲, 谢高岗, 文吉刚. 面向大规模网络测量的数据恢复算法：基于关联学习的张量填充[J]. 电子学报, 2022, 50(7): 1653-1663.
[9]	刘文杰, 赵胶胶, 张颖, 葛业波. 一种量子条件生成对抗网络算法[J]. 电子学报, 2022, 50(7): 1586-1593.
[10]	彭闯, 王伦文, 胡炜林. 融合深度特征的电磁频谱异常检测算法[J]. 电子学报, 2022, 50(6): 1359-1369.
[11]	杨伟超, 杜宇, 文伟, 侯舒维, 徐常志, 张建华. 基于多重分形谱智能分析的卫星信号调制识别研究[J]. 电子学报, 2022, 50(6): 1336-1343.
[12]	李政伟, 李佳树, 尤著宏, 聂茹, 赵欢, 钟堂波. 基于异质图注意力网络的miRNA与疾病关联预测算法[J]. 电子学报, 2022, 50(6): 1428-1435.
[13]	张波, 陆云杰, 秦东明, 邹国建. 一种卷积自编码深度学习的空气污染多站点联合预测模型[J]. 电子学报, 2022, 50(6): 1410-1427.
[14]	廖勇, 李玉杰. 一种轻量化低复杂度的FDD大规模MIMO系统CSI反馈方法[J]. 电子学报, 2022, 50(5): 1211-1217.
[15]	冀振燕, 韩梦豪, 宋晓军, 冯其波. 面向激光光条图像修复的循环相似度映射网络[J]. 电子学报, 2022, 50(5): 1234-1242.