结合显著性的MVD视频整帧丢失错误隐藏

doi:10.12263/DZXB.20210393

电子学报 ›› 2022, Vol. 50 ›› Issue (9): 2146-2154.DOI: 10.12263/DZXB.20210393

结合显著性的MVD视频整帧丢失错误隐藏

崔金鹏, 周洋, 殷海兵(), 黄晓峰(), 陆宇()

杭州电子科技大学通信工程学院, 浙江杭州 310018

收稿日期:2021-03-25 修回日期:2022-02-08 出版日期:2022-09-25
- 通讯作者:
- 周洋
- 作者简介:
- 崔金鹏男，1996年9月出生于山东省潍坊市.现为杭州电子科技大学通信工程学院硕士生.主要研究方向为3D视频编解码与差错控制.E‑mail：reader@163.com
  周洋男，1979年1月出生于浙江省台州市.现为杭州电子科技大学通信工程学院副教授，硕士生导师，主要研究方向为3D视频编解码、视频图像处理.
  殷海兵男，1974年6月出生于湖北黄梅.教授，博士生导师.主要研究方向为智能信息处理.E‑mail：yhb@hdu.edu.cn
  黄晓峰男，1988年1月出生于江西婺源.主要研究方向为视频芯片架构.E‑mail：xfhuang@hdu.edu.cn
  陆宇男，1977年5月出生于广西博白.主要研究方向视频编解码.E‑mail：y_lu@hdu.edu.cn
- 基金资助:
- 浙江省自然科学基金 (Y21F020076); 国家自然科学基金 (61972123); 浙江省影视媒体技术研究重点实验室开放基金 (CM2021001)

Error Concealment for Frame Losses in MVD Video Using Saliency

CUI Jin-peng, ZHOU Yang, YIN Hai-bing(), HUANG Xiao-feng(), LU Yu()

School of Telecommunication Engineering，Hangzhou Dianzi University，Hangzhou，Zhejiang 310018，China

Received:2021-03-25 Revised:2022-02-08 Online:2022-09-25 Published:2022-10-26
- Corresponding author:
- ZHOU Yang

摘要/Abstract

摘要：

基于多视点视频加深度（Multiview Video plus Depth， MVD）格式的三维视频能给用户提供多视角的沉浸式视觉体验.针对MVD视频流传输中彩色帧发生整帧丢失的情况，提出了一种结合视觉显著性的三维视频整帧丢失错误隐藏算法.先将丢失帧各区域划分为三种不同的显著等级，然后对低、中、高显著区域，分别采用时域直接拷贝、时域和视点间像素自适应搜索法和新型的运动矢量补偿法进行恢复.相较于目前的对比算法，经提出算法重建后的丢失帧的峰值信噪比值（Peak Signal to Noise Ratio， PSNR）和结构相似度值（Structural Similarity Index Measure， SSIM）能分别提高0.99~2.61 dB和0.005~0.012，并且重建图像主观视觉质量更佳.

关键词: 多视点加深度, 错误隐藏, 整帧丢失, 视觉显著性, 运动补偿, 视差补偿

Abstract:

The 3D video based on multiview video plus depth(MVD) format can provide users with immersive experiences from different viewpoints. To solve the whole color frame losses in MVD video stream transmission, we propose a novel error concealment algorithm of whole-frames loss using visual saliency. Firstly, each region of the lost frame is classified into three different saliency levels. The low, medium and high saliency regions are recovered by direct temporal copy, adaptive pixel matching search between temporally and inter-view domains, and a new motion vector compensation, respectively. Compared with the recent algorithms, the lost frame concealed by the proposed algorithm achieves significant improvements in the average PSNR and SSIM with approximate gains of 0.99~2.61 dB and 0.005~0.012, respectively. In addition, our concealed results have better visual quality than compared methods.

Key words: multiview video plus depth, error concealment, whole frame loss, visual saliency, motion compensation, disparity compensation

中图分类号:

TP391

崔金鹏, 周洋, 殷海兵, 黄晓峰, 陆宇. 结合显著性的MVD视频整帧丢失错误隐藏[J]. 电子学报, 2022, 50(9): 2146-2154.

CUI Jin-peng, ZHOU Yang, YIN Hai-bing, HUANG Xiao-feng, LU Yu. Error Concealment for Frame Losses in MVD Video Using Saliency[J]. Acta Electronica Sinica, 2022, 50(9): 2146-2154.

图/表 11

参考文献 24

1	WANG T, LI F, QIAO X, et al. Low-complexity error resilient HEVC video coding: a deep learning approach[J]. IEEE Transactions on Image Processing, 2021, 30(2): 1245-1260.
2	XIANG X, CHENG Y. A frame loss error concealment scheme for multi-view video coding[C]//Proceedings of 2014 International Conference on Instrumentation, Measurement, Computer, Communication and Control. Harbin, China: IEEE Computer Society, 2014: 857-861.
3	KUAN Y K, LI G L, CHEN M J, et al. Error concealment algorithm using inter-view correlation for multi-view video[J]. EURASIP Journal on Image and Video Processing, 2014, 38(1): 1-11.
4	ZHOU Y, XIANG W, WANG G. Frame loss concealment for multiview video transmission over wireless multimedia sensor networks[J]. IEEE Sensors Journal, 2015, 15(3): 1892-1901.
5	YU H, RAN M, LI P, et al. A new hybrid error concealment for multi-view video[C]//Proceedings of 2015 IEEE China Summit and International Conference on Signal and Information Processing. Chengdu, China: IEEE, 2015: 176-180.
6	AHAMED Z G, SHARMA V K, JAIN A. Image error concealment using robust data hiding technique[C]//Proceedings of 2020 International Conference on Innovative Mechanisms for Industry Applications. Bangalore, India: IEEE, 2020: 470-474.
7	EL-SHAFAI W, EL-RABAIE S, EL-HALAWANY M M, et al. Recursive Bayesian filtering-based error concealment scheme for 3D video communication over severely lossy wireless channels[J]. Circuits, Systems, and Signal Processing, 2018, 37(11): 4810-4841.
8	YANG M, LAN X, ZHENG N, et al. Depth-assisted temporal error concealment for Intra frame slices in 3-D video[J]. IEEE Transactions on Broadcasting, 2014, 60(2): 385-393.
9	EL-SHAFAI W, EL-RABAIE S, EL-HALAWANY M, et al. Encoder-independent decoder-dependent depth-assisted error concealment algorithm for wireless 3D video communication[J]. Multimedia Tools and Applications, 2018, 77(11): 13145-13172.
10	HUANG P C, LIN J R, LI G L, et al. Improved depth- asisted error concealment algorithm for 3D video transmission[J]. IEEE Transactions on Multimedia, 2017, 19(11): 2625-2632.
11	TAI S C, WANG C C, HONG C S, et al. An effiicient full frame algorithm for object-based error concealment in 3D depth-based video[J]. Multimedia Tools and Applications, 2016, 75(16): 9927- 9947.
12	KHATTAK S, MAUGEY T, HAMZAOUI R, et al. Temporal and inter-view consistent error concealment technique for multiview plus depth video[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(5): 829-840.
13	LIN T L, WANG C J, DING T L, et al. Recovery of lost color and depth frames in multiview videos[J]. IEEE Transactions on Image Processing, 2018, 27(11): 5449-5463.
14	LEE P J, KUO K T. An adaptive error concealment method for depth map in 3D video coding[J]. International Journal of Fuzzy Systems, 2016, 18(1): 62-71.
15	ASSUNCAO P A, MARCELINO S, SOARES S, et al. Spatial error concealment for intra-coded depth maps in multiview video-plus-depth[J]. Multimedia Tools and Applications, 2017, 76(12): 13835-13858.
16	LEE G W, HAN J K. Hole concealment for depth image using pixel classification in multiview system[C]//Proceedings of 2021 IEEE International Conference on Consumer Electronics. Las Vegas, USA: IEEE Computer Society, 2021: 1-5.
17	ZHANG X, ZHAO Y, LIN C, et al. Warping-driven mode selection for depth error concealment[C]//Proceedings of 2014 IEEE Global Conference on Signal and Information Processing. Atlanta, USA: IEEE, 2014: 302-306.
18	丁颖,刘延伟,等. 虚拟现实全景图像显著性检测研究进展综述[J]. 电子学报, 2019, 47(7): 1575-1583.
	DING Ying, LIU Yan-wei, et al. An overview of research progress on saliency detection of panoramicVR images[J]. Acta Electronica Sinica, 2019, 47(7): 1575-1583. (in Chinese)
19	CONG R, LEI J, FU H, et al. Going from RGB to RGBD saliency: a depth-guided transformation model[J]. IEEE Transactions on Cybernetics, 2020, 50(8): 3627-3639.
20	CHENG M, MITRA N J, HUANG X, et al. Global contrast based salient region detection[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2011, 37(3): 409-416.
21	WANG J, DASILVA M P, LECALLET P, et al. Computational model of stereoscopic 3D visual saliency[J]. IEEE Transactions on Image Processing, 2013, 22(6): 2151-2165.
22	JEON B, HUU T, CANH T N. A frame loss concealment for spatial scalable HEVC using base layer motion[C]//Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. Jeju, Korea: IEEE, 2019: 1-4.
23	SONG K, CHUNG T, OH Y, et al. Error concealment of multi-view video sequences using inter-view and intra-view correlations[J]. Journal of Visual Communication and Image Representation, 2009, 20(4): 281-292.
24	LIU X, PENG Q, FAN X. Frame loss concealment for multi-view video plus depth[C]//Proceedings of the 15th IEEE International Symposium on Consumer Electronics. Singapore: IEEE, 2011: 208-211.

QP	序列	PSNR(dB)
QP	序列	无错	Jeon	Song	Liu	Lin	本文
25	Lovebird1	39.67	35.03	34.58	35.21	34.83	35.49
	PoznanS	38.95	31.98	30.61	32.47	31.57	34.24
	BookArri	39.89	30.06	28.54	30.71	30.07	32.67
	Newspap	40.48	34.35	33.91	34.67	34.32	34.95
	Kendo	44.59	32.95	31.27	33.64	32.68	35.13
	Balloons	42.52	36.13	35.76	36.21	35.99	36.42
28	Lovebird1	38.56	34.41	34.06	34.56	34.24	34.79
	PoznanS	38.17	31.56	30.22	32.01	31.12	33.74
	BookArri	39.14	29.65	28.18	30.32	29.68	32.23
	Newspap	39.58	33.87	33.50	34.16	33.85	34.39
	Kendo	43.76	32.51	30.86	33.17	32.25	34.62
	Balloons	41.50	35.61	35.33	35.66	35.49	35.79
30	Lovebird1	37.69	33.90	33.63	34.02	33.71	34.19
	PoznanS	37.46	31.19	29.91	31.63	30.74	33.28
	BookArri	38.52	29.28	27.86	29.96	29.34	31.83
	Newspap	38.74	33.45	33.19	33.72	33.47	33.93
	Kendo	43.02	32.10	30.54	32.76	31.86	34.16
	Balloons	40.63	35.12	35.01	35.16	35.04	35.22

QP	序列	PSNR(dB)
QP	序列	无错	Jeon	Song	Liu	Lin	本文
25	Lovebird1	39.67	35.03	34.58	35.21	34.83	35.49
	PoznanS	38.95	31.98	30.61	32.47	31.57	34.24
	BookArri	39.89	30.06	28.54	30.71	30.07	32.67
	Newspap	40.48	34.35	33.91	34.67	34.32	34.95
	Kendo	44.59	32.95	31.27	33.64	32.68	35.13
	Balloons	42.52	36.13	35.76	36.21	35.99	36.42
28	Lovebird1	38.56	34.41	34.06	34.56	34.24	34.79
	PoznanS	38.17	31.56	30.22	32.01	31.12	33.74
	BookArri	39.14	29.65	28.18	30.32	29.68	32.23
	Newspap	39.58	33.87	33.50	34.16	33.85	34.39
	Kendo	43.76	32.51	30.86	33.17	32.25	34.62
	Balloons	41.50	35.61	35.33	35.66	35.49	35.79
30	Lovebird1	37.69	33.90	33.63	34.02	33.71	34.19
	PoznanS	37.46	31.19	29.91	31.63	30.74	33.28
	BookArri	38.52	29.28	27.86	29.96	29.34	31.83
	Newspap	38.74	33.45	33.19	33.72	33.47	33.93
	Kendo	43.02	32.10	30.54	32.76	31.86	34.16
	Balloons	40.63	35.12	35.01	35.16	35.04	35.22

QP	序列	SSIM
QP	序列	无错	Jeon	Song	Liu	Lin	本文
25	Lovebird1	0.969	0.955	0.954	0.955	0.954	0.956
	PoznanS	0.944	0.904	0.897	0.907	0.902	0.918
	BookArri	0.954	0.893	0.883	0.897	0.893	0.909
	Newspap	0.967	0.946	0.944	0.947	0.946	0.948
	Kendo	0.968	0.911	0.903	0.915	0.910	0.922
	Balloons	0.974	0.957	0.956	0.957	0.957	0.958
28	Lovebird1	0.965	0.953	0.952	0.954	0.953	0.954
	PoznanStr	0.940	0.902	0.895	0.905	0.899	0.915
	BookArri	0.949	0.890	0.881	0.894	0.891	0.906
	Newspap	0.963	0.944	0.943	0.945	0.944	0.946
	Kendo	0.964	0.909	0.901	0.912	0.908	0.919
	Balloons	0.971	0.956	0.955	0.956	0.955	0.956
30	Lovebird1	0.963	0.952	0.950	0.952	0.951	0.953
	PoznanStr	0.938	0.900	0.893	0.902	0.898	0.912
	BookArri	0.945	0.888	0.879	0.892	0.889	0.904
	Newspap	0.960	0.943	0.942	0.944	0.943	0.945
	Kendo	0.961	0.908	0.899	0.911	0.907	0.917
	Balloons	0.968	0.955	0.954	0.955	0.954	0.955

QP	序列	SSIM
QP	序列	无错	Jeon	Song	Liu	Lin	本文
25	Lovebird1	0.969	0.955	0.954	0.955	0.954	0.956
	PoznanS	0.944	0.904	0.897	0.907	0.902	0.918
	BookArri	0.954	0.893	0.883	0.897	0.893	0.909
	Newspap	0.967	0.946	0.944	0.947	0.946	0.948
	Kendo	0.968	0.911	0.903	0.915	0.910	0.922
	Balloons	0.974	0.957	0.956	0.957	0.957	0.958
28	Lovebird1	0.965	0.953	0.952	0.954	0.953	0.954
	PoznanStr	0.940	0.902	0.895	0.905	0.899	0.915
	BookArri	0.949	0.890	0.881	0.894	0.891	0.906
	Newspap	0.963	0.944	0.943	0.945	0.944	0.946
	Kendo	0.964	0.909	0.901	0.912	0.908	0.919
	Balloons	0.971	0.956	0.955	0.956	0.955	0.956
30	Lovebird1	0.963	0.952	0.950	0.952	0.951	0.953
	PoznanStr	0.938	0.900	0.893	0.902	0.898	0.912
	BookArri	0.945	0.888	0.879	0.892	0.889	0.904
	Newspap	0.960	0.943	0.942	0.944	0.943	0.945
	Kendo	0.961	0.908	0.899	0.911	0.907	0.917
	Balloons	0.968	0.955	0.954	0.955	0.954	0.955

序列	Jeon	Song	Liu	Lin	本文
Lovebird1	5.403	5.271	5.731	5.815	5.384
PoznanStreet	6.141	5.978	7.632	7.751	6.041
BookArrival	5.612	5.403	5.738	5.887	5.628
Newspaper	5.584	5.398	5.742	5.845	5.559
Kendo	5.286	5.139	5.756	5.922	5.304
Balloons	5.247	5.054	5.703	5.764	5.226

结合显著性的MVD视频整帧丢失错误隐藏

Error Concealment for Frame Losses in MVD Video Using Saliency

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 24

相关文章 15

编辑推荐

Metrics

[1]	魏志超, 杨春玲. 时域注意力特征对齐的视频压缩感知重构网络[J]. 电子学报, 2022, 50(11): 2584-2592.
[2]	罗会兰, 袁璞, 童康. 基于深度学习的显著性目标检测方法综述[J]. 电子学报, 2021, 49(7): 1417-1427.
[3]	钱晓亮, 白臻, 陈渊, 张鼎文, 史坤峰, 王芳, 吴青娥, 毋媛媛, 王慰. 协同视觉显著性检测方法综述[J]. 电子学报, 2019, 47(6): 1352-1365.
[4]	李鹏, 王延江. 一种基于动态显著性的人眼注视点移动计算模型[J]. 电子学报, 2019, 47(12): 2582-2589.
[5]	丁鹏, 张叶, 贾平, 常旭岭, 刘让. 基于视觉显著性的海面舰船检测技术[J]. 电子学报, 2018, 46(1): 127-134.
[6]	于红绯, 刘威, 袁淮, 赵宏. 基于子块运动补偿的运动目标检测[J]. 电子学报, 2017, 45(1): 173-180.
[7]	姜维, 卢朝阳, 李静, 刘晓佩, 姚超. 基于视觉显著性与文字置信图的场景文字的背景抑制方法[J]. 电子学报, 2015, 43(1): 62-68.
[8]	贾高伟, 常文革. FMCW SAR运动补偿处理技术研究[J]. 电子学报, 2013, 41(9): 1665-1671.
[9]	罗海坤, 王永庆, 申宇瑶, 吴嗣亮. 大动态多普勒频率下微弱直扩信号的长时间积累算法[J]. 电子学报, 2013, 41(12): 2397-2402.
[10]	俞翔, 朱岱寅. 一种ISAR二维压缩感知成像的运动补偿方法[J]. 电子学报, 2012, 40(9): 1783-1789.
[11]	彭岁阳;张军;沈振康. 基于斜飞模式的双基地SAR成像算法[J]. 电子学报, 2011, 39(9): 1967-1974.
[12]	胡杰民;付耀文;黎湘;景宁. 基于波形预先设计和相位对消技术的步进频信号ISAR成像研究[J]. 电子学报, 2011, 39(7): 1575-1580.
[13]	李杨寰;宋千;王鹏宇;金添;周智敏. 多级多分辨快速后向投影成像算法[J]. 电子学报, 2011, 39(3): 518-524.
[14]	安道祥;黄晓涛;周智敏. 一种小型机载低频UWBSAR的三级运动补偿方法[J]. 电子学报, 2011, 39(12): 2776-2783.
[15]	张孝临;赵宏伟;王慧;戴金波. 基于对比敏感度和马尔可夫链的注意信息提取算法[J]. 电子学报, 2010, 38(2A): 213-217.