电子学报 ›› 2022, Vol. 50 ›› Issue (9): 2146-2154.DOI: 10.12263/DZXB.20210393
收稿日期:
2021-03-25
修回日期:
2022-02-08
出版日期:
2022-09-25
通讯作者:
作者简介:
基金资助:
CUI Jin-peng, ZHOU Yang, YIN Hai-bing(), HUANG Xiao-feng(
), LU Yu(
)
Received:
2021-03-25
Revised:
2022-02-08
Online:
2022-09-25
Published:
2022-10-26
Corresponding author:
摘要:
基于多视点视频加深度(Multiview Video plus Depth, MVD)格式的三维视频能给用户提供多视角的沉浸式视觉体验.针对MVD视频流传输中彩色帧发生整帧丢失的情况,提出了一种结合视觉显著性的三维视频整帧丢失错误隐藏算法.先将丢失帧各区域划分为三种不同的显著等级,然后对低、中、高显著区域,分别采用时域直接拷贝、时域和视点间像素自适应搜索法和新型的运动矢量补偿法进行恢复.相较于目前的对比算法,经提出算法重建后的丢失帧的峰值信噪比值(Peak Signal to Noise Ratio, PSNR)和结构相似度值(Structural Similarity Index Measure, SSIM)能分别提高0.99~2.61 dB和0.005~0.012,并且重建图像主观视觉质量更佳.
中图分类号:
崔金鹏, 周洋, 殷海兵, 黄晓峰, 陆宇. 结合显著性的MVD视频整帧丢失错误隐藏[J]. 电子学报, 2022, 50(9): 2146-2154.
CUI Jin-peng, ZHOU Yang, YIN Hai-bing, HUANG Xiao-feng, LU Yu. Error Concealment for Frame Losses in MVD Video Using Saliency[J]. Acta Electronica Sinica, 2022, 50(9): 2146-2154.
QP | 序列 | PSNR(dB) | |||||
---|---|---|---|---|---|---|---|
无错 | Jeon | Song | Liu | Lin | 本文 | ||
25 | Lovebird1 | 39.67 | 35.03 | 34.58 | 35.21 | 34.83 | 35.49 |
PoznanS | 38.95 | 31.98 | 30.61 | 32.47 | 31.57 | 34.24 | |
BookArri | 39.89 | 30.06 | 28.54 | 30.71 | 30.07 | 32.67 | |
Newspap | 40.48 | 34.35 | 33.91 | 34.67 | 34.32 | 34.95 | |
Kendo | 44.59 | 32.95 | 31.27 | 33.64 | 32.68 | 35.13 | |
Balloons | 42.52 | 36.13 | 35.76 | 36.21 | 35.99 | 36.42 | |
28 | Lovebird1 | 38.56 | 34.41 | 34.06 | 34.56 | 34.24 | 34.79 |
PoznanS | 38.17 | 31.56 | 30.22 | 32.01 | 31.12 | 33.74 | |
BookArri | 39.14 | 29.65 | 28.18 | 30.32 | 29.68 | 32.23 | |
Newspap | 39.58 | 33.87 | 33.50 | 34.16 | 33.85 | 34.39 | |
Kendo | 43.76 | 32.51 | 30.86 | 33.17 | 32.25 | 34.62 | |
Balloons | 41.50 | 35.61 | 35.33 | 35.66 | 35.49 | 35.79 | |
30 | Lovebird1 | 37.69 | 33.90 | 33.63 | 34.02 | 33.71 | 34.19 |
PoznanS | 37.46 | 31.19 | 29.91 | 31.63 | 30.74 | 33.28 | |
BookArri | 38.52 | 29.28 | 27.86 | 29.96 | 29.34 | 31.83 | |
Newspap | 38.74 | 33.45 | 33.19 | 33.72 | 33.47 | 33.93 | |
Kendo | 43.02 | 32.10 | 30.54 | 32.76 | 31.86 | 34.16 | |
Balloons | 40.63 | 35.12 | 35.01 | 35.16 | 35.04 | 35.22 |
表1 经不同错误隐藏方法后的PSNR值
QP | 序列 | PSNR(dB) | |||||
---|---|---|---|---|---|---|---|
无错 | Jeon | Song | Liu | Lin | 本文 | ||
25 | Lovebird1 | 39.67 | 35.03 | 34.58 | 35.21 | 34.83 | 35.49 |
PoznanS | 38.95 | 31.98 | 30.61 | 32.47 | 31.57 | 34.24 | |
BookArri | 39.89 | 30.06 | 28.54 | 30.71 | 30.07 | 32.67 | |
Newspap | 40.48 | 34.35 | 33.91 | 34.67 | 34.32 | 34.95 | |
Kendo | 44.59 | 32.95 | 31.27 | 33.64 | 32.68 | 35.13 | |
Balloons | 42.52 | 36.13 | 35.76 | 36.21 | 35.99 | 36.42 | |
28 | Lovebird1 | 38.56 | 34.41 | 34.06 | 34.56 | 34.24 | 34.79 |
PoznanS | 38.17 | 31.56 | 30.22 | 32.01 | 31.12 | 33.74 | |
BookArri | 39.14 | 29.65 | 28.18 | 30.32 | 29.68 | 32.23 | |
Newspap | 39.58 | 33.87 | 33.50 | 34.16 | 33.85 | 34.39 | |
Kendo | 43.76 | 32.51 | 30.86 | 33.17 | 32.25 | 34.62 | |
Balloons | 41.50 | 35.61 | 35.33 | 35.66 | 35.49 | 35.79 | |
30 | Lovebird1 | 37.69 | 33.90 | 33.63 | 34.02 | 33.71 | 34.19 |
PoznanS | 37.46 | 31.19 | 29.91 | 31.63 | 30.74 | 33.28 | |
BookArri | 38.52 | 29.28 | 27.86 | 29.96 | 29.34 | 31.83 | |
Newspap | 38.74 | 33.45 | 33.19 | 33.72 | 33.47 | 33.93 | |
Kendo | 43.02 | 32.10 | 30.54 | 32.76 | 31.86 | 34.16 | |
Balloons | 40.63 | 35.12 | 35.01 | 35.16 | 35.04 | 35.22 |
QP | 序列 | SSIM | |||||
---|---|---|---|---|---|---|---|
无错 | Jeon | Song | Liu | Lin | 本文 | ||
25 | Lovebird1 | 0.969 | 0.955 | 0.954 | 0.955 | 0.954 | 0.956 |
PoznanS | 0.944 | 0.904 | 0.897 | 0.907 | 0.902 | 0.918 | |
BookArri | 0.954 | 0.893 | 0.883 | 0.897 | 0.893 | 0.909 | |
Newspap | 0.967 | 0.946 | 0.944 | 0.947 | 0.946 | 0.948 | |
Kendo | 0.968 | 0.911 | 0.903 | 0.915 | 0.910 | 0.922 | |
Balloons | 0.974 | 0.957 | 0.956 | 0.957 | 0.957 | 0.958 | |
28 | Lovebird1 | 0.965 | 0.953 | 0.952 | 0.954 | 0.953 | 0.954 |
PoznanStr | 0.940 | 0.902 | 0.895 | 0.905 | 0.899 | 0.915 | |
BookArri | 0.949 | 0.890 | 0.881 | 0.894 | 0.891 | 0.906 | |
Newspap | 0.963 | 0.944 | 0.943 | 0.945 | 0.944 | 0.946 | |
Kendo | 0.964 | 0.909 | 0.901 | 0.912 | 0.908 | 0.919 | |
Balloons | 0.971 | 0.956 | 0.955 | 0.956 | 0.955 | 0.956 | |
30 | Lovebird1 | 0.963 | 0.952 | 0.950 | 0.952 | 0.951 | 0.953 |
PoznanStr | 0.938 | 0.900 | 0.893 | 0.902 | 0.898 | 0.912 | |
BookArri | 0.945 | 0.888 | 0.879 | 0.892 | 0.889 | 0.904 | |
Newspap | 0.960 | 0.943 | 0.942 | 0.944 | 0.943 | 0.945 | |
Kendo | 0.961 | 0.908 | 0.899 | 0.911 | 0.907 | 0.917 | |
Balloons | 0.968 | 0.955 | 0.954 | 0.955 | 0.954 | 0.955 |
表2 经不同错误隐藏方法后的SSIM值
QP | 序列 | SSIM | |||||
---|---|---|---|---|---|---|---|
无错 | Jeon | Song | Liu | Lin | 本文 | ||
25 | Lovebird1 | 0.969 | 0.955 | 0.954 | 0.955 | 0.954 | 0.956 |
PoznanS | 0.944 | 0.904 | 0.897 | 0.907 | 0.902 | 0.918 | |
BookArri | 0.954 | 0.893 | 0.883 | 0.897 | 0.893 | 0.909 | |
Newspap | 0.967 | 0.946 | 0.944 | 0.947 | 0.946 | 0.948 | |
Kendo | 0.968 | 0.911 | 0.903 | 0.915 | 0.910 | 0.922 | |
Balloons | 0.974 | 0.957 | 0.956 | 0.957 | 0.957 | 0.958 | |
28 | Lovebird1 | 0.965 | 0.953 | 0.952 | 0.954 | 0.953 | 0.954 |
PoznanStr | 0.940 | 0.902 | 0.895 | 0.905 | 0.899 | 0.915 | |
BookArri | 0.949 | 0.890 | 0.881 | 0.894 | 0.891 | 0.906 | |
Newspap | 0.963 | 0.944 | 0.943 | 0.945 | 0.944 | 0.946 | |
Kendo | 0.964 | 0.909 | 0.901 | 0.912 | 0.908 | 0.919 | |
Balloons | 0.971 | 0.956 | 0.955 | 0.956 | 0.955 | 0.956 | |
30 | Lovebird1 | 0.963 | 0.952 | 0.950 | 0.952 | 0.951 | 0.953 |
PoznanStr | 0.938 | 0.900 | 0.893 | 0.902 | 0.898 | 0.912 | |
BookArri | 0.945 | 0.888 | 0.879 | 0.892 | 0.889 | 0.904 | |
Newspap | 0.960 | 0.943 | 0.942 | 0.944 | 0.943 | 0.945 | |
Kendo | 0.961 | 0.908 | 0.899 | 0.911 | 0.907 | 0.917 | |
Balloons | 0.968 | 0.955 | 0.954 | 0.955 | 0.954 | 0.955 |
序列 | Jeon | Song | Liu | Lin | 本文 |
---|---|---|---|---|---|
Lovebird1 | 5.403 | 5.271 | 5.731 | 5.815 | 5.384 |
PoznanStreet | 6.141 | 5.978 | 7.632 | 7.751 | 6.041 |
BookArrival | 5.612 | 5.403 | 5.738 | 5.887 | 5.628 |
Newspaper | 5.584 | 5.398 | 5.742 | 5.845 | 5.559 |
Kendo | 5.286 | 5.139 | 5.756 | 5.922 | 5.304 |
Balloons | 5.247 | 5.054 | 5.703 | 5.764 | 5.226 |
表3 各算法重建单帧平均所需的解码时间 s
序列 | Jeon | Song | Liu | Lin | 本文 |
---|---|---|---|---|---|
Lovebird1 | 5.403 | 5.271 | 5.731 | 5.815 | 5.384 |
PoznanStreet | 6.141 | 5.978 | 7.632 | 7.751 | 6.041 |
BookArrival | 5.612 | 5.403 | 5.738 | 5.887 | 5.628 |
Newspaper | 5.584 | 5.398 | 5.742 | 5.845 | 5.559 |
Kendo | 5.286 | 5.139 | 5.756 | 5.922 | 5.304 |
Balloons | 5.247 | 5.054 | 5.703 | 5.764 | 5.226 |
1 | WANG T, LI F, QIAO X, et al. Low-complexity error resilient HEVC video coding: a deep learning approach[J]. IEEE Transactions on Image Processing, 2021, 30(2): 1245-1260. |
2 | XIANG X, CHENG Y. A frame loss error concealment scheme for multi-view video coding[C]//Proceedings of 2014 International Conference on Instrumentation, Measurement, Computer, Communication and Control. Harbin, China: IEEE Computer Society, 2014: 857-861. |
3 | KUAN Y K, LI G L, CHEN M J, et al. Error concealment algorithm using inter-view correlation for multi-view video[J]. EURASIP Journal on Image and Video Processing, 2014, 38(1): 1-11. |
4 | ZHOU Y, XIANG W, WANG G. Frame loss concealment for multiview video transmission over wireless multimedia sensor networks[J]. IEEE Sensors Journal, 2015, 15(3): 1892-1901. |
5 | YU H, RAN M, LI P, et al. A new hybrid error concealment for multi-view video[C]//Proceedings of 2015 IEEE China Summit and International Conference on Signal and Information Processing. Chengdu, China: IEEE, 2015: 176-180. |
6 | AHAMED Z G, SHARMA V K, JAIN A. Image error concealment using robust data hiding technique[C]//Proceedings of 2020 International Conference on Innovative Mechanisms for Industry Applications. Bangalore, India: IEEE, 2020: 470-474. |
7 | EL-SHAFAI W, EL-RABAIE S, EL-HALAWANY M M, et al. Recursive Bayesian filtering-based error concealment scheme for 3D video communication over severely lossy wireless channels[J]. Circuits, Systems, and Signal Processing, 2018, 37(11): 4810-4841. |
8 | YANG M, LAN X, ZHENG N, et al. Depth-assisted temporal error concealment for Intra frame slices in 3-D video[J]. IEEE Transactions on Broadcasting, 2014, 60(2): 385-393. |
9 | EL-SHAFAI W, EL-RABAIE S, EL-HALAWANY M, et al. Encoder-independent decoder-dependent depth-assisted error concealment algorithm for wireless 3D video communication[J]. Multimedia Tools and Applications, 2018, 77(11): 13145-13172. |
10 | HUANG P C, LIN J R, LI G L, et al. Improved depth- asisted error concealment algorithm for 3D video transmission[J]. IEEE Transactions on Multimedia, 2017, 19(11): 2625-2632. |
11 | TAI S C, WANG C C, HONG C S, et al. An effiicient full frame algorithm for object-based error concealment in 3D depth-based video[J]. Multimedia Tools and Applications, 2016, 75(16): 9927- 9947. |
12 | KHATTAK S, MAUGEY T, HAMZAOUI R, et al. Temporal and inter-view consistent error concealment technique for multiview plus depth video[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(5): 829-840. |
13 | LIN T L, WANG C J, DING T L, et al. Recovery of lost color and depth frames in multiview videos[J]. IEEE Transactions on Image Processing, 2018, 27(11): 5449-5463. |
14 | LEE P J, KUO K T. An adaptive error concealment method for depth map in 3D video coding[J]. International Journal of Fuzzy Systems, 2016, 18(1): 62-71. |
15 | ASSUNCAO P A, MARCELINO S, SOARES S, et al. Spatial error concealment for intra-coded depth maps in multiview video-plus-depth[J]. Multimedia Tools and Applications, 2017, 76(12): 13835-13858. |
16 | LEE G W, HAN J K. Hole concealment for depth image using pixel classification in multiview system[C]//Proceedings of 2021 IEEE International Conference on Consumer Electronics. Las Vegas, USA: IEEE Computer Society, 2021: 1-5. |
17 | ZHANG X, ZHAO Y, LIN C, et al. Warping-driven mode selection for depth error concealment[C]//Proceedings of 2014 IEEE Global Conference on Signal and Information Processing. Atlanta, USA: IEEE, 2014: 302-306. |
18 | 丁颖,刘延伟,等. 虚拟现实全景图像显著性检测研究进展综述[J]. 电子学报, 2019, 47(7): 1575-1583. |
DING Ying, LIU Yan-wei, et al. An overview of research progress on saliency detection of panoramicVR images[J]. Acta Electronica Sinica, 2019, 47(7): 1575-1583. (in Chinese) | |
19 | CONG R, LEI J, FU H, et al. Going from RGB to RGBD saliency: a depth-guided transformation model[J]. IEEE Transactions on Cybernetics, 2020, 50(8): 3627-3639. |
20 | CHENG M, MITRA N J, HUANG X, et al. Global contrast based salient region detection[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2011, 37(3): 409-416. |
21 | WANG J, DASILVA M P, LECALLET P, et al. Computational model of stereoscopic 3D visual saliency[J]. IEEE Transactions on Image Processing, 2013, 22(6): 2151-2165. |
22 | JEON B, HUU T, CANH T N. A frame loss concealment for spatial scalable HEVC using base layer motion[C]//Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. Jeju, Korea: IEEE, 2019: 1-4. |
23 | SONG K, CHUNG T, OH Y, et al. Error concealment of multi-view video sequences using inter-view and intra-view correlations[J]. Journal of Visual Communication and Image Representation, 2009, 20(4): 281-292. |
24 | LIU X, PENG Q, FAN X. Frame loss concealment for multi-view video plus depth[C]//Proceedings of the 15th IEEE International Symposium on Consumer Electronics. Singapore: IEEE, 2011: 208-211. |
[1] | 魏志超, 杨春玲. 时域注意力特征对齐的视频压缩感知重构网络[J]. 电子学报, 2022, 50(11): 2584-2592. |
[2] | 罗会兰, 袁璞, 童康. 基于深度学习的显著性目标检测方法综述[J]. 电子学报, 2021, 49(7): 1417-1427. |
[3] | 钱晓亮, 白臻, 陈渊, 张鼎文, 史坤峰, 王芳, 吴青娥, 毋媛媛, 王慰. 协同视觉显著性检测方法综述[J]. 电子学报, 2019, 47(6): 1352-1365. |
[4] | 李鹏, 王延江. 一种基于动态显著性的人眼注视点移动计算模型[J]. 电子学报, 2019, 47(12): 2582-2589. |
[5] | 丁鹏, 张叶, 贾平, 常旭岭, 刘让. 基于视觉显著性的海面舰船检测技术[J]. 电子学报, 2018, 46(1): 127-134. |
[6] | 于红绯, 刘威, 袁淮, 赵宏. 基于子块运动补偿的运动目标检测[J]. 电子学报, 2017, 45(1): 173-180. |
[7] | 姜维, 卢朝阳, 李静, 刘晓佩, 姚超. 基于视觉显著性与文字置信图的场景文字的背景抑制方法[J]. 电子学报, 2015, 43(1): 62-68. |
[8] | 贾高伟, 常文革. FMCW SAR运动补偿处理技术研究[J]. 电子学报, 2013, 41(9): 1665-1671. |
[9] | 罗海坤, 王永庆, 申宇瑶, 吴嗣亮. 大动态多普勒频率下微弱直扩信号的长时间积累算法[J]. 电子学报, 2013, 41(12): 2397-2402. |
[10] | 俞翔, 朱岱寅. 一种ISAR二维压缩感知成像的运动补偿方法[J]. 电子学报, 2012, 40(9): 1783-1789. |
[11] | 彭岁阳;张军;沈振康. 基于斜飞模式的双基地SAR成像算法[J]. 电子学报, 2011, 39(9): 1967-1974. |
[12] | 胡杰民;付耀文;黎湘;景宁. 基于波形预先设计和相位对消技术的步进频信号ISAR成像研究[J]. 电子学报, 2011, 39(7): 1575-1580. |
[13] | 李杨寰;宋千;王鹏宇;金添;周智敏. 多级多分辨快速后向投影成像算法[J]. 电子学报, 2011, 39(3): 518-524. |
[14] | 安道祥;黄晓涛;周智敏. 一种小型机载低频UWBSAR的三级运动补偿方法[J]. 电子学报, 2011, 39(12): 2776-2783. |
[15] | 张孝临;赵宏伟;王 慧;戴金波. 基于对比敏感度和马尔可夫链的注意信息提取算法[J]. 电子学报, 2010, 38(2A): 213-217. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||