Quality Assessment for Windowed-6DoF Synthesized Video Based on Multilayer Features Fusion

TANG Ting-yan; ZOU Wen-hui; PENG Zong-ju; CHEN Fen; JIN Chong-chong

doi:10.12263/DZXB.20240101

您当前的位置：

首页 >

文章列表页 >

Quality Assessment for Windowed-6DoF Synthesized Video Based on Multilayer Features Fusion

PAPERS | 更新时间：2025-12-08

- Quality Assessment for Windowed-6DoF Synthesized Video Based on Multilayer Features Fusion
- ACTA ELECTRONICA SINICA Vol. 53, Issue 1, Pages: 193-208(2025)
- 作者机构：
  
  1.重庆理工大学电气与电子工程学院，重庆 400054
  2.宁波大学信息科学与工程学院，浙江宁波 315211
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62371081);National Natural Science Foundation of Chongqing(cstc2021jcyj-msxmX0411;CSTB2022NSCQ-MSX0873)
- DOI：10.12263/DZXB.20240101
  CLC： TP391;
- Received：26 January 2024，
  
  Revised：2024-08-08，
  
  Published：25 January 2025
- 稿件说明：
移动端阅览
唐婷琰, 邹文辉, 彭宗举, 等. 融合多层特征的窗口6DoF合成视频质量评价[J]. 电子学报, 2025, 53(01): 193-208.

TANG Ting-yan, ZOU Wen-hui, PENG Zong-ju, et al. Quality Assessment for Windowed-6DoF Synthesized Video Based on Multilayer Features Fusion[J]. Acta Electronica Sinica, 2025, 53(01): 193-208.
唐婷琰, 邹文辉, 彭宗举, 等. 融合多层特征的窗口6DoF合成视频质量评价[J]. 电子学报, 2025, 53(01): 193-208. DOI：10.12263/DZXB.20240101

TANG Ting-yan, ZOU Wen-hui, PENG Zong-ju, et al. Quality Assessment for Windowed-6DoF Synthesized Video Based on Multilayer Features Fusion[J]. Acta Electronica Sinica, 2025, 53(01): 193-208. DOI：10.12263/DZXB.20240101

摘要

六自由度（Six Degrees of Freedom， 6DoF）视频允许用户从全方位、任意视角身临其境体验场景，是下一代沉浸式视频产业的发展方向.部分自由度受限的窗口6DoF视频近年来成为研究热点，本文提出面向窗口6DoF合成视频的主观数据库和客观质量评价方法.在主观数据库方面，构建了包含两种交互路径不适性失真、四种绘制失真和四种压缩失真的窗口6DoF合成视频主观质量数据库Windowed-6DoF，并开展主观质量测试及结果分析.在客观质量评价方法方面，设计了一种融合多层特征的窗口6DoF合成视频无参考客观质量评价方法.采用切比雪夫矩提取视频时域切片上的底层形状特征；采用Resnet-50网络提取视频的时域、空域高层语义特征并进行降维处理；最后采用随机森林将底层形状特征和高层语义特征进行融合，且训练得到窗口6DoF合成视频的客观质量评价模型.在提出的数据库Windowed-6DoF和公共数据库IRCCyN/IVC DIBR的测试结果表明，本文提出的客观质量评价方法预测分数的皮尔逊线性相关系数分别达到0.932 7和0.858 1，与主观评价分数具有较好的一致性.

Abstract

Six degrees of freedom (6DoF) video

allowing users to experience the scene from omnidirectional and arbitrary perspective

is the development direction of the next-generation immersive video system. The windowed 6DoF video with limited degrees of freedom is a hot research topic in recent years. This paper proposes a subjective database and an objective quality assessment method for the windowed 6DoF synthesized video. For subjective database

we build a subjective quality database called Windowed-6DoF. The database contains 128 windowed 6DoF synthesized videos which involve discomfort caused by two viewpoint switching paths

distortions caused by four rendering schemes

and four levels of compression. Then subjective quality tests are conducted on the database and the test results are analyzed. For objective quality assessment

we design a no reference quality assessment method for windowed 6DoF synthesized video which fuses multilayer features. Tchebichef moment is used to extract the low layer shape features of temporal video slices. Resnet-50 network is used to extract the high-level semantic features of video in temporal and spatial domains

and consequently reduce the dimensionality of features. Finally

the random forest is used to fuse the low layer shape features and high layer semantic features

and train the quality assessment model of windowed 6DoF synthesized video. We respectively test the method on the proposed Windowed-6DoF database and IRCCyN/IVC DIBR database. The experimental results show that the Pearson linear correlation coefficient of the proposed method are 0.932 7 and 0.858 1

respectively. The predicted scores of the objective method are consistent with the subjective assessment scores.

关键词

Keywords

references

LIN T C , AOUIDIDI A , CHEN Z T , et al . VIRD: Immersive match video analysis for high-performance badminton coaching [J ] . IEEE Transactions on Visualization and Computer Graphics , 2024 , 30 ( 1 ): 458 - 468 .

丁颖 , 刘延伟 , 刘金霞 , 等 . 虚拟现实全景图像显著性检测研究进展综述 [J ] . 电子学报 , 2019 , 47 ( 7 ): 1575 - 1583 .

DING Y , LIU Y W , LIU J X , et al . An overview of research progress on saliency detection of panoramic VR images [J ] . Acta Electronica Sinica , 2019 , 47 ( 7 ): 1575 - 1583 . (in Chinese)

王旭 , 刘琼 , 彭宗举 , 等 . 6DoF视频技术研究进展 [J ] . 中国图象图形学报 , 2023 , 28 ( 6 ): 1863 - 1890 .

WANG X , LIU Q , PENG Z J , et al . Research progress of six degree of freedom (6DoF) video technology [J ] . Journal of Image and Graphics , 2023 , 28 ( 6 ): 1863 - 1890 . (in Chinese)

JUNG J , KROON B , DORÉ R , et al . Common Test Conditions on 3DoF+ and Windowed 6DoF [R ] . San Diego : MPEG , 2018 .

WEN W , LI M , ZHANG Y , et al . Modular blind video quality assessment [C ] // Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2024 : 2763 - 2772 .

BI X D , HE X H , XIONG S H , et al . Blind video quality assessment based on spatio-temporal feature resolver [J ] . Neurocomputing , 2024 , 574 : 127249 .

YUAN K , LIU H B , LI M D , et al . PTM-VQA: Efficient video quality assessment leveraging diverse PreTrained models from the wild [C ] // 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2024 : 2835 - 2845 .

WU H N , CHEN C F , HOU J W , et al . FAST-VQA: Efficient end-to-end video quality assessment with fragment sampling [C ] //[M ] // Computer Vision - ECCV 2022 . Cham : Springer , 2022 : 538 - 554 .

WU H N , CHEN C F , LIAO L , et al . Neighbourhood representative sampling for efficient end-to-end video quality assessment [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023 , 45 ( 12 ): 15185 - 15202 .

WU H N , ZHANG E L , LIAO L , et al . Exploring video quality assessment on user generated contents from aesthetic and technical perspectives [C ] // 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2023 : 20087 - 20097 .

BOSC E , PEPION R , LE CALLET P , et al . Towards a new quality metric for 3-D synthesized view assessment [J ] . IEEE Journal of Selected Topics in Signal Processing , 2011 , 5 ( 7 ): 1332 - 1343 .

SONG R , KO H , JAY KUO C C . MCL-3D: A database for stereoscopic image quality assessment using 2D-image-plus-depth source [J ] . Journal of Information Science and Engineering , 2015 , 31 ( 5 ): 1593 - 1611 .

JUNG Y J , KIM H G , RO Y M . Critical binocular asymmetry measure for the perceptual quality assessment of synthesized stereo 3D images in view synthesis [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2016 , 26 ( 7 ): 1201 - 1214 .

TIAN S S , ZHANG L , MORIN L , et al . A benchmark of DIBR synthesized view quality assessment metrics on a new database for immersive media applications [J ] . IEEE Transactions on Multimedia , 2019 , 21 ( 5 ): 1235 - 1247 .

BOSC E , HANHART P , LE CALLET P , et al . A quality assessment protocol for free-viewpoint video sequences synthesized from decompressed depth data [C ] // 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX) . Piscataway : IEEE , 2013 : 100 - 105 .

LIU X K , ZHANG Y , HU S D , et al . Subjective and objective video quality assessment of 3D synthesized views with texture/depth compression distortion [J ] . IEEE Transactions on Image Processing , 2015 , 24 ( 12 ): 4847 - 4861 .

WANG X C , WANG K , YANG B L , et al . Perceptual quality assessment on DIBR synthesized videos with composite distortions [C ] // 2020 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2020 : 186 - 190 .

PENG Z J , WANG S P , CHEN F , et al . Quality assessment of stereoscopic video in free viewpoint video system [J ] . Journal of Visual Communication and Image Representation , 2019 , 63 : 102569 .

高敏娟 , 党宏社 , 魏立力 , 等 . 全参考图像质量评价回顾与展望 [J ] . 电子学报 , 2021 , 49 ( 11 ): 2261 - 2272 .

GAO M J , DANG H S , WEI L L , et al . Review and prospect of full reference image quality assessment [J ] . Acta Electronica Sinica , 2021 , 49 ( 11 ): 2261 - 2272 . (in Chinese)

Sadbhawna , JAKHETIYA V , CHAUDHARY S , et al . Perceptually unimportant information reduction and cosine similarity-based quality assessment of 3D-synthesized images [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 2027 - 2039 .

ZHANG H , ZHENG D S , ZHANG Y , et al . Quality assessment for DIBR-synthesized views based on wavelet transform and gradient magnitude similarity [J ] . IEEE Transactions on Multimedia , 2024 , 26 : 6834 - 6847 .

THAKUR S , JAKHETIYA V , SUBUDHI B N , et al . Context region identification based quality assessment of 3D synthesized views [J ] . IEEE Transactions on Multimedia , 2022 , 25 : 6183 - 6193 .

ZHANG Y , ZHANG H , YU M , et al . Sparse representation based video quality assessment for synthesized 3D videos [J ] . IEEE Transactions on Image Processing , 2020 , 29 : 509 - 524 .

ZHANG Y , YANG X X , LIU X K , et al . High-efficiency 3D depth coding based on perceptual quality of synthesized video [J ] . IEEE Transactions on Image Processing , 2016 , 25 ( 12 ): 5877 - 5891 .

JAKHETIYA V , GU K , JAISWAL S P , et al . Kernel-ridge regression-based quality measure and enhancement of three-dimensional-synthesized images [J ] . IEEE Transactions on Industrial Electronics , 2021 , 68 ( 1 ): 423 - 433 .

TIAN S S , ZHANG L , MORIN L , et al . NIQSV: A no reference image quality assessment metric for 3D synthesized views [C ] // 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Piscataway : IEEE , 2017 : 1248 - 1252 .

TIAN S S , ZHANG L , MORIN L , et al . NIQSV+: A No-reference synthesized view quality assessment metric [J ] . IEEE Transactions on Image Processing , 2018 , 27 ( 4 ): 1652 - 1664 .

GU K , JAKHETIYA V , QIAO J F , et al . Model-based referenceless quality metric of 3D synthesized images using local image description [J ] . IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society , 2018 , 27 ( 1 ): 394 - 405 .

GU K , QIAO J F , LEE S , et al . Multiscale natural scene statistical analysis for no-reference quality evaluation of DIBR-synthesized views [J ] . IEEE Transactions on Broadcasting , 2020 , 66 ( 1 ): 127 - 139 .

WANG G , WANG Z , GU K , et al . Blind quality metric of DIBR-synthesized images in the discrete wavelet transform domain [J ] . IEEE Transactions on Image Processing , 2020 , 29 : 1802 - 1814 .

Sadbhawna , JAKHETIYA V , MUMTAZ D , et al . Stretching artifacts identification for quality assessment of 3D-synthesized views [J ] . IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society , 2021 , 31 : 1737 - 1750 .

LING S Y , LI J , CHE Z H , et al . Re-visiting discriminator for blind free-viewpoint image quality assessment [J ] . IEEE Transactions on Multimedia , 2020 , 23 : 4245 - 4258 .

KIM H G , RO Y M . Measurement of critical temporal inconsistency for quality assessment of synthesized video [C ] // 2016 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2016 : 1027 - 1031 .

WANG G C , WANG Z Y , GU K , et al . Reference-free DIBR-synthesized video quality metric in spatial and temporal domains [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2022 , 32 ( 3 ): 1119 - 1132 .

WANG G C , SUN K Z , TANG L J . No-reference DIBR-synthesized video quality assessment based on spatio-temporal texture inconsistency measurement [C ] // 2022 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS) . Piscataway : IEEE , 2022 : 1 - 4 .

SANDIC-STANKOVIC D D , KUKOLJ D D , LE CALLET P . Fast blind quality assessment of DIBR-synthesized video based on high-high wavelet subband [J ] . IEEE Transactions on Image Processing , 2019 , 28 ( 11 ): 5524 - 5536 .

SANDIC-STANKOVIC D D , KUKOLJ D D , LE CALLET P . Quality assessment of DIBR-synthesized views based on sparsity of difference of closings and difference of Gaussians [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 1161 - 1175 .

JIN C C , PENG Z J , CHEN F , et al . Multi-modal learning-based blind video quality assessment metric for synthesized views [J ] . IEEE Transactions on Broadcasting , 2024 , 70 ( 1 ): 208 - 222 .

YAN J B , LI J , FANG Y M , et al . Subjective and objective quality of experience of free viewpoint videos [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 3896 - 3907 .

JIA R L , ZHANG Y H , XU J , et al . Quality of experience assessment for free-viewpoint video [C ] // 2023 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB) . Piscataway : IEEE , 2023 : 1 - 6 .

WIEN M , BOYCE J M , STOCKHAMMER T , et al . Standardization status of immersive video coding [J ] . IEEE Journal on Emerging and Selected Topics in Circuits and Systems , 2019 , 9 ( 1 ): 5 - 17 .

BAE S J , PARK S , KIM J W , et al . Camera Array Based Windowed 6-DOF Moving Picture Contents [R ] . San Diego : MPEG , 2018 .

DOYEN D , BOISSON G , GENDROT R . EE_DEPTH: New Version of the Pseudo-Rectified Technicolor Painter Content [R ] . Ljubljana : MPEG , 2018 .

BOISSONADE P , JUNG J . Proposition of New Sequences for Windowed-6DoF Experiments on Compression Synthesis and Depth Estimation [R ] . Ljubljana : MPEG , 2018 .

JUNG J , BOISSONADE P , FOURNIER J , et al . Proposition of Navigation Paths and Subjective Evaluation Method for Windowed 6DoF Experiments on Compression, Synthesis, and Depth Estimation [R ] . Ljubljana : MPEG , 2018 .

INSTALLATIONS T , LINE L . Subjective video quality assessment methods for multimedia applications [J ] . Recommendation ITU-TP , 1999 , 910 ( 37 ): 5 .

CHA E Y , JALIL PIRAN M , SUH D Y . A gaze-based real-time and low complexity no-reference video quality assessment technique for video gaming [J ] . Multimedia Tools and Applications , 2024 , 83 ( 7 ): 20889 - 20908 .

LI L D , LIN W S , WANG X S , et al . No-reference image blur assessment based on discrete orthogonal moments [J ] . IEEE Transactions on Cybernetics , 2016 , 46 ( 1 ): 39 - 50 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 770 - 778 .

胡钢 , 徐翔 , 张维明 , 等 . 基于主成分分析的网络节点重要性指标贡献评价 [J ] . 电子学报 , 2019 , 47 ( 2 ): 358 - 365 .

HU G , XU X , ZHANG W M , et al . Contribution analysis for assessing node importance indices with principal component analysis [J ] . Acta Electronica Sinica , 2019 , 47 ( 2 ): 358 - 365 . (in Chinese)

CRIMINISI A , SHOTTON J , KONUKOGLU E . Decision forests: A unified framework for classification, regression, density estimation, manifold learning and semi-supervised learning [J ] . Foundations and Trends® in Computer Graphics and Vision , 2012 , 7 ( 2-3 ): 81 - 227 .

MOORTHY A K , BOVIK A C . A two-step framework for constructing blind image quality indices [J ] . IEEE Signal Processing Letters , 2010 , 17 ( 5 ): 513 - 516 .

MITTAL A , MOORTHY A K , BOVIK A C . No-reference image quality assessment in the spatial domain [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 12 ): 4695 - 4708 .

MITTAL A , SOUNDARARAJAN R , BOVIK A C . Making a “completely blind” image quality analyzer [J ] . IEEE Signal Processing Letters , 2013 , 20 ( 3 ): 209 - 212 .

MITTAL A , SAAD M A , BOVIK A C . A completely blind video integrity oracle [J ] . IEEE Transactions on Image Processing , 2016 , 25 ( 1 ): 289 - 300 .

LI D Q , JIANG T T , JIANG M , et al . Quality assessment of in-the-wild videos [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York : ACM , 2019 : 2351 - 2359 .

SUN W , MIN X K , LU W , et al . A deep learning based no-reference quality assessment model for UGC videos [C ] // Proceedings of the 30th ACM International Conference on Multimedia . New York : ACM , 2022 : 856 - 865 .

DENDI S V R , CHANNAPPAYYA S S . No-reference video quality assessment using natural spatiotemporal scene statistics [J ] . IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society , 2020 : 29 : 5612 - 5624 .

WANG Z , BOVIK A C , SHEIKH H R , et al . Image quality assessment: From error visibility to structural similarity [J ] . IEEE Transactions on Image Processing , 2004 , 13 ( 4 ): 600 - 612 .

周程灏 , 王治乐 , 刘尚阔 . 基于空间变化点扩展函数的图像直接复原方法 [J ] . 光学学报 , 2017 , 37 ( 1 ): 110001 .

ZHOU C H , WANG Z L , LIU S K . Method of image restoration directly based on spatial varied point spread function [J ] . Acta Optica Sinica , 2017 , 37 ( 1 ): 110001 . (in Chinese)

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

C&RM-MAKT Prediction Algorithm Using Multi-Head Attention Mechanism

Semantic Segmentation Network Based on Semantic and Morphological Feature Fusion

Fuzz Testing of the Linux Kernel Integrating Dependency Inference and Weight Optimization

A Lightweight Trajectory Prediction Model with a Spatio-Temporal Factorized Encoder and a Heterogeneous Decoder

Demodulation Method for Starlink Signals Based on Correlation and Clustering

Related Author

ZOU Wen-hui

WANG Lian-hong

LUO Zhi-hui

LIN Fei-peng

LI Xiao-yao

WEI Bo-wen

QUAN Hong-yan

Hong-yan QUAN

Related Institution

School of Electrical and Information Engineering， Hunan University

Software Engineering Institute， East China Normal University

School of Computer Science， East China Normal University

⁰