基于感知和记忆的视频动态质量评价

林丽群; 暨书逸; 何嘉晨; 赵铁松; 陈炜玲; 郭宗明

doi:10.12263/DZXB.20230283

您当前的位置：

首页 >

文章列表页 >

基于感知和记忆的视频动态质量评价

学术论文 | 更新时间：2025-12-11

- 基于感知和记忆的视频动态质量评价
- Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory
- 电子学报 2024年52卷第11期页码：3727-3740
- 作者机构：
  
  1.福州大学物理与信息工程学院福建省媒体信息智能处理与无线传输重点实验室，福建福州 350108
  2.中国福建光电信息科学与技术创新实验室（闽都创新实验室），福建福州 350108
  3.北京大学王选计算机研究所，北京 100871
- 作者简介：
  
  [ "林丽群女，1980年出生，福建莆田人. 2007年和2019年分别获得福州大学硕士和博士学位.目前任福州大学副教授，硕士生导师，主要研究方向为视频质量评价、视频编码和计算机视觉等. E-mail: lin_liqun@fzu.edu.cn" ]
  [ "暨书逸女，1997年出生，福建武夷山人. 2016年获得福建师范大学学士学位，2022年获得福州大学硕士学位，主要研究方向为视频质量评价和计算机视觉等.E-mail: 973060009@qq.com" ]
  [ "何嘉晨男，2001年出生，湖北武汉人. 2023年获得福州大学学士学位.主要研究方向为视频质量评价和计算机视觉等. E-mail: hjc_18995643869@126.com" ]
  [ "赵铁松男，1984年出生，河北衡水人. 2006年获得中国科学技术大学学士学位，2011年获得香港城市大学博士学位.目前任福州大学教授、博士生导师，“媒体信息智能处理与无线传输”福建省重点实验室主任，在相关领域有十余年的研发经验，曾获得国家级青年人才项目及若干省级人才项目支持.同时担任IEEE高级会员、中国计算机学会高级会员、IET Electronics Letters编委等，并入选团中央指导下的中国青年科技工作者协会会员，当选第六届理事.主要研究方向为多媒体通信系统、人工智能和视频编码等.中国电子学会会员编号：E190014840S. E-mail: t.zhao@fzu.edu.cn" ]
  [ "陈炜玲女，1991年出生，福建福州人. 2009年和2018年分别获得厦门大学学士和博士学位.目前任福州大学副教授，硕士生导师，主要研究方向为智慧海洋、计算机视觉、水下信号处理等.中国电子学会会员编号：E190157944M. E-mail: weiling.chen@fzu.edu.cn" ]
  [ "郭宗明男，1966年出生，江苏盐城人. 1987年和1994年分别获得北京大学学士和博士学位.目前任北京大学研究员，博士生导师，北京大学王选计算机研究所副所长，电子出版新技术国家工程研究中心主任，教育部中国文字字体设计与研究中心主任，主要研究方向为数字视频处理、数字水印、数字版权保护、计算机辅助卡通动画、视频编码、视频质量评价等. E-mail: guozongming@pku.edu.cn" ]
- 基金信息：
  
  国家自然科学基金(62171134);福建省自然科学基金(2022J02015)
- DOI：10.12263/DZXB.20230283
  中图分类号： TP181;TP391.41
- 收稿：2023-03-30，
  
  修回：2023-08-22，
  
  纸质出版：2024-11-25
- 稿件说明：
移动端阅览
林丽群, 暨书逸, 何嘉晨, 等. 基于感知和记忆的视频动态质量评价[J]. 电子学报, 2024, 52(11): 3727-3740.

LIN Li-qun, JI Shu-yi, HE Jia-chen, et al. Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory[J]. Acta Electronica Sinica, 2024, 52(11): 3727-3740.
林丽群, 暨书逸, 何嘉晨, 等. 基于感知和记忆的视频动态质量评价[J]. 电子学报, 2024, 52(11): 3727-3740. DOI：10.12263/DZXB.20230283

LIN Li-qun, JI Shu-yi, HE Jia-chen, et al. Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory[J]. Acta Electronica Sinica, 2024, 52(11): 3727-3740. DOI：10.12263/DZXB.20230283

摘要

由于网络环境的多变性，视频播放过程中容易出现卡顿、比特率波动等情况，严重影响了终端用户的体验质量.为优化网络资源分配并提升用户观看体验，准确评估视频质量至关重要.现有的视频质量评价方法主要针对短视频，普遍关注人眼视觉感知特性，较少考虑人类记忆特性对视觉信息的存储和表达能力，以及视觉感知和记忆特性之间的相互作用.而用户观看长视频的时候，其质量评价需要动态评价，除了考虑感知要素外，还要引入记忆要素.为了更好地衡量长视频的质量评价，本文引入深度网络模型，深入探讨了视频感知和记忆特性对用户观看体验的影响，并基于两者特性提出长视频的动态质量评价模型.首先，本文设计主观实验，探究在不同视频播放模式下，视觉感知特性和人类记忆特性对用户体验质量的影响，构建了基于用户感知和记忆的视频质量数据库（Video Quality Database with Perception And Memory，PAM-VQD）；其次，基于PAM-VQD数据库，采用深度学习的方法，结合视觉注意力机制，提取视频的深层感知特征，以精准评估感知对用户体验质量的影响；最后，将前端网络输出的感知质量分数、播放状态以及自卡顿间隔作为三个特征输入长短期记忆网络，以建立视觉感知和记忆特性之间的时间依赖关系.实验结果表明，所提出的质量评估模型在不同视频播放模式下均能准确预测用户体验质量，且泛化性能良好.

Abstract

Due to the variability of the network environment

video playback is prone to lag and bit rate fluctuations

which seriously affects the quality of end-user experience. In order to optimize network resource allocation and enhance user viewing experience

it is crucial to accurately evaluate video quality. Existing video quality evaluation methods mainly focus on the visual perception characteristics of short videos

with less consideration of the ability of human memory characteristics to store and express visual information

and the interaction between visual perception and memory characteristics. In contrast

when users watch long videos

video quality evaluation needs dynamic evaluation

which needs to consider both perceptual and memory elements. To better measure the quality evaluation of long videos

we introduce a deep network model to deeply explore the impact of video perception and memory characteristics on users' viewing experience

and proposes a dynamic quality evaluation model for long videos based on these two characteristics. Firstly

we design subjective experiments to investigate the influence of visual perceptual features and human memory features on user experience quality under different video playback modes

and constructs a video quality database with perception and memory (PAM-VQD) based on user perception and memory. Secondly

based on the PAM-VQD database

a deep learning methodology is utilized to extract deep perceptual features of videos

combined with visual attention mechanism

in order to accurately evaluate the impact of perception on user experience quality. Finally

the three features of perceptual quality score

playback status and self-lag interval output from the front-end network are fed into the long short-term memory network to establish the temporal dependency between visual perception and memory features. The experimental results show that the proposed quality assessment model can accurately predict the user experience quality under different video playback modes with good generalization performance.

关键词

Keywords

references

曹燕 , 董一鸿 , 邬少清 , 等 . 动态网络表示学习研究进展 [J ] . 电子学报 , 2020 , 48 ( 10 ): 2047 - 2059 .

CAO Y , DONG Y H , WU S Q , et al . Dynamic network representation learning: A review [J ] . Acta Electronica Sinica , 2020 , 48 ( 10 ): 2047 - 2059 . (in Chinese)

易令 , 李泽平 . 基于深度强化学习的码率自适应算法研究 [J ] . 电子学报 , 2022 , 50 ( 5 ): 1192 - 1200 .

YI L , LI Z P . Research of adaptive bitrate algorithm based on deep reinforcement learning [J ] . Acta Electronica Sinica , 2022 , 50 ( 5 ): 1192 - 1200 . (in Chinese)

高敏娟 , 党宏社 , 魏立力 , 等 . 全参考图像质量评价回顾与展望 [J ] . 电子学报 , 2021 , 49 ( 11 ): 2261 - 2272 .

GAO M J , DANG H S , WEI L L , et al . Review and prospect of full reference image quality assessment [J ] . Acta Electronica Sinica , 2021 , 49 ( 11 ): 2261 - 2272 . (in Chinese)

GHAFIL A S , ALI I H . Video streaming forecast quality of experience- A survey [C ] // 2021 1st Babylon International Conference on Information Technology and Science (BICITS) . Piscataway : IEEE , 2021 : 299 - 304 .

LI L , CHEN P , LIN W , et al . From whole video to frames: Weakly-supervised domain adaptive continuous-time QoE evaluation [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 4937 - 4951 .

BAMPIS C G , ZHI LI , MOORTHY A K , et al . Study of temporal effects on subjective video quality of experience [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 11 ): 5217 - 5231 .

DUANMU Z , REHMAN A , WANG Z . A quality-of-experience database for adaptive video streaming [J ] . IEEE Transactions on Broadcasting , 2018 , 64 ( 2 ): 474 - 487 .

SAWABE A , IWAI T . A QoS model to identify required QoS for guaranteeing quality of Internet video streaming services [C ] // ICC 2021 - IEEE International Conference on Communications . Piscataway : IEEE , 2021 : 1 - 6 .

WAHAB A , AHMAD N , SCHORMANS J . Direct propagation of network QoS distribution to subjective QoE for video on demand applications using VP9 codec [C ] // 2020 International Wireless Communications and Mobile Computing (IWCMC) . Piscataway : IEEE , 2020 : 929 - 933 .

SESHADRINATHAN K , SOUNDARARAJAN R , BOVIK A C , et al . Study of subjective and objective quality assessment of video [J ] . Journal of Advanced Pharmaceutical Technology & Research , 2010 , 19 ( 6 ): 1427 - 1441 .

MOORTHY A K , CHOI L K , BOVIK A C , et al . Video quality assessment on mobile devices: Subjective, behavioral and objective studies [J ] . IEEE Journal of Selected Topics in Signal Processing , 2012 , 6 ( 6 ): 652 - 671 .

WU W , LIU Z Z , CHEN Z Z , et al . No-reference video quality assessment based on similarity map estimation [C ] // 2020 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2020 : 181 - 185 .

ITU-R . Methodology for the Subjective Assessment of Video Quality in Multimedia Applications: BT.1788: 2007 [S/OL ] .[ 2023-8-18 ] . https://www.itu.int/rec/R-REC-BT.1788-0-200701-W/en https://www.itu.int/rec/R-REC-BT.1788-0-200701-W/en .

KIANI MEHR S , JOGALEKAR P , MEDHI D . Moving QoE for monitoring DASH video streaming: Models and a study of multiple mobile clients [J ] . Journal of Internet Services and Applications , 2021 , 12 ( 1 ): 1 - 26 .

REHMAN A , WANG Z . Perceptual experience of time-varying video quality [C ] // 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX) . Piscataway : IEEE , 2013 : 218 - 223 .

ITU-R . Methodology for the Subjective Assessment of the Quality of Television Pictures: BT.500-13: 2012 [S/OL ] . [ 2023-08-18 ] . https://www.itu.int/rec/R-REC-BT.500-13-201201-S/en https://www.itu.int/rec/R-REC-BT.500-13-201201-S/en .

ITU-T . Subjective Video Quality Assessment Methods for Multimedia Applications: Rec.P.910: 2008 [S/OL ] . [ 2023-08-18 ] . https://www.itu.int/rec/t-rec-p.910/en https://www.itu.int/rec/t-rec-p.910/en

ZHANG W , LIU H T . Toward a reliable collection of eye-tracking data for image quality research: Challenges, solutions, and applications [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 5 ): 2424 - 2437 .

梁永生 , 柳伟 , 周莺 , 等 . 基于视觉显著计算的视频流媒体渐进式表达方法 [J ] . 电子学报 , 2017 , 45 ( 7 ): 1567 - 1575 .

LIANG Y S , LIU W , ZHOU Y , et al . An approach to progressive description of video streaming based on visual saliency computation [J ] . Acta Electronica Sinica , 2017 , 45 ( 7 ): 1567 - 1575 . (in Chinese)

TRAN H T T , NGUYEN D V , NGOC N P , et al . Overall quality prediction for HTTP adaptive streaming using LSTM network [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 31 ( 8 ): 3212 - 3226 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR) . Piscataway : IEEE , 2016 : 770 - 778 .

WANG Q L , WU B G , ZHU P F , et al . ECA-net: Efficient channel attention for deep convolutional neural networks [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 11534 - 11542 .

GU J P , HU J J , JIANG L , et al . Object detection of overhead transmission lines based on improved YOLOv5s [C ] // 2022 12th International Conference on Power and Energy Systems (ICPES) . Piscataway : IEEE , 2022 : 388 - 392 .

SHI W J , SUN Y J , PAN J Q . Continuous prediction for quality of experience in wireless video streaming [J ] . IEEE Access , 2019 , 7 : 70343 - 70354 .

DONAHUE J , HENDRICKS L A , ROHRBACH M , et al . Long-term recurrent convolutional networks for visual recognition and description [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 4 ): 677 - 691 .

CHEN P , LI L , WU J , et al . Contrastive self-supervised pre-training for video quality assessment [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 458 - 471 .

BAMPIS C G , LI Z , KATSAVOUNIDIS I , et al . Recurrent and dynamic models for predicting streaming video quality of experience [J ] . IEEE Transactions on Image Processing , 2018 , 27 ( 7 ): 3316 - 3331 .

ESWARA N , ASHIQUE S , PANCHBHAI A , et al . Streaming video QoE modeling and prediction: A long short-term memory approach [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 3 ): 661 - 673 .

JOSEPH V , DE VECIANA G . NOVA: QoE-driven optimization of DASH-based video delivery in networks [C ] // IEEE INFOCOM 2014 - IEEE Conference on Computer Communications . Piscataway : IEEE , 2014 : 82 - 90 .

BAMPIS C G , GUPTA P , SOUNDARARAJAN R , et al . SpEED-QA: Spatial efficient entropic differencing for image and video quality [J ] . IEEE Signal Processing Letters , 2017 , 24 ( 9 ): 1333 - 1337 .

SOUNDARARAJAN R , BOVIK A C . Video quality assessment by reduced reference spatio-temporal entropic differencing [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2013 , 23 ( 4 ): 684 - 694 .

KORHONEN J . Two-level approach for no-reference consumer video quality assessment [J ] . IEEE Transactions on Image Processing , 2019 , 28 ( 12 ): 5923 - 5938 .

MITTAL A , MOORTHY A K , BOVIK A C . No-reference image quality assessment in the spatial domain [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 12 ): 4695 - 4708 .

MITTAL A , SOUNDARARAJAN R , BOVIK A C . Making a “Completely blind” image quality analyzer [J ] . IEEE Signal Processing Letters , 2013 , 20 ( 3 ): 209 - 212 .

MOORTHY A K , BOVIK A C . A two-step framework for constructing blind image quality indices [J ] . IEEE Signal Processing Letters , 2010 , 17 ( 5 ): 513 - 516 .

SHEN W H , ZHOU M L , LIAO X R , et al . An end-to-end no-reference video quality assessment method with hierarchical spatiotemporal feature representation [J ] . IEEE Transactions on Broadcasting , 2022 , 68 ( 3 ): 651 - 660 .

KIM W , KIM J , AHN S , et al . Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2018 : 224 - 241 .

CHEN C , CHOI L K , DE VECIANA G , et al . Modeling the time—Varying subjective quality of HTTP video streams with rate adaptations [J ] . IEEE Transactions on Image Processing , 2014 , 23 ( 5 ): 2206 - 2221 .

ESWARA N , MANASA K , KOMMINENI A , et al . A continuous QoE evaluation framework for video streaming over HTTP [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2018 , 28 ( 11 ): 3236 - 3250 .

BAMPIS CHRISTOS G , ZHI L , BOVIK ALAN C . Continuous prediction of streaming video QoE using dynamic networks [J ] . IEEE Signal Processing Letters , 2017 , 24 ( 7 ): 1083 - 1087 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [EB/OL ] . [ 2015-04-10 ] . https://arxiv.org/pdf/1409.1556v6 https://arxiv.org/pdf/1409.1556v6 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks [J ] . Communications of the ACM , 2012 , 60 : 84 - 90 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于EIMYOLO的高分遥感图像目标检测

基于多重注意力和感知加权学习的单图像高动态范围重建

复杂噪声环境下基于轻量化模型的车内交互语音增强和识别方法