Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory

LIN Li-qun; JI Shu-yi; HE Jia-chen; ZHAO Tie-song; CHEN Wei-ling; GUO Chong-ming

doi:10.12263/DZXB.20230283

您当前的位置：

首页 >

文章列表页 >

Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory

PAPERS | 更新时间：2025-12-11

- Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory
- ACTA ELECTRONICA SINICA Vol. 52, Issue 11, Pages: 3727-3740(2024)
- 作者机构：
  
  1.福州大学物理与信息工程学院福建省媒体信息智能处理与无线传输重点实验室，福建福州 350108
  2.中国福建光电信息科学与技术创新实验室（闽都创新实验室），福建福州 350108
  3.北京大学王选计算机研究所，北京 100871
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(62171134);Natural Science Foundations of Fujian Province(2022J02015)
- DOI：10.12263/DZXB.20230283
  CLC： TP181;TP391.41
- Received：30 March 2023，
  
  Revised：2023-08-22，
  
  Published：25 November 2024
- 稿件说明：
移动端阅览
林丽群, 暨书逸, 何嘉晨, 等. 基于感知和记忆的视频动态质量评价[J]. 电子学报, 2024, 52(11): 3727-3740.

LIN Li-qun, JI Shu-yi, HE Jia-chen, et al. Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory[J]. Acta Electronica Sinica, 2024, 52(11): 3727-3740.
林丽群, 暨书逸, 何嘉晨, 等. 基于感知和记忆的视频动态质量评价[J]. 电子学报, 2024, 52(11): 3727-3740. DOI：10.12263/DZXB.20230283

LIN Li-qun, JI Shu-yi, HE Jia-chen, et al. Research of Video Dynamic Quality Evaluation Based on Human Perception and Memory[J]. Acta Electronica Sinica, 2024, 52(11): 3727-3740. DOI：10.12263/DZXB.20230283

摘要

由于网络环境的多变性，视频播放过程中容易出现卡顿、比特率波动等情况，严重影响了终端用户的体验质量.为优化网络资源分配并提升用户观看体验，准确评估视频质量至关重要.现有的视频质量评价方法主要针对短视频，普遍关注人眼视觉感知特性，较少考虑人类记忆特性对视觉信息的存储和表达能力，以及视觉感知和记忆特性之间的相互作用.而用户观看长视频的时候，其质量评价需要动态评价，除了考虑感知要素外，还要引入记忆要素.为了更好地衡量长视频的质量评价，本文引入深度网络模型，深入探讨了视频感知和记忆特性对用户观看体验的影响，并基于两者特性提出长视频的动态质量评价模型.首先，本文设计主观实验，探究在不同视频播放模式下，视觉感知特性和人类记忆特性对用户体验质量的影响，构建了基于用户感知和记忆的视频质量数据库（Video Quality Database with Perception And Memory，PAM-VQD）；其次，基于PAM-VQD数据库，采用深度学习的方法，结合视觉注意力机制，提取视频的深层感知特征，以精准评估感知对用户体验质量的影响；最后，将前端网络输出的感知质量分数、播放状态以及自卡顿间隔作为三个特征输入长短期记忆网络，以建立视觉感知和记忆特性之间的时间依赖关系.实验结果表明，所提出的质量评估模型在不同视频播放模式下均能准确预测用户体验质量，且泛化性能良好.

Abstract

Due to the variability of the network environment

video playback is prone to lag and bit rate fluctuations

which seriously affects the quality of end-user experience. In order to optimize network resource allocation and enhance user viewing experience

it is crucial to accurately evaluate video quality. Existing video quality evaluation methods mainly focus on the visual perception characteristics of short videos

with less consideration of the ability of human memory characteristics to store and express visual information

and the interaction between visual perception and memory characteristics. In contrast

when users watch long videos

video quality evaluation needs dynamic evaluation

which needs to consider both perceptual and memory elements. To better measure the quality evaluation of long videos

we introduce a deep network model to deeply explore the impact of video perception and memory characteristics on users' viewing experience

and proposes a dynamic quality evaluation model for long videos based on these two characteristics. Firstly

we design subjective experiments to investigate the influence of visual perceptual features and human memory features on user experience quality under different video playback modes

and constructs a video quality database with perception and memory (PAM-VQD) based on user perception and memory. Secondly

based on the PAM-VQD database

a deep learning methodology is utilized to extract deep perceptual features of videos

combined with visual attention mechanism

in order to accurately evaluate the impact of perception on user experience quality. Finally

the three features of perceptual quality score

playback status and self-lag interval output from the front-end network are fed into the long short-term memory network to establish the temporal dependency between visual perception and memory features. The experimental results show that the proposed quality assessment model can accurately predict the user experience quality under different video playback modes with good generalization performance.

关键词

Keywords

references

曹燕 , 董一鸿 , 邬少清 , 等 . 动态网络表示学习研究进展 [J ] . 电子学报 , 2020 , 48 ( 10 ): 2047 - 2059 .

CAO Y , DONG Y H , WU S Q , et al . Dynamic network representation learning: A review [J ] . Acta Electronica Sinica , 2020 , 48 ( 10 ): 2047 - 2059 . (in Chinese)

易令 , 李泽平 . 基于深度强化学习的码率自适应算法研究 [J ] . 电子学报 , 2022 , 50 ( 5 ): 1192 - 1200 .

YI L , LI Z P . Research of adaptive bitrate algorithm based on deep reinforcement learning [J ] . Acta Electronica Sinica , 2022 , 50 ( 5 ): 1192 - 1200 . (in Chinese)

高敏娟 , 党宏社 , 魏立力 , 等 . 全参考图像质量评价回顾与展望 [J ] . 电子学报 , 2021 , 49 ( 11 ): 2261 - 2272 .

GAO M J , DANG H S , WEI L L , et al . Review and prospect of full reference image quality assessment [J ] . Acta Electronica Sinica , 2021 , 49 ( 11 ): 2261 - 2272 . (in Chinese)

GHAFIL A S , ALI I H . Video streaming forecast quality of experience- A survey [C ] // 2021 1st Babylon International Conference on Information Technology and Science (BICITS) . Piscataway : IEEE , 2021 : 299 - 304 .

LI L , CHEN P , LIN W , et al . From whole video to frames: Weakly-supervised domain adaptive continuous-time QoE evaluation [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 4937 - 4951 .

BAMPIS C G , ZHI LI , MOORTHY A K , et al . Study of temporal effects on subjective video quality of experience [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 11 ): 5217 - 5231 .

DUANMU Z , REHMAN A , WANG Z . A quality-of-experience database for adaptive video streaming [J ] . IEEE Transactions on Broadcasting , 2018 , 64 ( 2 ): 474 - 487 .

SAWABE A , IWAI T . A QoS model to identify required QoS for guaranteeing quality of Internet video streaming services [C ] // ICC 2021 - IEEE International Conference on Communications . Piscataway : IEEE , 2021 : 1 - 6 .

WAHAB A , AHMAD N , SCHORMANS J . Direct propagation of network QoS distribution to subjective QoE for video on demand applications using VP9 codec [C ] // 2020 International Wireless Communications and Mobile Computing (IWCMC) . Piscataway : IEEE , 2020 : 929 - 933 .

SESHADRINATHAN K , SOUNDARARAJAN R , BOVIK A C , et al . Study of subjective and objective quality assessment of video [J ] . Journal of Advanced Pharmaceutical Technology & Research , 2010 , 19 ( 6 ): 1427 - 1441 .

MOORTHY A K , CHOI L K , BOVIK A C , et al . Video quality assessment on mobile devices: Subjective, behavioral and objective studies [J ] . IEEE Journal of Selected Topics in Signal Processing , 2012 , 6 ( 6 ): 652 - 671 .

WU W , LIU Z Z , CHEN Z Z , et al . No-reference video quality assessment based on similarity map estimation [C ] // 2020 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2020 : 181 - 185 .

ITU-R . Methodology for the Subjective Assessment of Video Quality in Multimedia Applications: BT.1788: 2007 [S/OL ] .[ 2023-8-18 ] . https://www.itu.int/rec/R-REC-BT.1788-0-200701-W/en https://www.itu.int/rec/R-REC-BT.1788-0-200701-W/en .

KIANI MEHR S , JOGALEKAR P , MEDHI D . Moving QoE for monitoring DASH video streaming: Models and a study of multiple mobile clients [J ] . Journal of Internet Services and Applications , 2021 , 12 ( 1 ): 1 - 26 .

REHMAN A , WANG Z . Perceptual experience of time-varying video quality [C ] // 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX) . Piscataway : IEEE , 2013 : 218 - 223 .

ITU-R . Methodology for the Subjective Assessment of the Quality of Television Pictures: BT.500-13: 2012 [S/OL ] . [ 2023-08-18 ] . https://www.itu.int/rec/R-REC-BT.500-13-201201-S/en https://www.itu.int/rec/R-REC-BT.500-13-201201-S/en .

ITU-T . Subjective Video Quality Assessment Methods for Multimedia Applications: Rec.P.910: 2008 [S/OL ] . [ 2023-08-18 ] . https://www.itu.int/rec/t-rec-p.910/en https://www.itu.int/rec/t-rec-p.910/en

ZHANG W , LIU H T . Toward a reliable collection of eye-tracking data for image quality research: Challenges, solutions, and applications [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 5 ): 2424 - 2437 .

梁永生 , 柳伟 , 周莺 , 等 . 基于视觉显著计算的视频流媒体渐进式表达方法 [J ] . 电子学报 , 2017 , 45 ( 7 ): 1567 - 1575 .

LIANG Y S , LIU W , ZHOU Y , et al . An approach to progressive description of video streaming based on visual saliency computation [J ] . Acta Electronica Sinica , 2017 , 45 ( 7 ): 1567 - 1575 . (in Chinese)

TRAN H T T , NGUYEN D V , NGOC N P , et al . Overall quality prediction for HTTP adaptive streaming using LSTM network [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 31 ( 8 ): 3212 - 3226 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR) . Piscataway : IEEE , 2016 : 770 - 778 .

WANG Q L , WU B G , ZHU P F , et al . ECA-net: Efficient channel attention for deep convolutional neural networks [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2020 : 11534 - 11542 .

GU J P , HU J J , JIANG L , et al . Object detection of overhead transmission lines based on improved YOLOv5s [C ] // 2022 12th International Conference on Power and Energy Systems (ICPES) . Piscataway : IEEE , 2022 : 388 - 392 .

SHI W J , SUN Y J , PAN J Q . Continuous prediction for quality of experience in wireless video streaming [J ] . IEEE Access , 2019 , 7 : 70343 - 70354 .

DONAHUE J , HENDRICKS L A , ROHRBACH M , et al . Long-term recurrent convolutional networks for visual recognition and description [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 4 ): 677 - 691 .

CHEN P , LI L , WU J , et al . Contrastive self-supervised pre-training for video quality assessment [J ] . IEEE Transactions on Image Processing , 2022 , 31 : 458 - 471 .

BAMPIS C G , LI Z , KATSAVOUNIDIS I , et al . Recurrent and dynamic models for predicting streaming video quality of experience [J ] . IEEE Transactions on Image Processing , 2018 , 27 ( 7 ): 3316 - 3331 .

ESWARA N , ASHIQUE S , PANCHBHAI A , et al . Streaming video QoE modeling and prediction: A long short-term memory approach [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 3 ): 661 - 673 .

JOSEPH V , DE VECIANA G . NOVA: QoE-driven optimization of DASH-based video delivery in networks [C ] // IEEE INFOCOM 2014 - IEEE Conference on Computer Communications . Piscataway : IEEE , 2014 : 82 - 90 .

BAMPIS C G , GUPTA P , SOUNDARARAJAN R , et al . SpEED-QA: Spatial efficient entropic differencing for image and video quality [J ] . IEEE Signal Processing Letters , 2017 , 24 ( 9 ): 1333 - 1337 .

SOUNDARARAJAN R , BOVIK A C . Video quality assessment by reduced reference spatio-temporal entropic differencing [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2013 , 23 ( 4 ): 684 - 694 .

KORHONEN J . Two-level approach for no-reference consumer video quality assessment [J ] . IEEE Transactions on Image Processing , 2019 , 28 ( 12 ): 5923 - 5938 .

MITTAL A , MOORTHY A K , BOVIK A C . No-reference image quality assessment in the spatial domain [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 12 ): 4695 - 4708 .

MITTAL A , SOUNDARARAJAN R , BOVIK A C . Making a “Completely blind” image quality analyzer [J ] . IEEE Signal Processing Letters , 2013 , 20 ( 3 ): 209 - 212 .

MOORTHY A K , BOVIK A C . A two-step framework for constructing blind image quality indices [J ] . IEEE Signal Processing Letters , 2010 , 17 ( 5 ): 513 - 516 .

SHEN W H , ZHOU M L , LIAO X R , et al . An end-to-end no-reference video quality assessment method with hierarchical spatiotemporal feature representation [J ] . IEEE Transactions on Broadcasting , 2022 , 68 ( 3 ): 651 - 660 .

KIM W , KIM J , AHN S , et al . Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network [M ] // Lecture Notes in Computer Science . Cham : Springer International Publishing , 2018 : 224 - 241 .

CHEN C , CHOI L K , DE VECIANA G , et al . Modeling the time—Varying subjective quality of HTTP video streams with rate adaptations [J ] . IEEE Transactions on Image Processing , 2014 , 23 ( 5 ): 2206 - 2221 .

ESWARA N , MANASA K , KOMMINENI A , et al . A continuous QoE evaluation framework for video streaming over HTTP [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2018 , 28 ( 11 ): 3236 - 3250 .

BAMPIS CHRISTOS G , ZHI L , BOVIK ALAN C . Continuous prediction of streaming video QoE using dynamic networks [J ] . IEEE Signal Processing Letters , 2017 , 24 ( 7 ): 1083 - 1087 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [EB/OL ] . [ 2015-04-10 ] . https://arxiv.org/pdf/1409.1556v6 https://arxiv.org/pdf/1409.1556v6 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks [J ] . Communications of the ACM , 2012 , 60 : 84 - 90 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Object Detection Based on EIMYOLO for High-Resolution Remote Sensing Images

Single-Image High Dynamic Range Reconstruction Based on Multi-Attention and Perceptual Weighted Learning

An In-Vehicle Interaction Speech Enhancement and Recognition Method Based on Lightweight Models in Complex Environment

Related Author

GUO Chong-ming

LIN Li-qun

JI Shu-yi

HE Jia-chen

ZHAO Tie-song

CHEN Wei-ling

GUO Chong-ming

CAO Feng

Related Institution

Wangxuan Institute of Computer Technology, Peking University, Haidian

Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information， College of Physics and Information Engineering， Fuzhou University

Wangxuan Institute of Computer Technology， Peking University， Haidian

School of Information and Technology, Shanxi University

School of Computer Science and Technology, Soochow University

⁰