No-Reference Screen Content Image Quality Assessment Based on Edge Assistance and Multi-Scale Transformer

CHEN Yu-zhong; CHEN You-kun; LIN Min-hu; NIU Yu-zhen

doi:10.12263/DZXB.20230607

您当前的位置：

首页 >

文章列表页 >

No-Reference Screen Content Image Quality Assessment Based on Edge Assistance and Multi-Scale Transformer

PAPERS | 更新时间：2025-12-24

- No-Reference Screen Content Image Quality Assessment Based on Edge Assistance and Multi-Scale Transformer
- ACTA ELECTRONICA SINICA Vol. 52, Issue 7, Pages: 2242-2256(2024)
- 作者机构：
  
  1.福州大学计算机与大数据学院，福建福州 350108
  2.福建省网络计算与智能信息处理重点实验室（福州大学），福建福州 350108
  3.大数据智能教育部工程研究中心，福建福州 350108
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(U21A20472;61972097);National Key Research and Development Plan of China(2021YFB3600503);Natural Science Foundation of Fujian Province(2021HZ022007);Industry-Academy Cooperation Project of Fujian Province (China)(2021H6022)
- DOI：10.12263/DZXB.20230607
  CLC： TN911.73;TP391
- Received：03 July 2023，
  
  Revised：2024-03-04，
  
  Published：25 July 2024
- 稿件说明：
移动端阅览
陈羽中, 陈友昆, 林闽沪, 等. 基于边缘辅助和多尺度Transformer的无参考屏幕内容图像质量评估[J]. 电子学报, 2024, 52(07): 2242-2256.

CHEN Yu-zhong, CHEN You-kun, LIN Min-hu, et al. No-Reference Screen Content Image Quality Assessment Based on Edge Assistance and Multi-Scale Transformer[J]. Acta Electronica Sinica, 2024, 52(07): 2242-2256.
陈羽中, 陈友昆, 林闽沪, 等. 基于边缘辅助和多尺度Transformer的无参考屏幕内容图像质量评估[J]. 电子学报, 2024, 52(07): 2242-2256. DOI：10.12263/DZXB.20230607

CHEN Yu-zhong, CHEN You-kun, LIN Min-hu, et al. No-Reference Screen Content Image Quality Assessment Based on Edge Assistance and Multi-Scale Transformer[J]. Acta Electronica Sinica, 2024, 52(07): 2242-2256. DOI：10.12263/DZXB.20230607

摘要

与从现实场景中拍摄的自然图像不同，屏幕内容图像是一种合成图像，通常由计算机生成的文本、图形和动画等各种多媒体形式组合而成.现有评估方法通常未能充分考虑图像边缘结构信息和全局上下文信息对屏幕内容图像质量感知的影响.为解决上述问题，本文提出一种基于边缘辅助和多尺度Transformer的无参考屏幕内容图像质量评估模型.首先，使用高斯拉普拉斯算子构造由失真屏幕内容图像高频信息组成的边缘结构图，然后通过卷积神经网络（Convolutional Neural Network，CNN）对输入的失真屏幕内容图像和相应的边缘结构图进行多尺度的特征提取与融合，以图像的边缘结构信息为模型训练提供额外的信息增益.此外，本文进一步构建了基于Transformer的多尺度特征编码模块，从而在CNN获得的局部特征基础上更好地建模不同尺度图像和边缘特征的全局上下文信息.实验结果表明，本文提出的方法在指标上优于其他现有的无参考和全参考屏幕内容图像质量评估方法，能够取得更高的主客观视觉感知一致性.

Abstract

Different from the natural images captured from real-world scenes

screen content images (SCI) are synthetic images typically composed of various multimedia contents

such as computer-generated text

graphics

and animations. Existing SCI quality assessment methods usually fail to fully consider the impacts of image edge and global context on the perceived quality of screen content images. To address the above issues

this paper proposed a no-reference screen content image quality assessment model based on edge assistance and multi-scale Transformer. Firstly

an edge structure map consisting of the high-frequency information in a distorted SCI is constructed using Gaussian Laplace operators. Then a convolutional neural network (CNN) is used to extract and fuse the multi-scale features from the input distorted SCI and the corresponding edge structure map

thus providing additional edge information gain for model training. In addition

this paper further proposed a multi-scale feature encoding module based on Transformer to better model the global context information of different scale images and edge features on the basis of the local features obtained by CNN. The experimental results show that the model proposed in this paper outperforms the state-of-the-art no-reference and full-reference SCI quality assessment methods

and achieves higher consistency with the subjective visual perception.

关键词

Keywords

references

PENG W H , WALLS F G , COHEN R A , et al . Overview of screen content video coding: Technologies, standards, and beyond [J ] . IEEE Journal on Emerging and Selected Topics in Circuits and Systems , 2016 , 6 ( 4 ): 393 - 408 .

贾旭 , 曹玉东 , 孙福明 , 等 . 基于无参考质量评价模型的静脉图像采集方法 [J ] . 电子学报 , 2015 , 43 ( 2 ): 236 - 241 .

JIA X , CAO Y D , SUN F M , et al . Vein image acquisition method based on quality assessment model without reference [J ] . Acta Electronica Sinica , 2015 , 43 ( 2 ): 236 - 241 . (in Chinese)

陈文俊 , 杨春玲 . 图像压缩感知的特征域优化及自注意力增强神经网络重构算法 [J ] . 电子学报 , 2022 , 50 ( 11 ): 2629 - 2637 .

CHEN W J , YANG C L . Feature-space optimization-inspired and self-attention enhanced neural network reconstruction algorithm for image compressive sensing [J ] . Acta Electronica Sinica , 2022 , 50 ( 11 ): 2629 - 2637 . (in Chinese)

李群迎 , 张晓林 . 基于多描述和不等差错保护的航空遥感图像传输方法 [J ] . 电子学报 , 2010 , 38 ( 11 ): 2655 - 2659 .

LI Q Y , ZHANG X L . Aerial remote sensing image transmission using multiple description coding and unequal error protection [J ] . Acta Electronica Sinica , 2010 , 38 ( 11 ): 2655 - 2659 . (in Chinese)

NI Z K , ZENG H Q , MA L , et al . A Gabor feature-based quality assessment model for the screen content images [J ] . IEEE Transactions on Image Processing , 2018 , 27 ( 9 ): 4516 - 4528 .

魏乐松 , 陈俊豪 , 牛玉贞 . 基于边缘和结构的无参考屏幕内容图像质量评估 [J ] . 北京航空航天大学学报 , 2019 , 45 ( 12 ): 2449 - 2455 .

WEI L S , CHEN J H , NIU Y Z . Blind quality assessment for screen content images based on edge and structure [J ] . Journal of Beijing University of Aeronautics and Astronautics , 2019 , 45 ( 12 ): 2449 - 2455 . (in Chinese)

林冠妙 , 魏乐松 , 牛玉贞 . 基于多尺度特征的无参考屏幕内容图像质量评估 [J ] . 小型微型计算机系统 , 2022 , 43 ( 2 ): 372 - 380 .

LIN G M , WEI L S , NIU Y Z . No reference image quality assessment for screen content image based on multi-scale features [J ] . Journal of Chinese Computer Systems , 2022 , 43 ( 2 ): 372 - 380 . (in Chinese)

ZHANG Y , CHANDLER D M , MOU X Q . Quality assessment of screen content images via convolutional-neural-network-based synthetic/natural segmentation [J ] . IEEE Transactions on Image Processing , 2018 , 27 ( 10 ): 5113 - 5128 .

JIANG X H , SHEN L Q , DING Q , et al . Screen content image quality assessment based on convolutional neural networks [J ] . Journal of Visual Communication and Image Representation , 2020 , 67 : 102745 .

GAO R , HUANG Z Q , LIU S G . Multi-task deep learning for no-reference screen content image quality assessment [C ] // International Conference on Multimedia Modeling . Cham : Springer , 2021 : 213 - 226 .

徐少平 , 李芬 , 陈孝国 , 等 . 一种利用改进深度图像先验构建的图像降噪模型 [J ] . 电子学报 , 2022 , 50 ( 7 ): 1573 - 1578 .

XU S P , LI F , CHEN X G , et al . An image denoising model using the improved deep image prior [J ] . Acta Electronica Sinica , 2022 , 50 ( 7 ): 1573 - 1578 . (in Chinese)

YANG J C , BIAN Z L , LIU J C , et al . No-reference quality assessment for screen content images using visual edge model and Adaboosting neural network [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 6801 - 6814 .

ZENG C , KWONG S . Learning transformer features for image quality assessment [EB/OL ] . ( 2021-12-01 )[ 2023-07-01 ] . http://arxiv.org/abs/2112.00485 http://arxiv.org/abs/2112.00485 .

YOU J Y , KORHONEN J . Transformer for image quality assessment [C ] // 2021 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2021 : 1389 - 1393 .

CHEON M , YOON S J , KANG B , et al . Perceptual image quality assessment with transformers [C ] // 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE , 2021 : 433 - 442 .

GUO H , MA K K , ZENG H Q . A Log-Gabor feature-based quality assessment model for screen content images‍ [C ] // 2019 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2019 : 4499 - 4503 .

KUANG W , CHAN Y L , TSANG S H , et al . Online-learning-based Bayesian decision rule for fast intra mode and CU partitioning algorithm in HEVC screen content coding [J ] . IEEE Transactions on Image Processing , 2020 , 29 : 170 - 185 .

MIN X K , GU K , ZHAI G T , et al . Screen content quality assessment: Overview, benchmark, and beyond [J ] . ACM Computing Surveys , 2021 , 54 ( 9 ): 1 - 36 .

YUE G H , HOU C P , YAN W Q , et al . Blind quality assessment for screen content images via convolutional neural network [J ] . Digital Signal Processing , 2019 , 91 : 21 - 30 .

JIANG X H , SHEN L Q , YU L W , et al . No-reference screen content image quality assessment based on multi-region features [J ] . Neurocomputing , 2020 , 386 : 30 - 41 .

YANG J C , BIAN Z L , ZHAO Y , et al . Staged-learning: Assessing the quality of screen content images from distortion information [J ] . IEEE Signal Processing Letters , 2021 , 28 : 1480 - 1484 .

ZHANG C F , HUANG Z Q , LIU S G , et al . Dual-channel multi-task CNN for no-reference screen content image quality assessment [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2022 , 32 ( 8 ): 5011 - 5025 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems . New York : ACM , 2017 : 6000 - 6010 .

KE J J , WANG Q F , WANG Y L , et al . MUSIQ: Multi-scale image quality transformer [C ] // 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2021 : 5128 - 5137 .

WANG J , FAN H T , HOU X X , et al . MSTRIQ: No reference image quality assessment based on swin transformer with multi-stage fusion [C ] // 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) . Piscataway : IEEE , 2022 : 1268 - 1277 .

LIU Z , LIN Y T , CAO Y , et al . Swin transformer: Hierarchical vision transformer using shifted windows [C ] // 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE , 2021 : 9992 - 10002 .

ZHU M M , HOU G Q , CHEN X J , et al . Saliency-guided transformer network combined with local embedding for no-reference image quality assessment [C ] // 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) . Piscataway : IEEE , 2021 : 1953 - 1962 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE , 2016 : 770 - 778 .

ZEILER M D , FERGUS R . Visualizing and understanding convolutional networks [C ] // European Conference on Computer Vision . Cham : Springer , 2014 : 818 - 833 .

YANG H , FANG Y M , LIN W S . Perceptual quality assessment of screen content images [J ] . IEEE Transactions on Image Processing , 2015 , 24 ( 11 ): 4408 - 4421 .

NI Z K , MA L , ZENG H Q , et al . ESIM: Edge similarity for screen content image quality assessment [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 10 ): 4818 - 4831 .

VQEG . Final Report From the Video Quality Experts Group on the Validation of Objective Models of Video [R/OL ] . ( 1997-10-14 )[ 2023-06-27 ] . https://vqeg.org/vqeg-home.aspx https://vqeg.org/vqeg-home.aspx .

WANG Z , BOVIK A C , SHEIKH H R , et al . Image quality assessment: From error visibility to structural similarity‍ [J ] . IEEE Transactions on Image Processing , 2004 , 13 ( 4 ): 600 - 612 .

XUE W F , ZHANG L , MOU X Q , et al . Gradient magnitude similarity deviation: A highly efficient perceptual image quality index [J ] . IEEE Transactions on Image Processing , 2014 , 23 ( 2 ): 684 - 695 .

GU K , WANG S Q , YANG H , et al . Saliency-guided quality assessment of screen content images [J ] . IEEE Transactions on Multimedia , 2016 , 18 ( 6 ): 1098 - 1110 .

TOLIE H F , FARAJI M R . Screen content image quality assessment using distortion-based directional edge and gradient similarity maps [J ] . Signal Processing , 2022 , 101 : 116562 .

FANG Y M , DU R G , ZUO Y F , et al . Perceptual quality assessment for screen content images by spatial continuity‍ [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 30 ( 11 ): 4050 - 4063 .

YANG J C , ZHAO Y , LIU J C , et al . No reference quality assessment for screen content images using stacked autoencoders in pictorial and textual regions [J ] . IEEE Transactions on Cybernetics , 2022 , 52 ( 5 ): 2798 - 2810 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cross-CNN: An Animation Cross-Frame Sketch Colorization Algorithm Based on Hybrid Model with CNN and Transformer

A Review of Single Image Super-Resolution Reconstruction Based on Deep Learning

Related Author

CHEN Yu-zhong

NIU Yu-zhen

YU Yi-feng

QIAN Jiang-bo

YAN Di-qun

WANG Chong

DONG Li

WU Jing

Related Institution

Big Data Intelligence Engineering Research Center of the Ministry of Education， Fuzhou University

Faculty of Electrical Engineering and Computer Science， Ningbo University

Zhejiang Key Laboratory of Mobile Network Application Technology

School of Mechanical Engineering and Automation, Fuzhou University

Advanced Technology Innovation Institute, Fuzhou University

⁰