基于对比性视觉-文本模型的光场图像质量评估

王汉灵; 柯逍; 江澳鑫; 郭文忠

doi:10.12263/DZXB.20240533

您当前的位置：

首页 >

文章列表页 >

基于对比性视觉-文本模型的光场图像质量评估

学术论文 | 更新时间：2026-05-07

- 基于对比性视觉-文本模型的光场图像质量评估
- Quality Assessment of Light Field Images Based on Contrastive Visual-Textual Model
- 电子学报 2024年52卷第10期页码：3562-3577
- 作者机构：
  
  1.福州大学计算机与大数据学院，福建福州350116
  2.中国地震局工程力学研究所地震工程与工程振动重点实验室，黑龙江哈尔滨 150080
  3.福建省网络计算与智能信息处理重点实验室，福建福州 350116
  4.大数据智能教育部工程研究中心，福建福州 350116
- 作者简介：
  
  [ "王汉灵男，1999年4月出生于福建省福州市.现为中国地震局工程力学研究所博士研究生. E-mail: 2209187058@qq.com" ]
  [ "柯逍男，1983年10月出生于福建省福州市.现为福州大学计算机与大数据学院教授、博士生导师.主要研究方向为计算机视觉与机器学习. E-mail: kex@fzu.edu.cn" ]
  [ "江澳鑫男，1999年出生于福建省福州市. 现为福州大学计算机技术硕士研究生.主要研究方向为计算机视觉. E-mail: jax991220@163.com" ]
  [ "郭文忠男，1979年生出生于福建. 现为福州大学副校长，福州大学计算机科学与技术学科带头人、博士生导师." ]
- 基金信息：
  
  国家重点研发计划(2021YFB3600503);国家自然科学基金(61972097;U21A20472);福建省科技重大专项(2021HZ022007);福建省自然科学基金(2021J01612)
- DOI：10.12263/DZXB.20240533
  中图分类号： TP183
- 收稿：2024-06-05，
  
  修回：2024-07-23，
  
  纸质出版：2024-10-25
- 稿件说明：
移动端阅览
王汉灵, 柯逍, 江澳鑫, 等. 基于对比性视觉-文本模型的光场图像质量评估[J]. 电子学报, 2024, 52(10): 3562-3577.

WANG Han-ling, KE Xiao, JIANG Ao-xin, et al. Quality Assessment of Light Field Images Based on Contrastive Visual-Textual Model[J]. Acta Electronica Sinica, 2024, 52(10): 3562-3577.
王汉灵, 柯逍, 江澳鑫, 等. 基于对比性视觉-文本模型的光场图像质量评估[J]. 电子学报, 2024, 52(10): 3562-3577. DOI：10.12263/DZXB.20240533

WANG Han-ling, KE Xiao, JIANG Ao-xin, et al. Quality Assessment of Light Field Images Based on Contrastive Visual-Textual Model[J]. Acta Electronica Sinica, 2024, 52(10): 3562-3577. DOI：10.12263/DZXB.20240533

摘要

光场图像作为一种能够捕获场景每个位置光线信息的图像类型，在电子成像、医学影像和虚拟现实等领域具有广泛的应用前景.光场图像质量评估（Light Field Image Quality Assessment，LFIQA）旨在衡量此类图像的质量，但当前方法面临视觉效果与文本模态间异构性的重要挑战.为解决上述问题，本文提出了一种基于文本-视觉的多模态光场图像质量评估模型.具体来说，在视觉模态方面，我们设计了多任务模型，结合边缘自动阈值算法有效丰富了光场图像的关键表示特征.在文本模态方面，基于输入噪声特征与预测噪声特征的对比，准确识别光场图像的噪声类别，并验证了噪声预测对优化视觉表示的重要性.基于上述研究，进一步提出了一种优化的通用噪声文本配置方法，并结合边缘增强策略，显著提升了基线模型在光场图像质量评估中的准确性和泛化能力.此外，通过消融实验，评估了各组件对整体模型性能的贡献，验证了本文方法的有效性和稳健性.实验结果表明，该方法不仅在公开数据集Win5-LID和NBU-LF1.0的实验中表现出色，还在融合数据集中展示出优秀的实验结果，与现有最优算法相比，本文所提方法在两个数据库中的性能分别提升了2%和6%.本文提出的噪声验证策略和配置方法不仅为图像质量评估中的噪声预测任务提供了有价值的参考，也可用于其它噪声预测类型的辅助任务.

Abstract

Light field imaging

as an image type capable of capturing light information from every position in a scene

holds broad application prospects in fields such as electronic imaging

medical imaging

and virtual reality. Light field image quality assessment (LFIQA) aims to measure the quality of such images

yet current methods confront significant challenges arising from the heterogeneity between visual effects and textual modalities. To address these issues

this paper proposes a multi-modal light field image quality assessment model grounded in text-vision integration. Specifically

for the visual modality

we devise a multi-task model that effectively enriches the crucial representational features of light field images by incorporating an edge auto-thresholding algorithm. On the textual side

we accurately identify noise categories in light field images based on the comparison between input noise features and predicted noise features

thereby validating the importance of noise prediction in optimizing visual representations. Building upon these findings

we further introduce an optimized universal noise text configuration approach combined with an edge enhancement strategy

which notably enhances the accuracy and generalization capabilities of the baseline model in LFIQA. Additionally

ablation experiments are conducted to assess the contribution of each component to the overall model performance

thereby verifying the effectiveness and robustness of our proposed method. Experimental results demonstrate that our approach not only excels in tests on public datasets like Win5-LID and NBU-LF1.0 but also shows remarkable outcomes in fused datasets. Compared to the state-of-the-art algorithms

our method achieves performance improvements of 2% and 6% respectively on the two databases. The noise verification strategy and configuration method presented in this paper not only provide valuable insights for light field noise prediction tasks but can also be applied as auxiliary tools for other noise prediction types.

关键词

Keywords

references

WU G , MASIA B , JARABO A , et al . Light field image processing: An overview [J ] . IEEE Journal of Selected Topics in Signal Processing , 2017 , 11 ( 7 ): 926 - 954 .

CAO Y , LI S , LIU Y , et al . A comprehensive survey of ai-generated content (AIGC): A history of generative ai from gan to chatgpt [EB/OL ] . ( 2023-03-07 )[ 2024-06-05 ] . https://arxiv.org/pdf/2303.04226 https://arxiv.org/pdf/2303.04226 .

林华 . 无人机载太赫兹合成孔径雷达成像分析与仿真 [J ] . 信息与电子工程 , 2010 , 8 ( 4 ): 373 - 377 .

LIN H . Analysis and simulation of UAV terahertz wave synthetic aperture radar imaging [J ] . Information and Electronic Engineering , 2010 , 8 ( 4 ): 373 - 377, 382 . (in Chinese)

刘慧芳 , 周骛 , 蔡小舒 , 等 . 基于光场成像的三维粒子追踪测速技术 [J ] . 光学学报 , 2020 , 40 ( 1 ): 0111014 .

LIU H F , ZHOU W , CAI X S , et al . Three-dimensional particle tracking velocimetry based on light field imaging [J ] . Acta Optica Sinica , 2020 , 40 ( 1 ): 0111014 . (in Chinese)

WANG Y , WANG L , LIANG Z , et al . NTIRE 2023 challenge on light field image super-resolution: Dataset, methods and results [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 1320 - 1335 .

WOOD D N , AZUMA D I , ALDINGER K , et al . Surface light fields for 3D photography [M ] // Seminal Graphics Papers: Pushing the Boundaries , Volume 2 . New York : ACM , 2023: 487 - 496 .

WANG Z , BOVIK A C . Modern Image Quality Assessment [D ] . Kentfield : Morgan & Claypool Publishers , 2006 .

SHEIKH H R , SABIR M F , BOVIK A C . A statistical evaluation of recent full reference image quality assessment algorithms [J ] . IEEE Transactions on Image Processing , 2006 , 15 ( 11 ): 3440 - 3451 .

BOSSE S , MANIRY D , MULLER K R , et al . Deep neural networks for no-reference and full-reference image quality assessment [J ] . IEEE Transactions on Image Processing , 2017 , 27 ( 1 ): 206 - 219 .

LARSON E C , CHANDLER D M . Most apparent distortion: full-reference image quality assessment and the role of strategy [J ] . Journal of Electronic Imaging , 2010 , 19 ( 1 ): 011006 .

TANG Z , ZHENG Y , GU K , et al . Full-reference image quality assessment by combining features in spatial and frequency domains [J ] . IEEE Transactions on Broadcasting , 2018 , 65 ( 1 ): 138 - 151 .

MITTAL A , MOORTHY A K , BOVIK A C . No-reference image quality assessment in the spatial domain [J ] . IEEE Transactions on Image Processing , 2012 , 21 ( 12 ): 4695 - 4708 .

KANG L , YE P , LI Y , et al . Convolutional neural networks for no-reference image quality assessment [C ] // 2014 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2014 : 1733 - 1740 .

WANG Y , YANG J , WANG L , et al . Light field image super-resolution using deformable convolution [J ] . IEEE Transactions on Image Processing , 2020 , 30 : 1057 - 1071 .

CHEN M J , CORMACK L K , BOVIK A C . No-reference quality assessment of natural stereopairs [J ] . IEEE Transactions on Image Processing , 2013 , 22 ( 9 ): 3379 - 3391 .

YANG J , WANG Y , LI B , et al . Quality assessment metric of stereo images considering cyclopean integration and visual saliency [J ] . Information Sciences , 2016 , 373 : 251 - 268 .

HUANG H , ZENG H , TIAN Y , et al . Light field image quality assessment: An overview [C ] // 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) . Piscataway : IEEE , 2020 : 348 - 353 .

TIAN Y , ZENG H , HOU J , et al . Light field image quality assessment via the light field coherence [J ] . IEEE Transactions on Image Processing , 2020 , 29 : 7945 - 7956 .

TIAN Y , ZENG H , HOU J , et al . A light field image quality assessment model based on symmetry and depth features [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2020 , 31 ( 5 ): 2046 - 2050 .

HAFNER M , KATSANTONI M , KOSTER T , et al . CLIP and complementary methods [J ] . Nature Reviews Methods Primers , 2021 , 1 ( 1 ): 1 - 23 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [C ] // 2021 International Conference on Machine Learning . New York : PMLR , 2021 : 8748 - 8763 .

CHAI J X , TONG X , CHAN S C , et al . Plenoptic sampling [C ] // Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques . New York : ACM , 2000 : 307 - 318 .

ADELSON E H , BERGEN J R . The plenoptic function and the elements of early vision [M ] // Computational Models of Visual Processing . Cambridge : The MIT Press , 1991 .

SAHA N , IFTHEKHAR M S , LE N T , et al . Survey on optical camera communications: challenges and opportunities [J ] . Iet Optoelectronics , 2015 , 9 ( 5 ): 172 - 183 .

HUNT J , GOLLUB J , DRISCOLL T , et al . Metamaterial microwave holographic imaging system [J ] . Journal of the Optical Society of America A , 2014 , 31 ( 10 ): 2109 - 2119 .

PARK M J , KIM D J , LEE U , et al . A literature overview of virtual reality (VR) in treatment of psychiatric disorders: Recent advances and limitations [J ] . Frontiers in Psychiatry , 2019 , 10 : 505 .

ARDINY H , KHANMIRZA E . The role of AR and VR technologies in education developments: opportunities and challenges [C ] // 2018 6th RSI International Conference on Robotics and Mechatronics . Piscataway : IEEE , 2018 : 482 - 487 .

YUEN S C Y , YAOYUNEYONG G , JOHNSON E . Augmented reality: An overview and five directions for AR in education [J ] . Journal of Educational Technology Development and Exchange (JETDE) , 2011 , 4 ( 1 ): 11 .

SPEICHER M , HALL B D , NEBELING M . What is mixed reality? [C ] // Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . New York : ACM , 2019 : 1 - 15 .

陈琦 , 徐熙平 , 姜肇国 , 等 . 基于光场相机的四维光场图像水印及质量评价 [J ] . 光学学报 , 2018 , 38 ( 4 ): 0411003 .

CHEN Q , XU X P , JIANG Z G , et al . Watermarking scheme for four dimensional light field imaging based on light field camera and its evaluation [J ] . Acta Optica Sinica , 2018 , 38 ( 4 ): 0411003 . (in Chinese)

赵圆圆 , 施圣贤 . 融合多尺度特征的光场图像超分辨率方法 [J ] . 光电工程 , 2020 , 47 ( 12 ): 200007 .

ZHAO Y Y , SHI S X . Light-field image super-resolution based on multi-scale feature fusion [J ] . Opto-Electronic Engineering , 2020 , 47 ( 12 ): 200007 . (in Chinese)

KARAMAN M , O'DONNELL M . Subaperture processing for ultrasonic imaging [J ] . IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control , 1998 , 45 ( 1 ): 126 - 135 .

CALLOWAY T M , DONOHOE G W . Subaperture autofocus for synthetic aperture radar [J ] . IEEE Transactions on Aerospace and Electronic Systems , 1994 , 30 ( 2 ): 617 - 621 .

XIANG J , YU M , JIANG G , et al . Blind light field image quality assessment with tensor color domain and 3D shearlet transform [J ] . Signal Processing , 2023 , 211 : 109083 .

CUI Y , JIANG G , YU M , et al . Stitched wide field of view light field image quality assessment: Benchmark database and objective metric [J ] . IEEE Transactions on Multimedia , 2024 , 26 : 5092 - 5107 .

PAN Z , YU M , JIANG G , et al . Combining tensor slice and singular value for blind light field image quality assessment [J ] . IEEE Journal of Selected Topics in Signal Processing , 2021 , 15 ( 3 ): 672 - 687 .

DAFFERTSHOFER A , LAMOTH C J C , MEIJER O G , et al . PCA in studying coordination and variability: a tutorial [J ] . Clinical Biomechanics , 2004 , 19 ( 4 ): 415 - 428 .

HORE A , ZIOU D . Image quality metrics: PSNR vs. SSIM [C ] // 2010 20th International Conference on Pattern Recognition . Piscataway : IEEE , 2010 : 2366 - 2369 .

SARA U , AKTER M , UDDIN M S . Image quality assessment through FSIM, SSIM, MSE and PSNR—A comparative study [J ] . Journal of Computer and Communications , 2019 , 7 ( 3 ): 8 - 18 .

SETIADI D R I M . PSNR vs SSIM: Imperceptibility quality assessment for image steganography [J ] . Multimedia Tools and Applications , 2021 , 80 ( 6 ): 8423 - 8444 .

PAUDYAL P , BATTISTI F , CARLI M . Reduced reference quality assessment of light field images [J ] . IEEE Transactions on Broadcasting , 2019 , 65 ( 1 ): 152 - 165 .

黄虹 , 张建秋 . 一个图像质量盲评估的统计测度 [J ] . 电子学报 , 2014 , 42 ( 7 ): 1419 - 1423 .

HUANG H , ZHANG J Q . A statistical measure for blind image quality assessment [J ] . Acta Electronica Sinica , 2014 , 42 ( 7 ): 1419 - 1423 . (in Chinese)

王长淼 , 李晖 , 张水平 , 等 . 基于深度学习的光场显微像差校正 [J ] . 光学学报 , 2024 , 44 ( 14 ): 90 - 99 .

WANG C M , LI H , ZHANG S P , et al . Light field microscopic aberration correction based on deep learning [J ] . Acta Optica Sinica , 2024 , 44 ( 14 ): 90 - 99 . (in Chinese)

梁丹 , 张海苗 , 邱钧 . 基于自监督学习的光场空间域超分辨成像 [J ] . 激光与光电子学进展 , 2024 , 61 ( 4 ): 172 - 184 .

LIANG D , ZHANG H M , QIU J . Self-supervised learning for spatial-domain light-field super-resolution imaging [J ] . Laser & Optoelectronics Progress , 2024 , 61 ( 4 ): 172 - 184 . (in Chinese)

SHAO S , XING L , XU R , et al . MDFM: Multi-decision fusing model for few-shot learning [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2021 , 32 ( 8 ): 5151 - 5162 .

SANDIĆ-STANKOVIĆ D , KUKOLJ D , LE CALLET P . Multi-scale synthesized view assessment based on morphological pyramids [J ] . Journal of Electrical Engineering , 2016 , 67 ( 1 ): 3 - 11 .

叶佳 , 张建秋 , 胡波 . 客观评估彩色图像质量的超复数奇异值分解法 [J ] . 电子学报 , 2007 , 35 ( 1 ): 28 - 33 .

YE J , ZHANG J Q , HU B . Hyper complex singular value decomposition approach to objectively assessing color image quality [J ] . Acta Electronica Sinica , 2007 , 35 ( 1 ): 28 - 33 . (in Chinese)

SHI L , ZHAO S , CHEN Z . BELIF: Blind quality evaluator of light field image with tensor structure variation index [C ] // 2019 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE , 2019 : 3781 - 3785 .

ZHANG Z , TIAN S , ZOU W , et al . Deeblif: Deep blind light field image quality assessment by extracting angular and spatial information [C ] // 2022 IEEE International Conference on Image Processing . Piscataway : IEEE , 2022 : 2266 - 2270 .

XIANG J , YU M , CHEN H , et al . VBLFI: Visualization-based blind light field image quality assessment [C ] // 2020 IEEE International Conference on Multimedia and Expo . Piscataway : IEEE , 2020 : 1 - 6 .

ZHANG W , ZHAI G , WEI Y , et al . Blind image quality assessment via vision-language correspondence: A multitask learning perspective [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2023 : 14071 - 14081 .

DING L , GOSHTASBY A . On the Canny edge detector [J ] . Pattern Recognition , 2001 , 34 ( 3 ): 721 - 725 .

李季瑀 , 付章杰 , 王帆 . Canny-Gauss通用域图像隐写算法 [J ] . 计算机学报 , 2024 , 47 ( 1 ): 213 - 230 .

LI J Y , FU Z J , WANG F . Canny-Gauss universal domain image steganography algorithm [J ] . Chinese Journal of Computers , 2024 , 47 ( 1 ): 213 - 230 . (in Chinese)

PEREZ-LOMBARD L , ORTIZ J , MAESTRE I R . The map of energy flow in HVAC systems [J ] . Applied Energy , 2011 , 88 ( 12 ): 5020 - 5031 .

GAO W , ZHANG X , YANG L , et al . An improved Sobel edge detection [C ] // 2010 3rd International Conference on Computer Science and Information Technology . Piscataway : IEEE , 2010 , 5 : 67 - 71 .

WU J , CUI Z , SHENG V S , et al . A comparative study of SIFT and its variants [J ] . Measurement Science Review , 2013 , 13 ( 3 ): 122 - 131 .

ATHAR S , WANG Z . Degraded reference image quality assessment [J ] . IEEE Transactions on Image Processing , 2023 , 32 : 822 - 837 .

TIAN Y , ZENG H , XING L , et al . A multi-order derivative feature-based quality assessment model for light field image [J ] . Journal of Visual Communication and Image Representation , 2018 , 57 : 212 - 217 .

MA J , ZHANG X , JIN C , et al . Light field image quality assessment using natural scene statistics and texture degradation [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2024 , 34 ( 3 ): 1696 - 1711 .

SHI L , ZHOU W , CHEN Z , et al . No-reference light field image quality assessment based on spatial-angular measurement [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2019 , 30 ( 11 ): 4114 - 4128 .

DENDI S V R , CHANNAPPAYYA S S . No-reference video quality assessment using natural spatiotemporal scene statistics [J ] . IEEE Transactions on Image Processing , 2020 , 29 : 5612 - 5624 .

TU Z , WANG Y , BIRKBECK N , et al . UGC-VQA: Benchmarking blind video quality assessment for user generated content [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 4449 - 4464 .

ZHANG L , ZHANG L , BOVIK A C . A feature-enriched completely blind image quality evaluator [J ] . IEEE Transactions on Image Processing , 2015 , 24 ( 8 ): 2579 - 2591 .

ZHOU W , SHI L , CHEN Z , et al . Tensor oriented no-reference light field image quality assessment [J ] . IEEE Transactions on Image Processing , 2020 , 29 : 4070 - 4084

刘玉轩 , 张力 , 艾海滨 , 等 . 光场相机三维重建研究进展与展望 [J ] . 电子学报 , 2022 , 50 ( 7 ): 1774 - 1792 .

LIU Y X , ZHANG L , AI H B , et al . Progress and prospect of 3D reconstruction based on light field cameras [J ] . Acta Electronica Sinica , 2022 , 50 ( 07 ): 1774 - 1792 . (in Chinese)

周广福 , 文成林 , 高敬礼 . 基于小波变换与稀疏傅里叶变换相结合的光场重构方法 [J ] . 电子学报 , 2017 , 45 ( 4 ): 782 - 790 .

ZHOU G f , WEN C L , GAO J L . Light field reconstruction based on wavelet transform and sparse Fourier Transform [J ] . Acta Electronica Sinica , 2017 , 45 ( 4 ): 782 - 790 . (in Chinese)

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于内容语义感知多模态融合的图像增强方法

基于渐进式边缘感知交互的多退化暗光图像增强

结合多尺度循环卷积和多聚类空间的红外图像增强

基于快速模糊聚类的动态多直方图均衡化算法