视频编码率失真优化技术研究综述

doi:10.3969/j.issn.0372-2112.2020.05.024

摘要/Abstract

摘要： 率失真优化技术能有效地提升编码器的压缩性能，是视频编码领域重要的研究内容.本文综合论述了基于拉格朗日乘子的率失真优化方法，从独立率失真优化、依赖率失真优化和码率控制中的比特分配等几方面归纳和分析最新研究进展.首先，通过形象的举例解释了约束优化问题通过拉格朗日乘子法转换为非约束优化问题，并根据相关文献分析拉格朗日乘子与量化参数的关系.其次，分析了编码过程中存在的率失真依赖性，并分类讨论近年提出的依赖率失真优化方法.然后，简要介绍了率失真依赖关系在码率控制比特分配中的影响.最后通过分析比较不同率失真优化技术的特征和性能，提出当前面临的挑战和值得进一步探索的工作.

关键词: 视频编码, 率失真优化, 时域依赖, 空域依赖, 自适应拉格朗日乘子

Abstract: Rate-distortion optimization (RDO) can effectively improve the compression performance for encoders,which is an important topic in video coding.In this paper,the Lagrange multiplier based RDO method is reviewed comprehensively from several aspects including independent RDO,dependent RDO,bit allocation for rate control,etc.First,an example is employed to explain how the constrained optimization problem is converted into an unconstrained optimization problem based on the Lagrange multiplier method,and the relationship between Lagrange multiplier and quantization parameter is analyzed according to relevant references.Second,the rate-distortion dependencies in the process of video coding are analyzed,and the dependent RDO methods presented in recent years are classified and discussed.Third,the effect of the rate-distortion dependencies on bit allocation for rate control is discussed.Finally,the features and performance of different RDO methods are tabulated for analysis and comparison,and the existing challenges and the research works deserving further exploration are presented.

Key words: video coding, rate-distortion optimization, temporal dependency, spatial dependency, adaptive Lagrange multiplier

中图分类号:

TN919.8

郭红伟, 朱策, 刘宇洋. 视频编码率失真优化技术研究综述[J]. 电子学报, 2020, 48(5): 1018-1029.

GUO Hong-wei, ZHU Ce, LIU Yu-yang. Overview of Rate-Distortion Optimization for Video Coding[J]. Acta Electronica Sinica, 2020, 48(5): 1018-1029.

参考文献

[1] SULLIVAN G J,WIEGAND T.Rate-distortion optimization for video compression[J].IEEE Signal Processing Magazine,1998,15(6):74-90.
[2] SHANNON C E.Coding theorems for a discrete source with a fidelity criterion[J].Institute of Radio Engineers,International Convention Record,1959,7(7):142-163.
[3] 杨天武,彭强,邓云.一种基于率失真优化的关键参考帧选择算法[J].电子学报,2006,34(7):1241-1245. YANG T-W,PENG Q,DENG Y.A novel key frame reference picture selection algorithm based on rate-distortion optimization[J].Acta Electronica Sinica,2006,34(7):1241-1245.(in Chinese)
[4] LI H Q,LI B,XU J Z.Rate-distortion optimized reference picture management for high efficiency video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2012,22(12):1844-1857.
[5] 陆寄远,朝红阳,黄承慧.计算能力可伸缩的运动估计率失真优化[J].电子学报,2014,42(8):1495-1502. LU J-Y,CHAO H-Y,HUANG C-H.Rate distortion optimization of complexity scalable motion estimation[J].Acta Electronica Sinica,2014,42(8):1495-1502.(in Chinese)
[6] ORTEGA A,RAMCHANDRAN K.Rate-distortion methods for image and video compression[J].IEEE Signal Processing Magazine,1998,15(6):23-50.
[7] WIEGAND T,GIROD B.Lagrange multiplier selection in hybrid video coder control[A].Proceedings of International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2001.542-545.
[8] LI X,OERTEL N,KAUP A.Adaptive Lagrange multiplier selection for intra-frame video coding[A].Proceedings of IEEE International Symposium on Circuits and Systems (ISCAS)[C].Piscataway:IEEE,2007.3643-3646.
[9] LI X,OERTEL N,HUTTER A,et al.Laplace distribution based lagrangian rate distortion optimization for hybrid video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2009,19(2):193-205.
[10] ZHANG J,YI X Q,LING N,et al.Context adaptive lagrange multiplier (CALM) for rate-distortion optimal motion estimation in video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2010,20(6):820-828.
[11] GONZALEZ-DE-SUSO J L,JIMENEZ-MORENO A,et al.Improved method to select the Lagrange multiplier for rate-distortion based motion estimation in video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2014,24(3):452-464.
[12] ZHANG F,BULL D R.Rate-distortion optimization using adaptive Lagrange multipliers[J].IEEE Transactions on Circuits and Systems for Video Technology,2019,29(10):3121-3131.
[13] AN C H,NGUYEN T Q.Adaptive Lagrange multiplier selection using classification-maximization and its application to chroma qp offset decision[J].IEEE Transactions on Circuits and Systems for Video Technology,2011,21(6):783-791.
[14] LI B,XU J Z,ZHANG D,et al.QP refinement according to lagrange multiplier for high efficiency video coding[A].Proceedings of International Symposium on Circuits and Systems[C].Piscataway:IEEE,2013.477-480.
[15] LI B,LI H Q,LI L,et al. λ-domain rate control algorithm for high efficiency video coding[J].IEEE Transactions on Image Processing,2014,23(9):3841-3854.
[16] WANG M H,NGAN K N,LI H L.An efficient frame-content based intra frame rate control for high efficiency video coding[J].IEEE Signal Processing Letters,2015,22(7):896-900.
[17] LI L,LI B,LIU D,et al.λ-domain rate control algorithm for HEVC scalable extension[J].IEEE Transactions on Multimedia,2016,18(10):2023-2039.
[18] LI B,XU J,LI H.Refining QP to improve coding efficiency in AVS[A].Proceedings of IEEE International Conference on Image Processing[C].Piscataway:IEEE,2013.1528-1531.
[19] 张方,吴成柯,肖嵩.一种基于MPEG-4的感兴趣区域视频编码新方法[J].电子学报,2005,33(4):634-637. ZHANG F,WU C K,XIAO S.A new regions of interest video coding method based on MPEG-4[J].Acta Electronica Sinica,2005,33(4):634-637.(in Chinese)
[20] LI Y,TAO X M,LU J.Hybrid model-and-object-based real-time conversational video coding[J].Signal Processing-Image Communication,2015,35(2015):9-19.
[21] HU P,SHUAI B,LIU J,et al.Deep level sets for salient object detection[A].Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition[C].Piscataway:IEEE,2017.540-549.
[22] XU M,JIANG L,SUN X Y,et al.Learning to detect video saliency with HEVC features[J].IEEE Transactions on Image Processing,2017,26(1):369-385.
[23] XIONG B,FAN X J,ZHU C,et al.Face region based conversational video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2011,21(7):917-931.
[24] YANG X K,LIN W S,LU Z K,et al.Rate control for videophone using local perceptual cues[J].IEEE Transactions on Circuits and Systems for Video Technology,2005,15(4):496-507.
[25] LIU Y,LI Z G,SOH Y C.Region-of-interest based resource allocation for conversational video communication of H.264/AVC[J].IEEE Transactions on Circuits and Systems for Video Technology,2008,18(1):134-139.
[26] XU M,DENG X,LI S X,et al.Region-of-Interest based conversational HEVC coding with hierarchical perception model of face[J].IEEE Journal of Selected Topics in Signal Processing,2014,8(3):475-489.
[27] ZHANG Z W,JING T,HAN J N,et al.A new rate control scheme for video coding based on region of interest[J].IEEE Access,2017,5(0):13677-13688.
[28] HADIZADEH H,BAJIC I V.Saliency-aware video compression[J].IEEE Transactions on Image Processing,2014,23(1):19-33.
[29] LI S,XU M,DENG X,et al.Weight-based R-λ rate control for perceptual HEVC coding on conversational videos[J].Signal Processing-Image Communication,2015,38(2015):127-140.
[30] ZENG H Q,YANG A S,NGAN K N,et al.Perceptual sensitivity-based rate control method for high efficiency video coding[J].Multimedia Tools and Applications,2016,75(17):10383-10396.
[31] LI S X,XU M,et al.Closed-form optimization on saliency-guided image compression for HEVC-MSP[J].IEEE Transactions on Multimedia,2018,20(1):155-170.
[32] ZENG H,NGAN K N,WANG M.Perceptual adaptive Lagrangian multiplier for high efficiency video coding[A].Proceedings of Picture Coding Symposium[C].Piscataway:IEEE,2013:69-72.
[33] LAINEMA J,BOSSEN F,et al.Intra coding of the HEVC standard[J].IEEE Transactions on Circuits and Systems for Video Technology,2012,22(12):1792-1801.
[34] LIN J L,CHEN Y W,HUANG Y W,et al.Motion vector coding in the HEVC standard[J].IEEE Journal of Selected Topics in Signal Processing,2013,7(6):957-968.
[35] BICHON M,TANOU J L,et al.Inter-block dependencies consideration for intra coding in H.264/AVC and HEVC standards[A].Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing[C].Piscataway:IEEE,2017.1537-1541.
[36] BICHON M,LE TANOU J,ROPERT M,et al.Low complexity joint RDO of prediction units couples for HEVC intra coding[A].Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing[C].Piscataway:IEEE,2018.1733-1737.
[37] YOU J,CHOI C,JEONG J.Modified rate distortion optimization using inter-block dependence for H.264/AVC intra coding[J].IEEE Transactions on Consumer Electronics,2008,54(3):1383-1388.
[38] SUN L,AU O C,DAI W,et al.Modified distortion redistribution problem for high efficiency video coding (HEVC)[A].Proceedings of IEEE 14th International Workshop on Multimedia Signal Processing (MMSP)[C].Piscataway:IEEE,2012.278-282.
[39] PANG C,AU O C,ZOU F,et al.Optimal distortion redistribution in block-based image coding using successive convex optimization[A].Proceedings of IEEE International Conference on Multimedia and Expo (ICME)[C].Piscataway:IEEE,2011.1-5.
[40] WU Q B,XIONG J,LUO B,et al.A novel joint rate distortion optimization scheme for intra prediction coding in H.264/AVC[J].IEICE Transactions on Information and Systems,2014,E97-D(4):989-992.
[41] KIM I-K,MCCANN K,SUGIMOTO K,et al.High Efficiency Video Coding (HEVC) Test Model 11(HM11) Encoder Description[R].Document JCTVC-M1002,Incheon,KR,2013.
[42] SCHWARZ H,MARPE D,WIEGAND T.Overview of the scalable video coding extension of the H.264/AVC standard[J].IEEE Transactions on Circuits and Systems for Video Technology,2007,17(9):1103-1120.
[43] LI X A,AMON P,HUTTER A,et al.Model based analysis for quantization parameter cascading in hierarchical video coding[A].Proceedings of IEEE International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2009.3765-3768.
[44] WAN W X,CHEN Y,WANG Y K,et al.Efficient hierarchical inter picture coding for H.264/AVC baseline profile[A].Proceedings of Picture Coding Symposium (PCS)[C].Piscataway:IEEE,2009.1-4.
[45] LI X,AMON P,HUTTER A,et al.Adaptive quantization parameter cascading for hierarchical video coding[A].Proceedings of IEEE International Symposium on Circuits and Systems[C].Piscataway:IEEE,2010.4197-4200.
[46] XU Y,LI Q,LI X,et al.Efficient QP cascading in H.265/HEVC low-delay prediction[A].Proceedings of IEEE International Conference on Multimedia & Expo Workshops[C].Piscataway:IEEE,2017.1-6.
[47] AMER H,YANG E H.Low-delay HEVC adaptive quantization parameter selection through temporal propagation length estimation[A].Proceedings of IEEE International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2018.211-215.
[48] PAPADOPOULOS M A,ZHANG F,et al.An adaptive QP offset determination method for HEVC[A].Proceedings of IEEE International Conference on Image Processing[C].Piscataway:IEEE,2016.4220-4224.
[49] YANG Y,WAN S,GONG Y,et al.Adaptive quantization parameter cascading for random-access prediction in H.265/HEVC based on dependent R-D models[A].Proceedings of IEEE International Conference on Image Processing[C].Piscataway:IEEE,2016.4235-4239.
[50] ZHAO T S,WANG Z,CHEN C W.Adaptive quantization parameter cascading in HEVC hierarchical coding[J].IEEE Transactions on Image Processing,2016,25(7):2997-3009.
[51] GONG Y C,WAN S,YANG K F,et al.Rate-distortion-optimization-based quantization parameter cascading technique for random-access configuration in H.265/HEVC[J].IEEE Transactions on Circuits and Systems for Video Technology,2017,27(6):1304-1312.
[52] YANG K,WAN S,GONG Y,et al.Content adaptive quantization parameter cascading for random-access structure in HEVC[A].Proceedings of IEEE International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2017.2498-2502.
[53] ZHOU Y M,WANG H Y,TIAN L,et al.Temporal corelation based hierarchical quantization parameter determination for HEVC video coding[A].Proceedings of IEEE International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2017.2478-2482.
[54] HE J,YANG E H,et al.Adaptive quantization parameter selection for H.265/HEVC by employing inter-frame dependency[J].IEEE Transactions on Circuits and Systems for Video Technology,2018,28(12):3424-3436.
[55] WANG M H,NGAN K N,et al.Improved block level adaptive quantization for high efficiency video coding[A].Proceedings of IEEE International Symposium on Circuits and Systems[C].Piscataway:IEEE,2015.509-512.
[56] YIN H B,CAI H,FAN M T,et al.Quantization parameter cascading for video coding:leveraging a new temporal distortion propagation model[J].Signal Image and Video Processing,2017,11(5):801-808.
[57] XIANG G Q,JIA H Z,YANG M Y,et al.A novel adaptive quantization method for video coding[J].Multimedia Tools and Applications,2018,77(12):14817-14840.
[58] ROPERT M,TANOU J L,BICHON M,et al.R-D spatio-temporal adaptive quantization based on temporal distortion backpropagation in HEVC[A].Proceedings of IEEE International Workshop on Multimedia Signal Processing (MMSP)[C].Piscataway:IEEE,2017.1-6.
[59] BICHON M,TANOU J L,ROPERT M,et al.Temporal adaptive quantization using accurate estimations of inter and skip probabilities[A].Proceedings of Picture Coding Symposium[C].Piscataway:IEEE,2018.81-85.
[60] YANG T,ZHU C,FAN X,et al.Source distortion temporal propagation model for motion compensated video coding optimization[A].Proceedings of International Conference on Multimedia & Expo (ICME)[C].Piscataway:IEEE,2012.85-90.
[61] GAO Y,ZHU C,LI S,et al.Temporal dependent rate-distortion optimization for low-delay hierarchical video coding[J].IEEE Transactions on Image Processing,2017,26(9):4457-4470.
[62] GAO Y,ZHU C,et al.Source distortion temporal propagation analysis for random-access hierarchical video coding optimization[J].IEEE Transactions on Circuits and Systems for Video Technology,2019,29(2):546-559.
[63] LI S,ZHU C,GAO Y B,et al.Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[J].IEEE Transactions on Circuits and Systems for Video Technology,2016,26(1):117-129.
[64] XU J Z,JOSHI R,COHEN R A.Overview of the emerging HEVC screen content coding extension[J].IEEE Transactions on Circuits and Systems for Video Technology,2016,26(1):50-62.
[65] XIAO W,LI B,XU J,et al.Weighted rate-distortion optimization for screen content coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2018,28(2):499-512.
[66] GONZALEZ-DE-SUSO J L,MARTINEZ-ENRIQUEZ E,DIAZ-DE-MARIA F.Adaptive Lagrange multiplier estimation algorithm in HEVC[J].Signal Processing-Image Communication,2017,56(2017):40-51.
[67] YANG K F,WAN S,GONG Y C,et al.An efficient Lagrangian multiplier selection method based on temporal dependency for rate-distortion optimization in H.265/HEVC[J].Signal Processing-Image Communication,2017,57(2017):68-75.
[68] ZHANG F,BULL D R.An adaptive Lagrange multiplier determination method for rate-distortion optimisation in hybrid video codecs[A].Proceedings of IEEE International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2015.671-675.
[69] WANG X,SONG L,LUO Z,et al.Lagrangian method based rate-distortion optimization revisited for dependent video coding[A].Proceedings of IEEE International Conference on Image Processing (ICIP)[C].Piscataway:IEEE,2017.3021-3025.
[70] DE ABREU A,CHEUNG G,FROSSARD P,et al.Optimal Lagrange multipliers for dependent rate allocation in video coding[J].Signal Processing-Image Communication,2018,63(2018):113-124.
[71] 崔子冠,朱秀昌,干宗良.H.264视频编码率失真优化和码率控制技术研究进展[J].电子学报,2013,41(12):2443-2450. CUI Z G,ZHU X C,GAN Z L.Advances in rate distortion optimization and rate control techniques for h.264 video coding[J].Acta Electronica Sinica,2013,41(12):2443-2450.(in Chinese)
[72] CHEN Z Z,NGAN K N.Recent advances in rate control for video coding[J].Signal Processing-Image Communication,2007,22(2007):19-38.
[73] CHOI H,YOO J,NAM J,et al.Pixel-wise unified rate-quantization model for multi-level rate control[J].IEEE Journal of Selected Topics in Signal Processing,2013,7(6):1112-1123.
[74] LEE B,KIM M,NGUYEN T Q.A frame-level rate control scheme based on texture and nontexture rate models for high efficiency video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2014,24(3):465-479.
[75] WANG S,REHMAN A,ZENG K,et al.SSIM-motivated two-pass VBR coding for HEVC[J].IEEE Transactions on Circuits and Systems for Video Technology,2017,27(10):2189-2203.
[76] ZUPANCIC I,NACCARI M,MRAK M,et al.Two-pass rate control for improved quality of experience in UHDTV delivery[J].IEEE Journal of Selected Topics in Signal Processing,2017,11(1):167-179.
[77] WEN J T,FANG M Y,TANG M H,et al.R-lambda model based improved rate control for HEVC with pre-encoding[A].Proceedings of Data Compression Conference (DCC)[C].Piscataway:IEEE,2015.53-62.
[78] LIU M H,REN P,XIANG Z.Frame-level bit allocation for hierarchical coding of H.265/HEVC considering dependent rate-distortion characteristics[J].Signal Image and Video Processing,2016,10(8):1457-1463.
[79] LI L,LI B,LI H,et al. λ-domain optimal bit allocation algorithm for high efficiency video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2018,28(1):130-142.
[80] FIENGO A,CHIERCHIA G,CAGNAZZO M,et al.Rate allocation in predictive video coding using a convex optimization framework[J].IEEE Transactions on Image Processing,2017,26(1):479-489.
[81] WANG S S,MA S W,et al.Rate-GOP based rate control for high efficiency video coding[J].IEEE Journal of Selected Topics in Signal Processing,2013,7(6):1101-1111.
[82] Gao W,Kwong S,Yuan H,et al.DCT coefficient distribution modeling and quality dependency analysis based frame-level bit allocation for HEVC[J].IEEE Transactions on Circuits and Systems for Video Technology,2016,26(1):139-153.
[83] GUO H W,ZHU C,LI S X,et al.Optimal bit allocation at frame level for rate control in HEVC[J].IEEE Transactions on Broadcasting,2019,65(2):270-281.
[84] LI S,XU M,WANG Z,et al.Optimal bit allocation for CTU level rate control in HEVC[J].IEEE Transactions on Circuits and Systems for Video Technology,2017,27(11):2409-2424.
[85] GUO H W,ZHU C,XU M,et al.Inter-block dependency based CTU level rate controlfor HEVC[J].IEEE Transactions on Broadcasting,2020,66(1):113-126.
[86] WIEN M,BOYCE J M,STOCKHAMMER T,et al.Standardization status of immersive video coding[J].IEEE Journal on Emerging and Selected Topics in Circuits and Systems,2019,9(1):5-17.
[87] GARRIDO M J,PESCADOR F,CHAVARRÍAS M,et al.A 2-D multiple transform processor for the versatile video coding standard[J].IEEE Transactions on Consumer Electronics,2019,65(3):274-283.
[88] YANG H,SHEN L,DONG X,et al.Low complexity CTU partition structure decision and fast intra mode decision for versatile video coding[J].IEEE Transactions on Circuits and Systems for Video Technology,2019,doi:10.1109/TCSVT.2019.2904198.