CABAC is the entropy coding method adopted in H.264/AVC main profile

and the combination of RDO and CABAC can represent an improvement in bit rate of around 20%.During the RDO mode decision

entropy coding is performed for every candidate encoding mode

which increases the computation complexity of video encoding drastically.To avoid the expensive computation of Lagrange costs

this paper proposes a transform-domain bit-rate estimation method for CABAC for the first time

in which the bit-rate is estimated based on the quantized transform coefficients and the motion vector differences.After that

a fast mode decision algorithm based on the bit-rate estimation technique is presented

in which the bit-rate estimation is performed instead of actual entropy coding in mode decision

thus reducing the computing time of H.264 encoder.Simulation results show that the proposed transform-domain bit-rate estimation method for CABAC achieves about 51% computation time reduction of Lagrange cost for mode decision and saves about 33% total encoding time with FS(full search)motion estimation algorithm

while the losses of PSNR and bit-rate are negligible.