The rate distortion optimization (RDO) which uses mode-selection based prediction coding can achieve the highest coding efficiency in H.264/AVC video coding standard.However

exhaustive checking of all the prediction modes increases computational complexity significantly.In this paper

we propose a hierarchical decision structure for fast inter-frame mode selection to reduce the complexity of the exhaustive RDO mode decision process.Different modes of macroblock are first classified into 4 layers in terms of the characteristics of the structure of inter-frame prediction coding in H.264/AVC.Using the coupled relationship between the spatial and temporal complexity of the motion-estimation prediction coding

the adaptive threshold for each layer can be calculated.The obtained threshold can then be applied to each layer to terminate the mode selection process while maintaining almost the same R-D performance in H.264.Experimental results show a 45.79% reduction on computational time and the sane RD performance as the baseline coder by using the proposed fast selection method.The results also demonstrate that the proposed algorithm is superior to two recent fast mode decision algorithms

with the peak signal-to-noise ratio (PSNR) being improved by 0.05dB on average when compared to Zeng's algorithm and the entire encoding time being reduced by about 20% on average when compared to Choi's mode decision algorithm.