Moving object detection in complex scenes is an important problem in computer vision domain

and the detection accuracy is still a great challenge. In this paper

we propose and design a deep frame difference convolution neural network (DFDCNN) for moving object detection in complex scenes. DFDCNN consists of DifferenceNet and AppearanceNet

which can predict and segment the foreground pixels simultaneously without post-processing. DifferenceNet has Siamese Encoder-Decoder structure

which is used to learn changes between two consecutive frames and to obtain temporal information from inputs

while AppearanceNet is used to extract spatial information from the input frame

and fuse the temporal information and spatial information by fusion of feature maps. Finally

multi-scale spatial information is retained through multi-scale feature map fusion and stepwise up-sampling to improve the sensitivity to small objects. Experiments on two public standard datasets: CDnet2014 and I2R demonstrate that the proposed DFDCNN outperforms the classic algorithms significantly from both qualitative and quantitative aspects. The experimental results illustrate that the proposed DFDCNN shows much better detection performance in complex scenes where dynamic background

illumination variation and shadow exist

and there is improvement for scenes

in which small objects exist.