This paper proposes an alogrithm for fusing motion and stereo cues to get robust estimates of both motion and structure. Our algorithm can compute a MAP estimate of the displacement at each pixel and an associated confidence measure by calculating the instantaneous FOE. Using the displacement estimates we calculate a relative depth map from one of the image sequences. This depth map is then used to derive a prior probability distribution for disparity that is used in matching the two frames of stereo paris. This method is used to estimate the disparity at each pixel independently and does not need any assumptions about smoothness.