A disparity function setting unit configured to set a plurality of disparity relationships expressing disparities as functions of an image position; a data term calculating unit configured to calculate the similarity of corresponding areas between images specified by the preset disparity functions; a smoothing term calculating unit configured to calculate the consistency between the disparity functions and the pixels located in the vicinity; and a disparity function selecting unit configured to select the disparity function for each point of the image from the plurality of preset disparity functions are provided.