An image matching method and system for use with multiple images of a scene captured from different angles. Image matching is performed by identifying a plurality of segments within at least two images, determining an initial disparity values for pixels in the images and then determining initial disparity planes for the segments by fitting a plane to initial disparity values for the segments. A refined disparity plane set is created by iteratively refitting the disparity planes by using various fitting cost functions and weighted linear systems. A labeling of each segment to a disparity plane is made by minimizing a global energy function that includes energy terms for segment to disparity plane matching as well as penalizing disparity plane discontinuities between adjacent image segments.