A method for creating a novel viewpoint image from a plurality of images includes the steps of simultaneously and continuously acquiring sets of images from a plurality of cameras. Predetermined background based correspondence fields are used to detect novel objects. Image representations are assigned for these objects likely new correspondences. These likely new correspondences are tested and further improved upon in a refinement step. The resulting correspondences are used to construct a novel viewpoint image.