The invention features an improved, photo hull based method of generating a three-dimensional representation of a visual scene based upon a set of multi-view reference images. In one aspect, a visual hull containing the visual scene is computed. Visibility of points on the computed visual hull with respect to each reference image is computed. Photo-consistency of points on the computed visual hull along rays back-projected from a desired view is computed based upon reference images having visibility of the points. A photo hull containing the visual scene is generated by stepping along each back-projected ray. The photo-consistent points visible in the desired view are colored using color information contained in reference images having visibility of the photo-consistent points.