A system for generating three-dimensional models from still imagery or video streams from uncalibrated views, is presented. The system comprises a computer system including a processor, a memory coupled with the processor, an input coupled with the processor for receiving imagery captured from multiple pan-tilt settings of an uncalibrated image capturing device, and an output coupled with the processor for outputting an overall three-dimensional model of a complex scene containing rigid and non-rigid objects. The computer system includes a method for forming a three-dimensional model from uncalibrated views of an object, and a method for automatically stitching together three-dimensional models extracted from uncalibrated views of uncalibrated image capturing device locations, without the need of manual "image registration" of "points in common" between the models. Furthermore, the system uses the object's prior shape information to generate a robust matching scheme that supports the detection of missing features and occlusions between views.