A method and apparatus for directly sensing both the focused image and the three-dimensional shape of a scene are disclosed. This invention is based on a novel mathematical transform named Rao Transform (RT) and its inverse (IRT). RT and IRT are used for accurately modeling the forward and reverse image formation process in a camera as a linear shift-variant integral operation. Multiple images recorded by a camera with different camera parameter settings are processed to obtain 3D scene information. This 3D scene information is used in computer vision applications and as input to a virtual digital camera which computes a digital still image. This same 3D information for a time-varying scene can be used by a virtual video camera to compute and produce digital video data.