The interactive video system and method (collectively the "system") provides users with the ability to navigate and otherwise interact with a representation of physical space. Each representation can include a number of viewpoints and objects that can be selected by the user to provide interactive functionality. For example, a user can decide to examine the contents a bookshelf, tilt upwards to look at a chandelier, or zoom in on a tree visible from an exterior window. The motion of tilting, panning, or zooming is conveyed in the form of video clips that accurately resemble the way a human being would perceive such activities. The viewing of objects and viewpoints can be accompanied by the display of information relevant to the user. The various representations can be stored in highly compact files that allow even low-speed dial-up Internet users to effectively navigate a representation in a real-time manner.