Provided is a background recovering apparatus for deleting an object of an arbitrary region and recovering a background covered with the object. The apparatus includes a storing block for storing the input image sequence, and storing the three dimensional (3D) position and posture information, and focal length information; a geometric information extracting block for extracting 3D with respect to the background; a background image generating block for generating the background image; and a background image inserting block for recovering the background by inserting the background image into an arbitrary region of a first view point.