There is provided a stereoscopic image forming apparatus which can reduce the image memory capacity and the time required for synthesizing a three-dimensional striped image. A three-dimensional image is synthesized such that pixels of the same coordinates in respective images of a multiple viewpoint image sequence of the subject viewed from a plurality of viewpoints are arranged as adjacent pixels in accordance with the arrangement of the viewpoints of the images based on the acquired subject image and an acquired parallax map corresponding thereto. Pixels values of the subject image corresponding to the respective pixels of the three-dimensional image are found according to a parallax deviation derived from the corresponding parallax map and are written into the respective pixels of the three-dimensional image. The synthesized three-dimensional image is outputted, whereby a stereoscopic image can be observed through an optical member having a cyclic structure superposed on the outputted image.