To provide a stereo image processing device enabling sufficiently accurate three-dimensional data to be obtained from satellite stereo images and aerial stereo images automatically with no involvement of operators and for complicated objects such as buildings in urban areas. A satellite image storing unit stores satellite image data. A stereo processing unit subjects satellite image data obtained from the satellite image storing unit to stereo matching processing to generate three-dimensional data. A DEM data automatic correcting unit uses information of external shape of buildings and the like obtained from map data in a map data storing unit to correct automatically erroneous data such as noises and losses in the three-dimensional data obtained with the stereo processing unit. The map data storing unit provides map data such as external shapes of buildings and the like to the DEM data automatically correcting unit.