An image processing device generates an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information obtained by an image sensor which has multiple pixels each photoelectrically converting light of a plurality of colors. The device includes: a simulation unit for generating a second image as an estimated image of the sparsely color sampled image; a comparison unit for comparing the second image with the first image, and generating first information thereof; a converting unit for converting the first information into second information corresponding to the color image; a penalty computing unit for computing the correction amount of the input image; and a correction unit for generating a third image by correcting the input image based on the second information and the correcting amount; this third image being used as an output image.