The sharpness of an image obtained by an endoscope is adjusted. A sharpness enhancement amount is determined for each pixel based on the value of each pixel which forms a luminance distribution image. Further, a portion contributing to adjustment of brightness and a portion contributing to adjustment of saturation in the sharpness enhancement amount are determined for each pixel of the image so that the sum thereof is equal to the sharpness enhancement amount. Then, an adjustment value of saturation, which is obtained by weighting the value of each pixel of the image based on the portion contributing to saturation, and an adjustment value of brightness, which is obtained by weighting the value of each pixel of the luminance distribution image, which corresponds to each pixel of the image, based on the portion contributing to brightness, are added to the value of each pixel of the image.