The present invention provides a method of processing an image, which includes the steps of identifying one candidate for human face region within the image, selecting a mouth neighborhood within the candidate for human face region, processing the mouth neighborhood, and classifying the candidate for human face region based on results of the processing step. According to the method of the present invention, human faces are detected just based on pixels included in the mouth neighborhood, but not all the pixels of the entire face.