The invention includes a reference oint setting unit configured to extract a plurality of reference points from an input image; a pattern extractor configured to extract a local pattern of the reference points; a characteristic set holder configured to hold a group of characteristic sets having both local patterns of the reference points extracted from a learned image and vectors from the reference points to characteristic points to be detected; a matching unit configured to compare the local patterns extracted from the reference points and the group of characteristic sets and select the nearest characteristic set as a characteristic set having the most similar pattern; and a characteristic point detector configured to detect a final position of the characteristic point based on a vector from the reference point to the characteristic point included in the selected nearest characteristic set.