An autofocus system includes: an imaging device which captures a subject image; a matching image detection device which detects an image that most matches a stored reference pattern image of a target subject to focus on; a determining device that determines that an image of the target subject was detected when a correlation value indicating a degree of matching of the detected image with respect to the reference pattern image is equal to or greater than a predetermined threshold value and determines that an image of the target subject was not detected when the correlation value is less than the threshold value; a changing device which changes the threshold value; and an autofocus device which, only when an image of the target subject was detected, takes a position of the detected image as a position of the target subject to focus on and performs focus adjustment.