A focus adjustment apparatus comprises a moving unit configured to move a focusing lens, a focus adjusting unit configured to move the focusing lens such that the sharpness of an image obtained from an image pickup unit picking up an object image is increased, and an in-focus position measuring unit configured to measure an in-focus position in a different manner from the focus adjusting unit. The focus adjusting unit moves the focusing lens to the in-focus position measured by the in-focus position measuring unit when a direction of movement of the focusing lens in which the image sharpness is increased is matched with the direction of a current position of the focusing lens relative to the in-focus position measured by the in-focus position measuring unit, and does not move the focusing lens to the measured in-focus position when both the directions are not matched with each other.