A control apparatus for controlling a driver driving an optical element of an image pickup device to adjust a focus position of the image pickup device in the capturing of an image of a subject, includes a relative angle variation calculating unit for calculating a relative angle variation, a mode change detecting unit for detecting of a change of a mode of the relative angle variation through determining whether the mode of the relative angle variation, calculated by the relative angle variation calculating unit, is a stable mode involving a small amount of change or an unstable mode involving a large amount of change, and a focus position control process startup unit for managing a startup of a focus position control process controlling the focus position through controlling the driver in response to a pattern of the change of the mode detected by the mode change detecting unit.