An optical element retracting mechanism includes a linearly movable ring; a swingable holder mounted supported inside the linearly movable ring; a position-controller holding the swingable holder; a support plate including first and second elongated holes; a support plate fixing device fixing the support plate to the linearly movable ring; first and second rotatable shafts extending substantially parallel to the optical axis having first and second eccentric pins engaged in the first and second elongated holes, wherein when the first or second rotatable shaft is rotated, a movement force is applied on the support plate in a direction generally orthogonal to the elongation direction of the elongated hole thereof; and a movement direction setting device which sets the movement direction of the support plate when the movement force is applied on the support plate by the rotation of the first or second rotatable shaft.