A retracting mechanism includes a linearly movable ring; a swingable holder supported inside the linearly movable ring; a position-controller holding the swingable holder; a pair of support plates attached to opposite ends of the linearly movable ring; a support plate fixing device fixing the support plates to the linearly movable ring; a rotatable shaft having a pair of eccentric pins at opposite ends thereof, and having a common axis eccentric to the rotatable shaft; and a pair of elongated holes on the pair of support plates, facing each other and elongated parallel to each other, the eccentric pins being engaged in the elongated holes and being movable therein. The pair of support plates are movable relative to the linearly movable ring in directions orthogonal to the optical axis via rotation of the rotatable shaft when the support plate fixing device is in a released state.