An optical element retracting mechanism includes a linearly movable ring; a swingable holder supported inside the linearly movable ring; a position-controller holding the swingable holder; an internal optical element positioned inside the linearly movable ring on one of opposite sides of a retractable optical element; a pair of parallel flat surfaces located on the linearly movable ring, each extending generally orthogonal to the optical axis and respectively positioned on front and rear sides of the internal optical element, the pair of parallel flat surfaces not overlapping a second optical element in the optical axis direction; a through-hole formed on the linearly movable ring so as to be open at the pair of parallel flat surfaces in the optical axis direction, the pivot being positioned in the through hole; and a pair of support plates attached to the pair of parallel flat surfaces to support the pivot.