An apparatus for changing cutting tool bits rapidly on milling machines, lathes, drills and other types of rotary power tools includes a first portion that is attached to the rotary power tool and retained thereto. A second portion includes the tool bit attached thereto. The second portion is inserted longitudinally toward the rotary power tool and into a bottom of the first portion after a slide has also first been longitudinally displaced in a direction toward the rotary power tool. A plurality of balls are displaced outward, the balls preferably being ball bearings, thereby permitting an upper conical taper of the second portion to bear against an inside taper of the first portion. The slide is released and urged by an inner spring in an opposite direction forcing the balls to engage with an upper edge of a circular recess provided in the second portion, thereby retaining the second portion in position. Tangs and corresponding recesses cooperate to supply torque that is applied by the rotary tool to the first portion to the second portion and to the tool bit.