A tool holder structure includes a holder installed on a connecting portion of a machine for holding a tool, the holder has a connecting cone for connecting the connecting portion, and a latching portion with an external diameter larger than the connecting cone disposed at the bottom of the connecting cone. The invention installs a ring between the latching portion and a surface of the connecting portion for pressing and connecting the connecting portion to define a tight engagement, so that the holder can be connected to a machine by the connecting cone as well as the ring. The invention can greatly enhance the stability for operations and the precision of manufacturing dimensions, and the invention can be used extensively for various different machines and holders of existing specifications while providing stable operations of the machines and extending the life of the machines.