A hand tool has an elongated shank with a handle at one end and a bit holder at the other end, the bit holder including a cylindrical body having a distal end surface and an axis, the body having an axial bore formed in the end surface of non-circular transverse cross section and terminating at an inner end surface. A neodymium permanent magnet is freely received in the bore and retained against the inner end surface by a thin circular retainer, formed of metal or plastic, which is interference-fitted in the bore. Both flat, disk-like and concave, bowl-shaped retainers are disclosed. A shock-absorbing cushion may be disposed between the magnet and the inner end surface of the bore. A bit formed of magnetizable material is mateably received in a socket portion of the bore and retained in place by the magnet. The portion of the bore receiving the magnet may have a different cross section from the socket portion. The magnet may be disposed in an encapsulation which interference fits in the bore.