A retractable handle assembly has a fixed seat, a grip device disposed in the fixed seat, a pair of cubic blocks, a pair of driven rods, a pair of inner pipes, a pair of middle pipes, a pair of outer pipes, a base seat, a pair of first positioning devices, and a pair of second positioning devices. The fixed seat has two oblong apertures. The grip device has a U-shaped seat disposed in the fixed seat. The U-shaped seat has two downward sleeves. The inner pipes are inserted through the downward sleeves. The cubic blocks are inserted in the inner pipes. The driven rods are inserted in the inner pipes. The inner pipes are inserted in the middle pipes. The middle pipes are inserted in the outer pipes. Each first positioning device is disposed on a bottom of the corresponding inner pipe. Each second positioning device is disposed on a bottom of the corresponding middle pipe.