An automatic moving object tracking device and a method thereof are capable of tracking a moving object through a signal process of an input image signal of each frame. The moving object tracking device converts a pre-processed input image signal into a binary disturbance image signal, and calculates a motion value of each pixel through an initial tracking window. After locating the tracking window at the pixel where the motion value is a maximum value, a size of the tracking window is adjusted so that the tracking window can include the moving object therein. Next, based on acquired information about the moving object and the tracking window and previously acquired information about previously frames, a pixel in a following frame is estimated to be a next location of the moving object. The tracking window is positioned at the estimated pixel. If the tracking result is satisfied, the tracking of the moving object continues.