A method and apparatus for detecting and tracking objects using infrared sensors are disclosed. The system is preferably an IRST (infrared, search and tracking) system, and it includes at least one or more infrared sensors, a receiver and a processor. The receiver receives image data from the infrared sensors having a predetermined field of view. The image data includes a reference image, which is preferably a current frame, and a search image, which is preferably a previous frame. The image data is classified to determine whether the image data includes relatively low clutter or relatively high clutter. When the image data includes relatively high clutter of the type typically found in an urban area, the processor performs spot time-difference processing on multiple frames of the image data. The processor detects a moving object from the multiple frames of image data, and it is also capable of detecting stationary objects and random objects.