A multi-stage infrared (IR) image-based tracking system is provided that accurately identifies and tracks targets across a variety of different situations and environmental conditions. The system includes primary and secondary trackers. The secondary tracker supports the primary tracker by identifying and tracking targets when the primary tracker loses or fails to identify one or more targets, and helps the primary tracker reacquire valid targets. In an exemplary embodiment of the invention, the primary tracker is a correlation-based tracker, and the secondary tracker is a feature-based tracker. A prescreener is also included, which operates concurrently with the correlation-based and feature-based trackers to generate a list of possible targets. The list can be used to provide a confidence level for the probable targets identified by each of the primary and secondary trackers.