A system and a method are disclosed for adaptive probabilistic tracking of an object within a motion video. The method utilizes a time-varying Eigenbasis and dynamic, observation and inference models. The Eigenbasis serves as a model of the target object. The dynamic model represents the motion of the object and defines possible locations of the target based upon previous locations. The observation model provides a measure of the distance of an observation of the object relative to the current Eigenbasis. The inference model predicts the most likely location of the object based upon past and present observations. The method is effective with or without training samples. A computer-based system provides a means for implementing the method. The effectiveness of the system and method are demonstrated through simulation.