A method (500, 600) of tracking a feature (201) across a sequence of image frames is disclosed. A current feature position in a current frame is estimated from at least a previous feature position in a previous frame. Feature data is extracted from pixel data of the current frame. The feature data is then compared with reference data. If a difference between the feature data and the reference data is larger than a predetermined number, then track of the feature (201) has been lost. Alternatively, the current feature position is appended to the trajectory of the feature, and the reference data is updated periodically with feature data of a plurality of frames. Preferably the reference data is a statistical representation of feature data of the plurality of frames.