Described is an active learning system for fingerprinting an object identified in an image frame. The active learning system comprises a flow-based object segmentation module for segmenting a potential object candidate from a video sequence, a fixed-basis function decomposition module using Haar wavelets to extract a relevant feature set from the potential object candidate, a static classifier for initial classification of the potential object candidate, an incremental learning module for predicting a general class of the potential object candidate, an oriented localized filter module to extract features from the potential object candidate, and a learning-feature graph-fingerprinting module configured to receive the features and build a fingerprint of the object for tracking the object.