A user interface (UI) for adaptive video fast forward provides a novel fully adaptive content-based UI for allowing user interaction with an image sequence or video relative to a user identified query sample. This query sample is drawn either from an image sequence being searched or from another image sequence entirely. The user interaction offered by the UI includes providing a user with computationally efficient searching, browsing and retrieval of one or more objects, frames or sequences of interest in video or image sequences, as well as automatic content-based variable-speed playback based on a computed similarity to the query sample. In addition, the UI also provides the capability to search for image frames or sequences that are dissimilar to the query sample, thereby allowing the user to quickly locate unusual or different activity within an image sequence.