A video segmentation system generates an S-distance measurement that is a representation of the similarity between adjacent frames of a video sequence. The video segmentation system employs frequency decomposition of a direct current (DC) luminance signal of a compressed video sequence. High and low frequency component signatures are generated from a frequency-decomposed signal using wavelet transformation. A cut detector identifies cut transitions from the low frequency component signature. A fade detector identifies fade transitions the high frequency component signature. A dissolve transition detector employs a double frame differencing algorithm to identify dissolve transitions. A video retrieval system likewise generates an S-distance between a query image and a database image. The video retrieval system employs the low and high frequency component signature to generate the S-distance measurement of the similarity between the query image and the database image. The results of the S-distance measurement allow browsing and searching of the similar database images.