A video summary description system for describing video summary by meta data to provide overview functionality that makes it feasible to understand the overall contents of the original video within a short time with navigation and browsing functionalities, and that makes it feasible to search the desired video contents efficiently. The system includes a HierarchicalSummary Description Scheme (DS) for describing a video summary that includes a HighlightLevel DS having at least one HighlightSegment DS. The HighlightSegment DS describes information on a highlight segment corresponding to one of the video summary intervals, and the HighlightSegment DS includes a VideoSegmentLocator DS for describing time information or the video itself of the highlight segment and an ImageLocator DS describing a representative frame of the highlight segment.