Disclosed are a method of constructing information on associate meanings between segments of a multimedia stream, which can describe the cause/effect or abstract/detail relationship between segments of the video streams to efficiently browse the video stream and a method of browsing a video using the same. The present invention defines the cause/effect or abstract/detail relationship between the segments, event intervals, scenes, shots, etc. existing within one video stream or between the video streams, and provides a method of describing the relationship in a data region based on the content of a video stream as well as a method of browsing a video by using the information on the cause/effect or abstract/detail relationship obtained by the aforementioned method. Accordingly, a video browsing on associate meanings is available with easy manipulation and easy access to a desired part, thereby providing an effective video browsing interface for easy browsing of desired segments in a short period of time.