A method and system for segmenting video into an optimal set of video clusters that correspond loosely to one or more events, date ranges, time ranges, chapters or other logical segmentations of the digital video that is meaningful to the user. Video shots are selected from one or more video files on a computer-readable medium. The video shots are arranged in chronological order and a distance (e.g., time gap) between each successive pair of the arranged video shots is determined. Video clustering are generated as function of the determined `distances` and/or user input, and an optimal video clustering is identified. After the optimal video clustering is identified, a time span for each of the video clusters is determined and a label is generated for each cluster. The clusters are then displayed for the user to browse using the generated labels and thumbnails derived from the video clips, transferred to a separate medium for display on another computer or other electronic device (e.g. DVD-video player) or made available for the user or other software to manipulate further.