The present invention includes a key frame extraction system and method for extracting key frames from a video based on motion analysis of frames within the video. Key frames are highlight frames that are effective in summarizing a video sequence. This allows a user to quickly find a desired spot in a video is long and contains differing subject matter. The key frame extraction system and method uses a triangle model of the motion energy in each frame and extracts key frames based on this model. More specifically, motion analysis is performed on the video frames in order to identify motion acceleration and motion deceleration points within the frames. A triangle model of motion then is constructed based on results of the motion analysis. The apex of the triangle represents a turning point between motion acceleration and motion deceleration. Frames corresponding to this apex are selected as key frames.