Segmenting encoding system with image segmentation performed at a decoder and encoding scheme for generating encoded data relying on decoder segmentation

An encoder includes a segmenter that segments a reference frame, assigning some or all of the pixels of the reference frame to segments based on pixel color values and/or pixel location. A kinetic information generator generates kinetic information that relates regions of a nonkey frame to segments of the reference frame as indicated by the segmentation. A decoder includes its own segmenter that segments at least a part of the reference frame extracted from the compressed video data, thereby eliminating the need for the encoder to include all of the segmentation in the compressed video data. The decoder includes a nonkey frame generator that regenerates a nonkey frame using kinetic information about the nonkey frame from the compressed video data and using at least a part of the segmentation of the reference frame generated by the decoder's segmenter. The encoder segmentation and decoder segmentation can be the same or different.