A video encoding scheme employs progressive fine-granularity layered coding to
encode video data frames into multiple layers, including a base layer of comparatively
low quality video and multiple enhancement layers of increasingly higher quality
video. Some of the enhancement layers in a current frame are predicted from at
least one lower quality layer in a reference frame, whereby the lower quality layer
is not necessarily the base layer.