A method decomposes multiview video acquired of a scene by multiple cameras. Each multiview video includes a sequence of frames, and each camera provides a different view of the scene. A prediction mode is selected from a temporal prediction mode, a spatial prediction mode, and a view interpolation prediction mode. The multiview videos are then decomposed into low band frames, high band frames, and side information according to the selected prediction mode.