A system (20) for decoding a data stream allocated into data packets contains a control unit (54), a stream demultiplexer (26), audio and video decoders (38 and 40), a memory management unit (60), and audio and video input and output buffers. Upon demultiplexing and depacketizing the data packets without interrupting the control unit, the demultiplexer sends encoded audio and video data to the audio and video input buffers. Video messages dealing with video timing information and identifying where encoded video data is stored in the video input buffer are furnished by the demultiplexer for use by the control unit. Utilizing corresponding video instructions provided from the control unit, the video decoder decodes encoded video data to produce decoded video data supplied to the video output buffer. The audio decoder decodes encoded audio data to produce decoded audio data supplied to the audio output buffer. The memory management unit controls transfer of decoded audio data to and from the audio output buffer.