Progressive scan encoded JPEGS are decoded sequentially on a Minimum Coded Unit (MCU) basis and then the JPEG image is output one row at a time to a frame buffer. Since the entire image does not need to be decoded at once, a huge JPEG buffer is generally not required. The present invention reconstructs each MCU one scan at a time to produce complete lines of image data and thus provides an output using the image data on a line-by-line basis. This technique may require that each MCU be reconstructed scan-by-scan. This technique provides a modified entropy (Huffman) decoder in which data for different scans of an MCU of the progressively scanned JPEG image may be stored in a buffer, and a parsing and scan table element creates pointers to the memory buffer corresponding to the start of each scan level. Scan tables are also extracted for each level of scan or points to the same tables if some scan levels use the same table.