An image decoding unit comprises a decoding unit, a data memory unit, a reconstruction unit, and a frame memory. The decoding unit includes an entropy decoder, a motion compensator, an inverse quantizer, and an inverse DCT unit. The data memory unit includes a data memory A and a data memory B. In the middle of the data transfer from the data memory unit to the frame memory, the reconstruction unit that inputs intermediate data of decoding and outputs reconstructed image data is provided; thereby, the processing of generating the reconstructed image data and the processing of storing the reconstructed image data into the frame memory can be performed in parallel. By the structure, a high-speed processing of image reconstruction can be performed.