This invention improves the arithmetic precision even for image data in which each component of one pixel is expressed by 8 bits by fully utilizing the 12-bit data processing performance of an Extended sequential DCT-based JPEG decoding/encoding apparatus, so that image deterioration due to JPEG compression which is observed in an image portion where the gray levels change slowly, i.e., a pseudo edge can be hardly generated, thus improving the image quality. To this end, a header interpreter interprets the header of encoded data to be decoded to determine if the encoded data is that of 8-bit image data per component or that of 12-bit image data per component, and outputs the result to a bit shift unit, rounding processor, and inverse quantizer. Upon reception of a message indicating that the encoded data is that of 8-bit color image data per component, the bit shift unit stores a result obtained by shifting a quantization step of a quantization table stored in the header by 4 bits in the MSB direction in a quantization table storage unit. The inverse quantizer multiplies a decoded value decoded by a Huffman decoder by the quantization step stored in the quantization table storage unit to generate 12-bit DCT coefficients. The DCT coefficients are processed by an IDCT unit to obtain image data, and the image data is rounded to 8 bits again, thus outputting image data as a decoded result.