An image data compression device including: a quantization section which quantizes image data; a FIFO buffer section which buffers quantized data for N frames (N is an integer greater than one) from the quantization section; an encoded data generation section which reads the quantized data from the FIFO buffer section asynchronously from writing into the FIFO buffer section, and generates encoded data; and a rate control section which changes a quantization step of the quantization section by using the data size of the encoded data. The rate control section calculates a quantization parameter to be equal to or less than an upper threshold of the quantization parameter or to be equal to or greater than a lower threshold of the quantization parameter by using an average data size of the encoded data for preceding N frames.