An image data compression device includes: a quantization section which quantizes image data based on a quantization parameter; a FIFO buffer section in which quantized data for at least N frames (N is an integer larger than one) is buffered; an encoded data generation section which reads the quantized data from the FIFO buffer section, and generates encoded data; a rate control section which changes a quantization step by using a data size of the encoded data; and a frame skip section which skips a quantization processing of the quantization section. The rate control section calculates the quantization parameter by using an average data size of the encoded data for previous N frames, and the frame skip section performs the skip processing when a frame in which the quantization parameter is greater than a threshold continues equal to or more than a number of times for a number threshold.