A buffer memory has a plurality of blocks which are virtually broken up into a matrix pattern. An image processing apparatus has a memory controller for writing R, G, and B image data in the buffer memory in units of lines and in a predetermined color order, and reading out the R, G, and B image data written in the buffer memory. The memory controller switches operation for writing the R, G, and B image data in the blocks that belong to corresponding rows, and operation for writing the R, G, and B image data in the blocks that belong to corresponding columns in units of lines, and reads out old image data from a region of the buffer memory in which new image data are expected to be overwritten before the new image data are overwritten by the write operation.