The present image processing apparatus includes a pixel arrangement controller and two data unit buffers coupled with the pixel arrangement controller. This controller can rearrange the addresses of the pixels in the corresponding data unit buffer according to the size of the block. Accordingly, the pixel data of the same block can be arranged in sequential addresses of the data unit buffer. Therefore, the pixel data may be processed as a batch.