A method and apparatus to reduce the system load of motion estimation for DSP discloses circular buffers, a plurality of absolute difference calculation circuits, a multiple input adder, a full adder, a plurality of accumulators, and a control circuit. The first four bytes from the reference block buffer and the first four bytes from the search window buffer are sent to the four absolute difference calculation circuits. The control circuit determines which of the accumulators requires incrementing the value already in that accumulator by the current output of the multiple input adder. A new set of bytes from the search window buffer is then sent to the absolute difference calculation circuits, a new sum is calculated, and a second accumulator is incremented by the new sum. When all accumulators have been updated, new reference block data used. Each byte of data is loaded from memory only once.