According to some embodiments, a Single-Instruction/Multiple-Data (SIMD) averaging instruction is used to process pixels of image data. The averaging instruction generates a set of four-pixel averages, where each average is generated from two pixels in a first source register and two pixels in a second source register. The first source register contains a plurality of pixels from a first row of pixels and the second source register contains a plurality of pixels from a second row. In one embodiment, the first and second rows are adjacent rows in an image and the averaging instruction is used, for example, to down-scale an image, perform color conversion, and the like. In another embodiment, the first and second rows are from different images and the averaging instruction is used, for example, in motion estimation for video encoding, in motion compensation for video decoding, and the like.