A moving picture encoding decoding processing apparatus for detecting a motion vector by forming the image data of the search area for motion vector with small capacity of memory rapidly and efficiently. A moving picture encoding decoding processing apparatus detects the macroblock number by a macroblock counter. A macroblock recognition part recognizes whether the search area for the motion vector of the macroblock is extending over the frame image area or not based on the position of the macroblock in the frame image. A macroblock recognition part generates an extrapolation pattern signal which indicates the pattern of the extrapolation for the inputted macroblock. If the whole search area is within the frame area, image data formed on a frame memory 2 are used. If the search area extends beyond the frame area, image data formed on a prediction memory 3 are used. A control unit 1 includes an image data transmission means. Image data of the current processing macroblock data and surrounding data existing on the frame memory are transferred from the frame memory 2 to the prediction memory 3. The prediction memory 3 includes an image data extrapolation means. The image data extrapolation means performs the extrapolation by specifying row address and column address necessary for the extrapolation based on the transmitted image data from the frame memory and the extrapolation pattern signal.