A method for balancing the load of a parallel processing system having a plurality of parallel processing elements arranged in a loop, each processing element (PE.sub.r) having a local number of tasks associated therewith, wherein r represents the number for a selected processing element and each of the processing elements is operable to communicate with a clockwise and an anti-clockwise adjacent processing element, the method comprises determining a total number of tasks present within the loop, calculating a local mean number of tasks for each processing element, calculating a local deviation for each processing element, determining a running partial deviation sum for each processing element, determining a clockwise transfer parameter and an anti-clockwise transfer parameter for each processing element, and redistributing tasks among the processing elements in response to the clockwise transfer parameter and the anti-clockwise parameter for each of the processing elements.