Described is the differentiation of replicas in a large distributed object
store as either being smoothing replicas based on an amount of load on
storage nodes (bricks), or as spreading replicas based on a substantially
random distribution among the system's bricks. The smoothing replicas are
placed among the lowest usage bricks, while the spreading replicas are
placed randomly throughout other bricks in the system independent of
load. As a result, fast, primarily parallel data repair is facilitated by
selecting a spreading replica when repair is needed, while load balancing
is facilitated by placing a smoothing replica on a low-usage brick when a
new replica is checked in, and selecting a smoothing replica when load
balancing is triggered by overloading of a brick and/or addition of a new
brick. Check-in, data repair and load balancing policies specify how to
use smoothing replicas and spreading replicas.