Large payload files are selectively partitioned in blocks and the blocks
distributed to a plurality of distribution stations at the edge of the
network qualified to have the data. Each qualified station decides how
much and what portion of the content to save locally, based on
information such as network location and environment, usage, popularity,
and other distribution criteria defined by the content provider.
Different pieces of a large payload file may be available from different
nodes, however, when a user requests access to the large payload file,
for example, through an application server, a virtual file control system
creates an illusion that the entire file is present at the connected
node. However, since only selective portions of the large payload file
may actually be resident at that node's storage at the time of request, a
cluster of distribution servers at the distribution station may download
the non-resident portions of the file as the application server is
servicing the user. The download may be in parallel and usually from the
least congested nodes. New nodes added to the network learn from other
nodes in the network what content they should have and download the
required content, in a desired amount, onto their local storage devices
from the nearest and least congested nodes without interrupting network
operation. Each node manages its local storage and decides what content
to prune based on information such as usage patterns.