Large payload files are selectively partitioned in blocks and the blocks distributed
to a plurality of distribution stations at the edge of the network qualified to
have the data. Each qualified station decides how much and what portion of the
content to save locally, based on information such as network location and environment,
usage, popularity, and other distribution criteria defined by the content provider.
Different pieces of a large payload file may be available from different nodes,
however, when a user requests access to the large payload file, for example, through
an application server, a virtual file control system creates an illusion that the
entire file is present at the connected node. However, since only selective portions
of the large payload file may actually be resident at that node's storage at the
time of request, a cluster of distribution servers at the distribution station
may download the non-resident portions of the file as the application server is
servicing the user. The download may be in parallel and usually from the least
congested nodes. New nodes added to the network learn from other nodes in the network
what content they should have and download the required content, in a desired amount,
onto their local storage devices from the nearest and least congested nodes without
interrupting network operation. Each node manages its local storage and decides
what content to prune based on information such as usage patterns.