A method and apparatus for directly connecting very large data streams from
an archive command into a backup data system using an "intelligent
process." An output stream-is piped into an intelligent pipe-reading
process and distributed over a set of temporary data stores built from raw
storage resources. A pipe interface process supervises backup of each
filled data store, while the remaining output stream continues to be piped
into another available data store. The backup system completes archiving
of the datastream, keeping a catalog of the datastream storage locations.
To retrieve the data, the intelligent process is run in reverse as a
pipe-writing process, requesting data from the backup system. Retrieved
data traverses the data stores from the backup system and are pumped into
the pipe-writing process for delivery to the pipe output file identified
by the retrieve or import command.