In certain aspects, the invention features a system and method for caching results,
including receiving a job for computation by a distributed computing system having
one or more node computing devices in communication with a cache, processing, on
one of the node computing devices, the job to create an intermediate result for
storage in the cache, wherein the intermediate result includes data wherein a time
required to obtain the data by computation or retrieval from a data storage external
to the distributed computer system is at least marginally greater than that of
retrieving the intermediate result from the cache. In accordance with such aspects,
the system and method further includes storing the intermediate result in the cache,
and accessing the cache by presenting a lookup function to the cache, wherein the
lookup function includes a key and a compute function configured to produce the
intermediate result.
