A system for maximal gathering of fresh information added to a network
such as the as the Internet and for processing the gathered fresh
information. A link server (2) sends a batch of links to check (3) to a
crawler (1B). Crawler (1B) them executes its crawling assignment by
filtering the encountered content and extracting only that which is new
or changed (4). Crawler (1B) then returns this content (4) to at least
one data center and any interested web mining application (5). By using
the crawlers (1A-E) to filter the data and only return or notify
regarding, the fresh content, less bandwidth is needed to get the
information to the web mining application (5).