A Web crawler data collection method is provided for collecting information associated with a plurality of queries, which is used to calculate estimates of return probabilities, clicking probabilities and incorrect response probabilities. The estimated return probabilities relate to a probability that a search engine will return a particular Web page in a particular position of a particular query result page. The estimated clicking probabilities relate to a frequency with which a client selects a returned Web page in a particular position of a particular query result. The estimated incorrect response probabilities relate to the probability that a query to a stale version of a particular Web page yields an incorrect or vacuous response. Further, information may be collected regarding the characteristics and update time distributions of a plurality of Web pages.