Rating information retrieval algorithms. A query is received and submitted to a search engine for execution on an index file. A list of index documents and a plurality of attributes are received from the search engine in response to the submitted query. A portion of the received list of index documents and the received plurality of attributes are stored in a subindex file. The received query is executed on the subindex file to obtain a list of subindex documents. The obtained list of subindex documents is ranked by a particular ranking algorithm using the attributes associated therewith. The list of ranked subindex documents is compared with a list of subjectively ranked documents to generate a score for the particular ranking algorithm. The generated score represents a degree of correlation between the list of ranked subindex documents and the list of subjectively ranked documents.