Techniques for automatically checking the relevance and variety of search results are provided. A query is submitted to a search engine, which uses a search algorithm to obtain search results based on the query. A set of the top n related terms for the query is identified. For each related term in the set of terms, its relative frequency in relation to all terms in the set of terms is determined. If the term does not occur in any of the results, then a loss in variety proportional to the relative term frequency for the term has occurred. Otherwise, the relevance of the search results is calculated by comparing the proportion of results containing the term with the relative term frequency for a term. This process is repeated for all terms in the set of related terms to produce a total variety and relevance for the results.