A search mechanism is evaluated or a relevance model checked using session level and result level diagnostics based on user behavior during a search session with respect to queries entered and user responses to result lists. Tracking occurs of when content desired by a user exists, but is not returned in a search result list when a query is made by the user with intent to find the desired content, when content desired by the user does not exist, when content desired by a user exists, but is not recognized by the user in a result list or is too low in a result list. A user's intent and search context is also taken into consideration when performing search mechanism diagnostics.