A method and computer-readable medium are provided for constructing a classifier for classifying search queries. The classifier is constructed by receiving a query from a user and applying the query to a classifier to identify the task. An unsupervised mapping between the query and the task is then identified and is used to train a new classifier. Under one embodiment, the unsupervised mapping is identified based on a user's selection of the task.