The techniques in the present invention allow both text and handwritten queries, and the queries can be single-word or multiword. Generally, each handwritten word in a handwritten document is converted to a document stack of words, where each document stack contains a list of text words and a word score of some type for each text word in the list. The query is also converted to one or more stacks of words. A measure is determined from each query and document stack. Documents that meet search criteria in the query are then selected based on the query and the values of the measures. The present invention also performs multiple recognitions, with multiple recognizers, on a handwritten document to create multiple recognized transcriptions of the document. The multiple transcriptions are used for document retrieval. In another embodiment, a single transcription is created from the multiple transcriptions, and the single transcription is used for document retrieval.