Systems and methods are provided for recognizing handwritten characters drawn on a paper form using a digital pen that records stroke coordinates corresponding to respective pen strokes. In one embodiment, a field on the paper form is assigned a lexical inference level. For example, the field may be assigned a word level, a word prefix level, and/or a word stem level. The assigned lexical inference level is used to recognize one or more stroke coordinates corresponding to pen strokes written in the field. Recognized characters are then used to create or modify an inference lexicon used to perform handwriting recognition for the entire field. In one embodiment, the inference lexicon is used for handwriting recognition in the same field on subsequently processed forms.