In a system and method verifying an identity of a speaker, feature vectors are received that were extracted from an utterance made by a speaker claiming an identity. Dissimilarity may be measured between the feature vectors and a codebook associated with a version of the utterance known to be made by the identity. The utterance may be further analyzed to ascertain information about repeating occurrences of the feature vectors in the utterance. The information about repeating occurrences of feature vectors occurring in the utterance may be compared to information about repeating occurrences of feature vectors in a version of the utterance known to be made by the claimed identity. Based on the comparison, a penalty may be assigned to the measured dissimilarity. Using the measured dissimilarity modified by the assigned penalty, a determination may be made as to whether to accept the speaker as the identity.