The present invention introduces a system and method for improved robustness in a speech system. In one embodiment, a method comprises receiving an utterance from an intended talker at a speech recognition system; computing a speaker verification score with a voice characteristic model associated and with the utterance; computing a speech recognition score associated with the utterance; and selecting a best hypothesis associated with the utterance and based on both the speaker verification score and the speech recognition score.