A device for compensating for a mismatch of a transmitter and a method of compensating for the mismatch is provided. The device for compensating for a mismatch of an up-converter in a direct conversion transmitter, comprises: a mismatch compensating unit for compensating for an input signal using a first mismatch compensation value to generate a mismatch compensation signal; a down-converter for down-converting a radio frequency (RF) signal to generate an intermediate frequency (IF) signal, the RF signal being generated by up-converting the mismatch compensation signal; a square unit for squaring the IF signal; and a controller for calibrating the first mismatch compensation value to determine a second mismatch compensation value such that a direct current (DC) component of the squared IF signal has a minimum value to compensate for the mismatch of the up-converter using the second mismatch compensation value.