In a representative embodiment, a transmitter of a WDM communication system is equipped with a Mach-Zehnder modulator (MZM) configured to produce an optical signal corresponding to a duobinary data stream by modulating a beam of light passing through the MZM. A three-level electrical signal is applied to the MZM to modulate the light. At the second signal level, the modulator substantially blocks light transmission. At the first and third levels, the modulator transmits light corresponding to the duobinary "-1" and "+1" bits, respectively, such that a relative phase shift between the optical bits is different than 180 degrees, i.e., the phase shift value employed in prior-art optical duobinary modulation. Advantageously, by appropriately selecting voltages for the first and third levels, the relative phase shift between the optical bits can be adjusted to reduce detrimental effects of optical bandpass filtering typically present in the WDM communication system, thereby reducing the number of decoding errors at the receiver.