Data throughput rates are increased in an optical fiber communication system without requiring replacement of the existing optical fiber in a link. Channel throughput is increased by upgrading the components and circuitry in the head and terminal of an optical fiber communication system link. Aggregate throughput in a fiber optic link is increased beyond the range of conventional Wavelength Division Multiplexed (WDM) upgrades, while precluding the necessity of replacing existing fiber plants. The increase in system throughput is achieved by using advanced modulation techniques to encode greater amounts of data into the transmitted spectrum of a channel, thereby increasing the spectral efficiency of each channel. This novel method of increasing transmission capacity by upgrading the head and terminal of the system to achieve greater spectral efficiency and hence throughput, alleviates the need to replace existing fiber plants. Spectrally efficient complex modulation techniques can be supported by interface circuits with an increased level of signal processing capability in order to both encode multiple bits into a transmitted symbol and decode the original data from the received symbols.