A multipoint controller unit in which the audio signal processing is distributed among several audio ports. The control of the audio signal processing may be centrally or distributively managed by one or more audio controllers. The audio signal processing may include selecting audio streams from a common interface and/or mixing the signals from the common interface. Each audio port may enhance the audio signal and generate control information from the audio signal. The audio ports are synchronized by a common information channel while the audio controllers are synchronized by a common control channel interface. Port fragmentation is reduced since the audio signal processing that relates to an endpoint within a conference is handled by a single audio port.