A system, method, and apparatus are directed towards managing a Voice over IP (VOIP) messages over a network, employing the Real-time Transport Protocol (RTP) and Session Initiation Protocol (SIP) over the Transmission Control Protocol (TCP). The VOIP messages are sent by a source device to a destination device through a relay server. The relay server may throttle the VOIP messages employing buffer management. When the buffer is substantially full, the relay server will drop packets from the source device. Indication of the lost packets may be provided to the source device through a Real-time Transport Control Protocol (RTCP) report. The source device may then employ the RTCP report to modify a type of codec employed, and thereby adjust a rate of flow of VOIP packets sent towards the destination device. Additionally, the relay server may provide port translation services for RTP/RTCP packets between the source and destination devices.