The system includes a jitter buffer for receiving speech packets in a Voice over Internet Protocol (VoIP) system, a playback device for adjusting the playback speed of the received speed packets, and a jitter buffer manager for detecting out of sequence packets in the jitter buffer and for sending commands to the playback device to adjust playback speed based on the detection. The speech signal is played back at the nominal speed when there are no out of sequence packets. The playback speed is decreased when an out of sequence packet is detected, thereby tending to increase the jitter buffer length. When an out of sequence packet arrives, the playback speed is increased in order to restore jitter buffer length to its nominal length.