A method for energy based, non-uniform time-scale compression of audio signals includes receiving a frame of data corresponding to an input audio signal and segmenting the data into a plurality of segments. The method further includes estimating a value related to energy of the frame of data, determining a peak energy estimate for the frame, determining an energy threshold based on the peak energy estimate of the frame and comparing the value related to energy of the frame of the data with the energy threshold to control time-scale compression of the audio data.