A speech recognition system includes a user profile to store acoustic data and a corresponding text transcript. A speech recognition ("SR") server downloads the acoustic data and the corresponding text transcript that are stored in the user profile. A speech recognition engine is included to adapt an acoustic model based on the acoustic data.