A voice-mail annotation system and method that enables callers to annotate the messages they leave at the time they leave them. Annotation may be performed using a touch screen or a telephone key-pad. In addition to allowing the caller to more clearly specify the intent of their message and the location of crucial information within the message, the annotations of this invention then allow the person retrieving the message flexibility in how they listen to the message. These choices include prioritizing the messages by the importance of subject, listening to summaries of the message and having random access to embedded information such as names and call back numbers.