The people a user is presumed to know or be associated with may be determined using a number of techniques. For example, people a user knows may be inferred based on a combination of two or more user actions, each of which separately support an inference that the person is associated with the user. This information about people that the user knows is used in relation to the user's communications.