The present invention provides a method for using stored data associated with simple actions to identify and classify an observed composite action which comprises a combination of simple actions. Image data comprising the composite action is received and used to generate a feature vector, such as a combination of a horizontal translation, a vertical translation, a scale value and a skin detection value, that is associated with the composite action. Baseline reference vectors representing combinations of simple actions represented by stored reference data are then generated. The generated feature vector is then preprocessed to modify the feature vector towards a baseline reference vector describing a combination of simple actions. Each of the baseline reference vectors is then compared to the modified feature vector to determine the baseline reference vector and corresponding combination of simple actions that most accurately describes the composite action.