A method (100) of tracking facial features in a video sequence is disclosed. The method receives (110) as input facial features for tracking in a first frame of the video sequence. The video sequence is also segmented (120) using spatiotemporal segmentation to provide a sequence of associated two-dimensional segments, with a first segment in the sequence of associated two-dimensional segments including the facial features for tracking. Candidate facial features are then identified (140) in at least a second two-dimensional segment in the sequence of associated segments. Method (100) then verifies (155) which of the candidate facial features correspond with the facial features for tracking. Method (100) also recovers lost facial features by using known geometric relations between facial features.