A system and method for detecting features in video from business meetings and video teleconferencing systems can use color, motion, morphological (shape) filtering and a set of heuristic rules to determine possible features in the scene. One embodiment includes obtaining a first video frame of images and a second video frame of images, performing a luminance conversion on the first video frame of images and the second video frame of images, performing a motion filter operation on the converted first and second frames of images, performing a chrominance normalization on the second video frame of images, performing a color filter operation on the normalized second video frame of images, and processing the first and second frames of video images after the motion filter and color filter operations.