A video input processor is provided to process different input video format, including RGB, RGB Bayer, YUV 4:2:2 interlaced and progressive video data. The video input processor also uses an advanced algorithm to efficiently convert video data in RGB color space to YUV color space. The video input processor further enables multi-functions such as video image scaling, video image filtering before the video data are output for further video compression.