A noise estimation system that estimates noises in a sequence of video frames, uses a an image structure remover and a noise variance calculator. The image structure remover computes the difference over a local window in two consecutive frames. The noise variance calculator estimates the noise variance (standard deviation) from the distribution of the local difference. If there is no motion or small motion between two consecutive frames, the image structure can be removed by calculating the local difference, whereby very robust estimation is obtained.