An arrangement for processing video signals comprises at least one linear filter and at least one non-linear filter with inputs for an input signal and selection means for receiving output signals of the at least one linear and of the at least one non-linear filter and for generating a selected output signal which corresponds to a selected one of the output signals whereby the selection is performed according to a predetermined rank order. The arrangement can be used in various video signal processing applications, especially image restoration or noise reduction, image or pattern recognition, image compression, image enhancement (contour sharpening or softening), image interpolation (up or down sampling), and flicker compensation.