A video signal processing apparatus for processing a video signal containing first and second color difference signals so as to adjust colors of an image represented by the video signal is disclosed. The apparatus includes a plurality of gain setting sections for setting respective ones of gain conversion characteristics. The gain conversion characteristics are relevant to saturations indicated by color difference signals. First and second conversion sections are included for converting gains of the color difference signals based on input values of respective color difference signals and the pain conversion characteristics.