Image processing enhances display quality by controlling both the image modulator and the light sources.The image processing system utilizes a combination of user inputs, system configuration, design information, sensor feedback, pixel modulation information, and lighting control information to characterize the display environment on a per pixel basis. This per pixel characterization information is combined with one or more frames of the incoming real time display data. The image processing system processes each incoming pixel and produces a modified corresponding output pixel. Each pixel of each frame is processed accordingly.The image processing system also produces control information for the light sources. The light source control information can control individual lamps, tubes or LEDs, or can control a block or subset of the light sources. The resulting display has improved image consistency, enhanced color gamut, higher dynamic range and is better able to portray high motion content.