An adaptive compensation system for aspect ratio conversion. Video information, intended for her first aspect ratio display, e.g. the standard NTSC aspect ratio, is processed to determine additional information that can enable that video to be displayed and fill a wider aspect ratio screen. The processing can be a calculation which calculates, for example, texture, color and/or brightness of the edge portions, and automatically calculates video information to match the texture, brightness or color. The processing can be a database lookup, which automatically looks up likely portions from a database. The processing can also be an adaptive determination of what vertical portions of the image can be stretched without affecting the viewability, and then an adaptive stretching of different portions by different amounts.