The present invention provides a scheme for compressing the color components of image data. The pixel data is grouped into a plurality of tiles for storage. A test is performed to determine if a tile can be compressed so that its size after compression is less than its size before compression. If so, the tile is compressed. A tile table is provided that includes a flag that can be set for each tile that is compressed. In a data transfer from memory to a graphics processor, the tile table is examined to identify those tiles that are compressed and must be decompressed prior to use. In one embodiment, a number of compression schemes are available for use on a tile and the best compression scheme is chosen on a tile by tile basis. The invention includes an identifying code for each compression scheme (stored as a value in each compressed tile).