A system for calculating mask data to create a desired layout pattern on a wafer reads all or a portion of a desired layout pattern. Mask data having pixels with transmission characteristics is defined along with corresponding optimal mask data pixel transmission characteristics. An objective function is defined having one or more penalty functions that promote solutions representing a desired resolution enhancement technique. Optimization of the objective function determines the transmission characteristics of the pixels in the mask data that is used to create one or more corresponding masks or reticles.