One embodiment of the present invention provides a system that converts a non-bandlimited pattern layout into a band-limited pattern image to facilitate simulating an optical lithography process. During operation, the system receives the non-bandlimited pattern layout which comprises one or more polygons. The system further receives an anti-aliasing filter (AAF) kernel, wherein the AAF kernel is configured to convert a non-bandlimited pattern into a band-limited pattern. The system then constructs an AAF lookup table for the AAF kernel, wherein the AAF lookup table contains precomputed values for a set of convolution functions which are obtained by convolving a set of basis functions with the AAF kernel. Next, the system creates a sampled pattern layout by applying a grid map over the pattern layout. The system then obtains the band-limited pattern image by using the AAF lookup table to convolve the AAF kernel with each grid location in the sampled pattern layout.