A method of inspecting defects of a plurality of patterns that are formed on a substrate to have naturally the same shape. According to this method, in order to detect very small defects of the patterns with high sensitivity without being affected by irregular brightness due to the thickness difference between the patterns formed on a semiconductor wafer, a first pattern being inspected is detected to produce a first image of the first pattern, the first image is stored, a second pattern being inspected is detected to produce a second image of said second pattern, the stored first image and the second image are matched in brightness, and the brightness-matched first and second images are compared with each other so that the patterns can be inspected.