A method and apparatus is provided for automatically classifying a defect on the surface of a semiconductor wafer into one of, e.g., seven core classes: a missing pattern on the surface, an extra pattern on the surface, a deformed pattern on the surface, a particle on the surface, a particle embedded in the surface, a particle and a deformed pattern on the surface, or craters and microscratches on the surface. The defect may also be further classified into a subclass of arbitrarily defined defects defined by the user or preprogrammed in the apparatus. Embodiments include using a scanning electron microscope (SEM) capable of collecting electrons emitted from a plurality of angular sectors to obtain an image of the defect and a reference image containing topographical and location information, then analyzing this information to classify the defect. As the defects are classified, counts are maintained of the number of occurrences of each type of defect, and an alarm is raised if the defect count in a particular class exceeds a predetermined level. Thus, defects are accurately and reliably classified and monitored to enable early detection and cure of processing problems.