Disclosed are methods and apparatus for classifying defects based on X-ray spectrum obtained from the defects. In general terms, the present invention provides pattern recognition techniques for accurately and efficiently classifying a defect based on an X-ray spectrum obtained from such defect and its surrounding substrate and structures, no matter the complexity of such substrate and structures. A pattern recognition technique generally includes training a pattern recognition process to recognize particular types of X-ray spectrum obtained from specimens as belonging to a particular defect type or other specific characteristic of a specimen. Once a pattern recognition process is set up to recognize or classify particular X-ray spectrum, the pattern recognition process can then be utilized to automatically classify specimens as having a specific characteristic or defect type.