First, a pattern inspection apparatus detects the first edge from an image of a pattern to-be-inspected. Next, the pattern inspection apparatus conducts matching of the image of the pattern to-be-inspected and the first reference pattern by comparing the first edge and an edge of the first reference pattern. Since, as a result of the matching, a shift quantity S.sub.1 can be obtained, and then the first reference pattern is shifted by this shift quantity S.sub.1. Subsequently the pattern to-be-inspected is inspected by comparing the first edge and the edge of the first reference pattern so shifted. In this first inspection, pattern deformation quantities are obtained and defects are detected. A shift quantity S.sub.2 can be obtained as one of the pattern deformation quantities. Next, in order to detect the second edge from the pattern image to-be-inspected, the corresponding second reference pattern is shifted by a shift quantity S.sub.1+S.sub.2. Using the second reference pattern so shifted, a profile is obtained on the pattern image to-be-inspected and the second edge is detected. Then, by comparing the second edge and the edge of the second reference pattern so shifted, the pattern to-be-inspected is inspected. Also in this second inspection, the pattern deformation quantities are obtained and defects are detected. A shift quantity S.sub.3 can be obtained as one of the pattern deformation quantities.