The present invention provides an image detection system capable of picking up a high resolution image of the surface condition of a circuit pattern-formed wafer without being affected by steep pattern steps, discontinuous reflectance distributions and optically transparent substances which are formed after resist patterns are formed and removed. A defect detection apparatus using such an image detection apparatus is also provided by the invention. The present invention is implemented by a configuration comprising a scanning stage having a sample mounted thereon, an image pickup system for picking up a surface image of the sample, height detection means for detecting the surface height of the sample at plural points including two points which are respectively on the opposite sides of the image pickup position in the scanning direction, sample height calculation means for calculating the height of the sample at the image pickup position by using the detected heights and focusing means for focusing the image pickup system by using the calculated sample height.