A first semiconductor layer is epitaxially grown on a semiconductor substrate and patterned to form concave and convex portions. A second semiconductor layer is formed on the first semiconductor layer using a top epitaxial mask covering the top surface of the convex portion. Lattice defects D propagating from the first semiconductor layer exist only in a region located above the center of the concave portion (a defect region Ra), while in the other region (a low defect region Rb) lattice defects D propagating from the first semiconductor layer hardly exist.