According to the present invention, an impurity region, to which a rare gas element (also called a rare gas) and one kind or a plurality of kinds of elements selected from the group consisting of H, H.sub.2, O, O.sub.2, and P are added, are formed in a semiconductor film having a crystalline structure, using a mask, and gettering for segregating a metal element contained in the semiconductor film to the impurity region by heat treatment. Thereafter, pattering is conducted using the mask, whereby a semiconductor layer made of the semiconductor film having a crystalline structure is formed.