The invention provides a technique of allowing fine and high-performance thin film semiconductor elements to be easily formed on a large-sized substrate. A method of manufacturing a semiconductor device includes: forming a peeling layer on a first substrate; forming an insulating film on the peeling layer; forming a plurality of fine holes in the insulating film; forming a semiconductor film on the insulating film and in the fine holes; melting and crystallizing the semiconductor film by a heat treatment to form a crystalline semiconductor film including substantially single-crystalline grains centered substantially on the respective fine holes; forming a semiconductor element T by using the crystalline semiconductor film; and causing peeling at the inside and/or the boundary surface of the peeling layer to separate the semiconductor element T from the first substrate and transferring the semiconductor element to a second substrate.