The present invention provides a semiconductor memory device comprising: a first conductivity type semiconductor substrate; and a plurality of memory cells constituted of an island-like semiconductor layer which is formed on the semiconductor substrate, and a charge storage layer and a control gate which are formed entirely or partially around a sidewall of the island-like semiconductor layer, wherein the plurality of memory cells are disposed in series, the island-like semiconductor layer which constitutes the memory cells has cross-sectional areas varying in stages in a horizontal direction of the semiconductor substrate, and an insulating film capable of passing charges is provided at least in a part of a plane of the island-like semiconductor layer horizontal to the semiconductor substrate.