The semiconductor memory device according to the present invention includes a charge storage layer 26 formed over a semiconductor substrate 10 and including a plurality of particles 16 as charge storage bodies in insulating films 12, 24, and a gate electrode 30 formed over the charge storage layer 26, in which the particles 16 are formed of metal oxide or metal nitride.