A semiconductor memory device includes a trench formed in the semiconductor substrate, a diffusion layer for a first electrode formed within the semiconductor substrate so as to be in contact with an inner surface of the trench, a capacitor insulating film formed on the diffusion layer, a conductive layer for a second electrode formed so as to bury a lower portion of the trench, a first insulating film formed on the conductive layer and along a side surface of the trench, a first conductive layer formed so as to bury an intermediate portion of the trench, a first contact layer formed so as to bury an upper portion of the trench, and a second contact layer formed on the surface of the semiconductor substrate so as to be in contact with the first contact layer.