A silicon nitride film for storing electric charge is formed on a semiconductor substrate while placing a tunnel oxide film in between, and the silicon nitride film is then subjected to hydrogen plasma treatment so as to effectively erase unnecessary charge stored therein during various process steps in fabrication of the semiconductor memory device, to thereby stabilize the threshold voltage (V.sub.th) of the semiconductor memory device.