An adhesion layer composed of a titanium film and a titanium nitride film is formed by CVD on the inner wall of a contact hole formed in a multilayer film composed of an interlayer insulating film, a silicon nitride film, and a silicon dioxide film. Then, a conductive film made of tungsten or polysilicon is filled by CVD in the contact hole and the respective portions of the conductive film and the adhesion layer which are located over the silicon dioxide film are removed by CMP. Subsequently, the silicon dioxide film is removed by an etch-back method or a CMP method so that the silicon nitride film is exposed. This can prevent the delamination of the adhesion layer from the silicon nitride film as a hydrogen barrier film and also prevent the formation of a scratch in the silicon nitride film.