A semiconductor device includes a semiconductor substrate, a gate insulating layer, a gate electrode structure and a side wall structure. The gate insulating layer is formed on the semiconductor substrate. The gate electrode structure is formed on the gate insulating layer, and includes a lower gate electrode layer and a cap gate layer. The side wall structure includes a nitride side wall spacer, and an oxide layer formed between the semiconductor substrate and the nitride side wall spacer and between the lower gate electrode layer and the nitride side wall spacer. A thickness of the oxide layer is greater than a thickness of the gate insulating layer, so as to prevent diffusion of nitrogen from the nitride side wall spacer to the semiconductor substrate. A height of the gate electrode structure is substantially equal to a height of the side wall structure after completion of the semiconductor device.