A negative resist composition is provided comprising a polymer comprising recurring units having formula (1), a photoacid generator, and a crosslinker. In formula (1), X is alkyl or alkoxy, R.sup.1 and R.sup.2 are H, OH, alkyl, substitutable alkoxy or halogen, R.sup.3 and R.sup.4 are H or CH.sub.3, n is an integer of 1 to 4, m and k are an integer of 1 to 5, p, q and r are positive numbers. The composition has a high contrast of alkali dissolution rate before and after exposure, high sensitivity, high resolution and good etching resistance.