Effects of diffraction of a spectral beam from an edge of the micromirrors are reduced in order to optimize the passband in a wavelength selective switch. The effects of diffraction on the pass band may be reduced by using rotation of the micromirror about both the attenuation axis and the switching axis to achieve the desired level of attenuation. Peak coupling can be attained by dithering the micromirror about a dither axis that is tangent to a contour of constant attenuation using simultaneous rotation about the switching and attenuation axes. A power level of a spectral channel may be attenuated by rotating the channel micromirror with respect to an effective attenuation axis that is non-orthogonal to the dither axis through a combination of rotations about the switching axis and the attenuation axis.