An interior rearview mirror system includes an interior rearview mirror assembly and a compass sensor having first and second magnetoresponsive sensing elements. A physical mount mounts the compass sensor at the rear of the reflective element whereby the compass sensor adjusts in tandem with adjustment of the reflective element by the driver of the vehicle. The physical mount is configured to at least partially offset the angle of the reflective element selected by the driver. A control receives a first signal indicative of a magnetic field sensed by the first magnetoresponsive sensing element and receives a second signal indicative of a magnetic field sensed by the second magnetoresponsive sensing element. The control determines a directional heading of the vehicle based on the first and second signals, and automatically compensates for a deviating magnetic field of the vehicle and generates a signal indicative of the directional heading of the vehicle.