A headlight beam control system includes an image taking apparatus for capturing an image to the rear of the user's vehicle and for generating image data from the captured image, a following vehicle information acquisition unit for acquiring following vehicle information from the image data, a passing action prediction unit for predicting a passing action based on the following vehicle information, a mode switching condition judgment unit for judging whether a mode switching condition is satisfied, based on the predicted passing action, and an automatic mode setting unit for switching the headlights between a high beam mode and a low beam mode when the mode switching condition is satisfied. When passing of the user's vehicle is predicted based on the following vehicle information, the headlights are switched from the high beam mode to the low beam mode.