For automatically detecting a moving object in an input image taken using a monitor camera or the like, an apparatus includes an input image extracting section, an input image retaining section, a background image retaining section, a background difference calculating section, a moving direction recognizing section, and a united judging section. A movement of an object is detected with precision even in an environment where illumination varies, irrespective of the relative position of a ridge line of the object and a border of unit blocks. And only the moving object can be detected stably even when the camera encounters swing, vibration, etc.