In general, the present invention is directed to systems and methods for finding the position and shape of an animal using video. The invention includes a system with a video camera coupled to a computer in which the computer is configured to automatically provide animal segmentation and identification, animal motion tracking (for moving animals), animal-posture classification, and behavior identification. In a preferred embodiment, the present invention may use background subtraction for animal identification and tracking, and a combination of decision tree classification and rule-based classification for posture and behavior identification. Thus, the present invention is capable of automatically monitoring a video image to identify, track and classify the actions of various animals and the animal's movements within the image. The image may be provided in real time or from storage. The invention is particularly useful for monitoring and classifying animal behavior for testing drugs and genetic mutations, but may be used in any of a number of other surveillance applications.