A method for adaptive video processing includes identifying an object in a frame of video data, determining an object confidence value associated with the identified object, and determining if the object confidence value meets a predefined threshold object confidence value. If not, the method can further include repeatedly processing the frame of video data with additional processing algorithms, and again identifying the object, determining an updated object confidence value, and determining if the updated object confidence value meets the predefined threshold object confidence value, until the updated object confidence value meets the predefined threshold object confidence value. Thereafter, an event can be identified based upon the identified object and an object primitive defining a relationship between the identified object and one or more other objects in the frame of video data, where the events can be identified in accordance with a hierarchical event structure.