A system and method for recognizing a goal of an actor in an environment by considering hypothesized unobserved actions. The method includes providing a plan library that includes a list of possible goals and corresponding plans. Actions of the actor are observed, and the occurrence of hypothesized unobserved actions are considered. Finally, one or more plans consistent with the observed actions and the consideration of hypothesized unobserved actions are identified.