An apparatus and method are disclosed for detecting intrusions in Voice
over Internet Protocol systems without an attack signature database. The
illustrative embodiment is based on two observations: (1) various
VoIP-related protocols are simple enough to be represented by a
finite-state machine (FSM) of compact size, thereby avoiding the
disadvantages inherent in signature-based intrusion-detection systems.;
and (2) there exist intrusions that might not be detectable locally by
the individual finite-state machines (FSMs) but that can be detected with
a global (or distributed) view of all the FSMs. The illustrative
embodiment maintains a FSM for each session/node/protocol combination
representing the allowed (or "legal") states and state transitions for
the protocol at that node in that session, as well as a "global" FSM for
the entire session that enforces constraints on the individual FSMs and
is capable of detecting intrusions that elude the individual FSMs.