A system and methods of monitoring sequences of operations in a process running on a computer system. A probabilistic detection model is defined which is configured to determine a predictive probability of an occurrence of a final operation in the sequence of operations that is conditional on a calculated number of previous operations in the sequence of operations. The probabilistic detection model is trained from a plurality of predetermined sequences of operations to calculate the number of previous operations evaluated in the probabilistic detection model. The predictive probability for the final operation in the sequence of operations is determined by using the probabilistic detection model. If the predictive probability is below a predetermined threshold, the sequence of operations is identified as an intrusion. The probabilistic detection model may use sparse distribution trees to generate a model which determines the optimal number of previous operations to be evaluated (i.e., the window size) and position of wildcards. The system and methods may be used to monitor sequences of system calls, application function calls, and machine code instructions, for example.