A method and system is presented for tracking patient motion in image guided surgery, using skeletal reference structures instead of fiducial markers. A non-rigid 2D/3D image registration is performed between pre-operative 3D scan data and intra-operative 2D images. The CT numbers are modified to compensate for the difference between the ratio of bone-to-tissue attenuation at the CT scan energy level, and the x-ray projection energy level. DRRs are generated from the modified CT numbers. An ROI, within which the image registration is restricted, is defined within the DRRs by calculating a modified Shannon entropy for each image, and seeking the region in which the image entropy is maximized. The DRRs and 2D images are enhanced by applying a top hat filter to increase the visibility of the skeletal structures. The 2D/3D transformation parameters are determined by generating a full motion field that accounts for non-rigid motions and deformations. The full motion field is derived by estimating a plurality of local motion fields, using multi-level block matching and a similarity measure based on pattern intensity.