A virtual interactive environment enables a surgeon or other medical professional to manipulate implants, prostheses, or other instruments using patient-specific data from virtual reality models. The patient data includes a combination of volumetric data, surface data, and fused images from various sources (e.g., CT, MRI, x-ray, ultrasound, laser interferometry, PET, etc.). The patient data is visualized to permit a surgeon to manipulate a virtual image of the patient's anatomy, the implant, or both, until the implant is ideally positioned within the virtual model as the surgeon would position a physical implant in actual surgery. Thus, the interactive tools can simulate changes in an anatomical structure (e.g., bones or soft tissue), and their effects on the external, visual appearance of the patient. CAM software is executed to fabricate the implant, such that it is customized for the patient without having to modify the structures during surgery or to produce a better fit.