A motion controlled handheld device includes a display having a viewable surface and operable to generate an image and a gesture database maintaining a plurality of gestures. Each gesture is defined by a motion of the device with respect to a first position of the device. The device includes a plurality of applications each having a plurality of predefined commands and a gesture mapping database comprising a plurality of command maps. Each of the command maps corresponds to a particular one of the applications and maps each of the predefined commands to one of the gestures. The device includes a motion detection module operable to detect motion of the handheld device within three dimensions and to identify components of the motion in relation to the viewable surface. The device also includes a control module operable to load one of the applications, to select one of the command maps corresponding to the loaded application, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the gestures to determine a matching gesture, to identify, using the selected command map, the predefined command mapped to the matching gesture, and to perform the identified command using the loaded application.