A handheld device includes a gesture database maintaining a plurality of gestures, each gesture defined by a motion of the device with respect to a first position of the handheld device. The gestures comprise a plurality of remote command gestures and at least one device selection gesture. The device includes a gesture mapping database comprising a plurality of command maps, each of the command maps corresponding to a particular controllable device and mapping at least one of the remote command gestures to a command for controlling operation of the particular controllable device. The device also includes a motion detection module operable to detect motion of the handheld device within three dimensions and a device selection module operable to detect the device selection gesture based on the motion of the handheld device and to select a currently controlled one of the controllable devices in response to the device selection gesture. The device includes a control module operable to select one of the command maps corresponding to the currently controlled controllable device, to track movement of the handheld device using the motion detection module, to compare the tracked movement against the remote command gestures to determine a matching gesture, and to identify, using the selected command map, the command mapped to the matching gesture. The device also includes a wireless interface operable to transmit the identified command to a remote receiver for delivery to the currently controlled controllable device.