A system and process for generating a video animation from the frames of a video sprite with user-controlled motion is presented. An object is extracted from the frames of an input video and processed to generate a new video sequence or video sprite of that object. In addition, the translation velocity of the object for each frame is computed and associated with each frame in the newly generated video sprite. The system user causes a desired path to be generated for the object featured in the video sprite to follow in the video animation. Frames of the video sprite showing the object of interest are selected and inserted in a background image, or frame of a background video, along the prescribed path. The video sprite frames are selected by comparing a last-selected frame to the other video sprite frames, and selecting a video sprite frame that is identified in the comparison as corresponding to an acceptable transition from the last-selected frame. Each newly selected video sprite frame is inserted at a point along the prescribed path dictated by the velocity associated with the object in the last-inserted frame. The process of selecting, inserting and comparing video sprite frames to create successive frames of the video animation continues for as long as it is desired to produce new frames of the video animation.