A user interface system comprises a server having a scene description converting device for converting scene description containing input user interaction into converted scene description while leaving parts containing the user interaction and a scene description decoding device for decoding converted scene description into decoded scene description, a remote terminal having a scene description decoding device for decoding scene description and converted scene description sent from the server and a display device for displaying decoded scenes and a user input device for receiving user input according to this display, and a display terminal for displaying decoded scenes sent from the server. Thus, decoding can be enabled at terminals having inferior decoding capabilities and display capabilities.