A speech communication system includes a plurality of portable terminals and a server device. The server device manages the talker right specifying a terminal which transmits speech when speech communication is performed between the portable terminals. The portable terminal includes a registration request notification unit. The registration request notification unit transmits, to the server device in response to predetermined external operation, a registration request notification for requesting registration of a reservation for the talker right. The server device includes a management table and a reservation registration unit. The management table stores management information containing the reservation status of the talker right. The reservation registration unit registers a reservation for the talker right in the management table in accordance with a registration request notification from the portable terminal. A talker right reservation method is also disclosed.