A method of handling handover security configuration for a mobile device of a wireless communication system includes performing an inter-radio access technology, hereinafter called RAT, handover procedure from a serving network including a service domain to a target network including a plurality of service domains when receiving a handover command, wherein the mobile device includes security configuration corresponding to the service domain of the serving network and security configuration corresponding to the plurality of service domains of the target network, and updating the security configuration corresponding to at least one of the plurality of service domains according to the handover command no later than the time a connection with the target network is successfully established during the inter-RAT handover procedure.