A method of setting up a connection in a communication system is provided, wherein the communication system includes a user equipment (UE). The present method includes following steps. When a first radio resource control (RRC) connection request message is received from the UE, whether the first RRC connection request message has been received before is determined. If the first RRC connection request message has been received, a first RRC connection setup message is sent back to the UE. If the first RRC connection request message has not been received, whether there is an accessible Femto access point (FAP) near the UE is further determined. If there is the accessible FAP near the UE, a first RRC connection reject message is sent back to the UE. If there is no accessible FAP near the UE, the first RRC connection setup message is sent back to the UE.