Provided are a relay multicast system and method for providing efficient group communication services. In the relay multicast system, each multicast feeder can make a peer to open connection with multicast feeders in other classes by selecting an optimal multicast feeder based on available resources of each relay multicast feeder in order to efficiently form a data forwarding route of a relay group. Also, in order to form a relay channel, the relay multicast system negotiates and uses a most appropriate channel between hops, not a data forwarding channel unified in a session or group. Also, in order to forward data to a specific group, the relay multicast system connects local multicast domains, in which it is possible to multicast or broadcast on a small scale as in a local area network (LAN) or a company network in which multicast members of the group are included, without changing an existing 1:1 unicast based Internet infrastructure.