A wireless communication method and antenna system for determining the direction of arrival (DOA) of received signals in azimuth and elevation, (i.e., in three dimensions), to form a beam for transmitting and receiving signals. The system includes two antenna arrays, each having a plurality of antenna elements, two first stage multi-mode-port matrices, at least one second stage multi-mode-port matrix, an azimuth phase detector, an elevation amplitude detector, a plurality of phase shifters and a transceiver. The antenna arrays and the first stage multi-mode-port matrices form a plurality of orthogonal omni-directional modes. Each of the modes has a characteristic phase set. Two of the modes' phases are used to determine DOA in azimuth. The second stage multi-mode-port matrix forms a sum-mode and a difference-mode used to determine the DOA of the received signals in elevation. A beam is formed in the direction of the received signals by adjusting the phase shifters.