An isolated polarization beam splitter or combiner, for joining light from different inputs into one common port, and for dividing a beam of light into orthogonal polarizations. In both modes of operation, the splitter/combiner provides isolation preventing transmission of light in a reverse direction. As a splitter, a beam of light is separated through a birefringent material into sub-beams of orthogonal polarization components, and each sub-beam is passed through a non-reciprocal polarization rotator to rotate the polarization so that a reflected beam, or other counter-transmitted light cannot return on the same path through the birefringent material to the source. As a combiner, two separate beams of light are launched with known orthogonal polarizations into a first birefringent material, passed through a non-reciprocal polarization rotator and then combined as orthogonal polarizations into a single output port.