A packer or plug features a main sealing element that swells after a delay long enough to get it into proper position. A sleeve eventually goes away to let the well fluids at the main sealing element to start the swelling process until contact with the surrounding tubular or the wellbore is established. Other sleeves that are disposed above and below the main sealing element preferably swell, but mainly in a longitudinal direction against the main sealing element to increase its contact pressure against the surrounding tubular or the wellbore. The longitudinally swelling members may also be covered to initiate their growth after the main sealing element has started or even completed its swelling action. The longitudinally swelling members can be constrained against radial growth to direct most or all of their swelling action longitudinally. Extrusion barriers above and below the main sealing element can optionally be used.