A modular building system (100) including preformed extrusion-molded polymeric material (116). Parts include integral connector elements (160, 162) which can be slidably and snappingly reversibly interconnected to form the basic frame (102) and panel structures (108) of an overall building. The interconnected region permits limited ranges of angular as well as translational relative motion between adjacent, connected elements. Parts are assembled simply and quickly in the field, without requiring skilled labor or specialized tools, to form a functionally solid and stable building which responds to loads and temperature changes with yieldable, accommodating deformations that minimize the likelihood of building structural damage, and effectively transmit necessary loads to the ground via load-transfer paths through the building.