A data storage subsystem in a distributed storage system having a plurality of predictive failure analyzing data storage devices. The subsystem furthermore has a circuit that is responsive to a predicted failure indication by a data storage device in relation to predetermined rules stored in memory for deterministically initiating a preventive recovery either by a data recovery procedure in the data storage device or by a fault tolerance storage arrangement in the subsystem.