A shared system resource such as a file server includes an integrated,
cooperative cluster of domains that include hierarchically related
domains and peer related domains, each performing functions supporting
the services provided by the system resource. Hierarchically related
domains include a higher level domain and a lower level domain
respectively performing higher and lower level operations of functions
supporting the system resource services and peer related domains include
parallel domains performing operations in mutual support of functions
supporting the system resource services. A domain may include fault
handling mechanisms operating independently of and cooperatively with
fault handling mechanisms of other domains, and domain may be comprised
of domains. In hierarchically related domains, the lower level domain
includes peer related domains performing operations in mutual support of
functions of the upper level domain and peer domain includes a monitoring
mechanism monitoring operations of the other peer domain that are related
to the operations performed in support of the upper level domain. Each
monitoring mechanism is responsive to detection of a failure in the other
peer domain for directing the peer domain in which the monitoring
mechanism resides in assuming the operations performed by the peer domain
in support of related functions of the upper level domain independently
of operations of the peer domain and independently of a source of the
failure in the other peer domain.