In a space incremental garbage collector, delays caused by scanning remembered sets are reduced by scanning the remembered sets mostly concurrently with the operation of the non-collection threads. Before scanning, remembered sets associated with regions in some collection set are "sequestered" (no further insertions are allowed by the non-collection threads or the collector) in order to prevent further changes. Any further changes to these sequestered remembered sets that occur before phase two of the scanning process are then handled by one or more "refinement threads." Each refinement thread is a thread that can record reference locations in remembered sets, scan entries in collection set remembered sets and update reference location lists for use in the aforementioned second phase of remembered set processing. The refinement threads operate concurrently with the operation of the non-collection threads.