Provided are a method, system, and program for allocating physical storage units to a data set, wherein one or more extents of storage units are assigned to the data set. A request is received to allocate additional storage units to the data set and a determination is made of a range of available storage units to allocate to the data set in response to the request. A determination is made of one pre-existing extent allocated in the data set before receiving the request and of whether at least one storage unit in the determined range can be allocated to the determined pre-existing extent. At least one storage unit in the determined range is allocated to the determined pre-existing extent if the at least one storage unit can be allocated to the determined pre-existing extent.