The erase voltage applied to a set of non-volatile storage elements being erased is structured to provide controlled shifts in the threshold voltage of the storage elements. The erase voltage is applied as a series of voltage pulses, when necessary, to shift the threshold voltage of to-be-erased memory cells below a verify level indicative of an erased condition. To avoid over-erasing the memory cells, the second erase voltage pulse is decreased, or not increased, in magnitude when compared to the previously applied voltage pulse. By decreasing or not increasing the size of the erase voltage, the amount of charge transferred from the cells by the second pulse is controlled to more accurately position an erased threshold voltage distribution for the cells near the verify level. Subsequent erase voltage pulses are increased in magnitude to provide further erasing when needed.