A system and method of removing background noise from a digital image of a scanned document is described. The system and method is a reversible background noise removal technique that allows a user to select whether background noise is removed or not. In addition, since the present invention divides the background noise removal operations into a two phase process, the overall efficiency of the system and method are significantly improved.