The present invention provides a method of treating fats and oils containing low concentrations of aromatic halogen compounds and other contaminants by contacting the fats or oils with an adsorbing agent. The adsorbing agent contains a porous body and a non-protonic polar solvent held in the interiors of fine pores in the porous body to adsorb the contaminants. In one embodiment, the porous body also carries noble metal fine particles. In another embodiment, the adsorbing body comprises a solid acid for adsorbing the contaminants and there is a step of contacting the adsorbing body with an organic solvent to extract the adsorbed contaminants into the organic solvent.