There is provided a hierarchical shadow detection system for color aerial images. The system performs well with highly complex images as well as images having different brightness and illumination conditions. The system consists of two hierarchical levels of processing. The first level involves, pixel level classification, through modeling the image as a reliable lattice and then maximizing the lattice reliability using the EM algorithm. Next, region level verification, through further exploiting the domain knowledge is performed. Further analysis show that the MRF model based segmentation is a special case of the pixel level classification model. A quantitative comparison of the system and a state-of-the-art shadow detection algorithm clearly indicates that the new system is highly effective in detecting shadow regions in an image under different illumination and brightness conditions.