An automated method for determining prognosis based on an analysis of abnormality (lesion) features and parenchymal features obtained from medical image data of a patient. The techniques include segmentation of lesions from radiographic images, extraction of lesion features, and a merging of the features (with and without clinical information) to yield as estimate of the prognosis for the specific case. An example is given for the prognosis of breast cancer lesions using mammographic data. A computerized image analysis system for assessing prognosis combines the computerized analysis of medical images of cancerous lesions with the training-based methods of assessing prognosis of a patient, using indicators such as lymph node involvement, presence of metastatic disease, local recurrence, and/or death. It is expected that use of such a system to assess the severity of the disease will aid in improved decision-making regarding treatment options.