In an image classification system (70), a plurality of generative models (30) correspond to a plurality of image classes. Each generative model embodies a merger of a general visual vocabulary and an image class-specific visual vocabulary. A gradient-based class similarity modeler (40) includes (i) a model fitting data extractor (46) that generates model fitting data of an image (72) respective to each generative model and (ii) a dimensionality enhancer (50) that computes a gradient-based vector representation of the model fitting data with respect to each generative model in a vector space defined by the generative model. An image classifier (76) classifies the image respective to the plurality of image classes based on the gradient-based vector representations of class similarity.