In the present invention, the histogram model used in H-PMHT is extended to treat the problem of tracking using hyper-spectral data. Completely general spectral density functions are handled via the use of non-parametric methods. The present invention is not restricted to derivations based on knowledge of the spectral character of the source being tracked. The source spectrum can be estimated in a non-parametric fashion based on an initial track, and this allows the invention to adapt to the source spectrum in situ. The resulting method has improved crossing track performance on sources that have some degree of spectral distinction and will perform no worse than regular H-PMHT on sources that have identical spectral densities.