A process to upgrade heavy oil and convert the heavy oil into lower boiling hydrocarbon products is provided. The process employs a catalyst slurry comprising catalyst particles with an average particle size ranging from 1 to 20 microns. In the upgrade process, spent slurry catalyst in heavy oil is generated as an effluent stream, which is subsequently recovered/separated from the heavy oil via membrane filtration. In one embodiment, dynamic filtration is used for the separation of the heavy oil from the catalyst particles. Valuable metals can be recovered from catalyst particles for subsequent re-use in a catalyst synthesis unit, generating a fresh slurry catalyst.