A video memory manager manages and virtualizes memory so that an application or multiple applications can utilize both system memory and local video memory in processing graphics. The video memory manager allocates memory in either the system memory or the local video memory as appropriate. The video memory manager may also manage the system memory accessible to the graphics processing unit via an aperture of the graphics processing unit. The video memory manager may evict memory from the local video memory as appropriate, thereby freeing a portion of local video memory use by other applications. In this manner, a graphics processing unit and its local video memory may be more readily shared by multiple applications.