An apparatus in a microprocessor for selectively retiring a prefetched cache line is disclosed. The microprocessor includes a prefetch buffer that stores a cache line prefetched from a system memory coupled to the microprocessor. The microprocessor includes a cache memory, comprising an array of storage elements for storing cache lines. The array is indexed by an index input. The microprocessor includes a counter that counts a number of accesses to a replacement candidate line in the cache. The replacement candidate line is stored in a storage element of the array indexed by an index portion of an address of the prefetched cache line stored in the prefetch buffer. The microprocessor also includes control logic that selectively replaces the replacement candidate cache line in the cache memory with the prefetched cache line from the prefetch buffer based on the number of accesses to the replacement candidate line.