Methods and apparatus, including computer program products, for filtering an image. A filter kernel is received to determine one or more filtered values for each pixel in a sequence of pixels, where adjacent pixels are separated by a characteristic distance in the image. A difference kernel is defined based on local differences between a first kernel and a second kernel that are defined by the filter kernel centered at a first location and a second location, respectively. The second location is separated from the first location by the characteristic distance separating adjacent pixels in the sequence. The difference kernel is used to determine a difference between filtered values of adjacent pixels in the sequence. For depth of field filtering, the filter kernel can include a blur filter kernel that is based upon depth values of pixels in the sequence.