A focus detection system including a sensor unit including a plurality of photoelectric conversion elements and a focus detection unit for performing focus adjustment using a pair of signals output from the sensor unit. In the sensor unit, an aperture corresponding to each of the photoelectric conversion elements positioned in the outer area is smaller than an aperture corresponding to each of the photoelectric conversion elements positioned in the central area. In the focus detection unit, the pair of signals is formed by pencils of rays passing through different regions of an exit pupil of an imaging lens.