There is provided an image recognition system which recognizes a subject, which is moving with respect to an image capturing unit, included in an image captured by the image capturing unit, comprising: the image capturing unit; a first recognition unit which recognizes a candidate of the subject included in the image captured by the image capturing unit by comparing the image captured by the image capturing unit with predetermined template data; a template selection unit which selects the template data to be compared with an image captured by the image capturing unit after a predetermined duration has been elapsed from a time of capturing an image used for recognition of the candidate by the recognition unit; and a second recognition unit which recognizes the subject by comparing the image, which is captured by the image capturing unit after a predetermined duration has been elapsed from a time of capturing an image used for recognition of the candidate by the first recognition unit, with the template data selected by the template selection unit.