A photo retrieval method, system, and medium using multimodal information which prevents a photo of an irrelevant category to be retrieved having a high ranking due to an unfavorable side effect from a query expansion, by reflecting a level of consistency between an object category of a query and a category of a photo retrieved by expansion retrieval using ontology, to a photo retrieval result ranking is provided. The photo retrieval method using the multimodal information includes: assigning an object category with respect to a query; retrieving photos associated with an expanded query term extracted from the query; determining a ranking of the retrieved photo by reflecting the assigned object category; and arranging and displaying the photos in response to the determined ranking.