A highlight portion is detected to a high accuracy from acoustic signals in say an event, and an index is added to the highlight portion. In an acoustic signal processing apparatus 10, a candidate domain extraction unit 13 retains a domain, a length of which with short-term amplitudes as calculated by an amplitude calculating unit 11 not being less than an amplitude threshold value is not less than a time threshold value, as a candidate domain. A feature extraction unit 14 extracts sound quality featuring quantities, relevant to the sound quality, from the acoustic signals, to quantify the sound quality peculiar to a climax. A candidate domain evaluating unit 15 calculates a score value, indicating the degree of the climax, using featuring quantities relevant to the amplitude or the sound quality for each candidate domain, in order to detect a true highlight domain, based on the so calculated score value. An index generating unit 16 generates and outputs an index including the start and end positions and the score values of the highlight domain.