An earmuff structure for headset or ear protector includes two earmuff shells and two integrally injection molded speaker mounting plates that are screwed to an inner side of the earmuff shells and each has a central hole for receiving a speaker therein. The speaker mounting plates are provided at a rear surface with wall portions of predetermined heights. The wall portions on one of the speaker mounting plates are formed from outer bottom surfaces of two battery compartments formed on a front surface of the speaker mounting plate at two sides of the central hole. With the speaker mounting plates, batteries and speakers may be quickly assembled to form the earmuffs or disassembled for replacement. The wall portions on each speaker mounting plate cooperates with a foam located behind the speaker to define a space functioning like a cabinet, creating clear and high quality stereo sounds output from the speakers.