A prime mover control device of a construction machine according to the present invention includes a hydraulic pump (11) driven by a prime mover (40), a hydraulic traveling motor (5) driven with pressure oil output from the hydraulic pump (11), and a control valve (12) that controls a flow of the pressure oil from the hydraulic pump (11) to the hydraulic motor (5) in response to an operation of an operating member (22a). The prime mover control device further includes a deceleration detection means (31) that detects a deceleration operation at the operating member (22a), a rotation speed detection means (35) that detects a rotation speed Nm of the hydraulic motor (5), and a prime mover rotation speed control means (30,43) that executes speed reduction control of the rotation speed of the prime mover (40) based upon detection results provided by the rotation speed detection means (35) if the deceleration operation is detected by the deceleration detection means (31) and controls the rotation speed of the prime mover (40) in correspondence to an operation of the operating member (22a) which is detected to be other than the deceleration operation.