A B picture mode determining method and apparatus in a video coding system includes the steps of: setting sum of absolute difference (SAD) initial values of a 44 and 1616 intra prediction mode, a forward prediction mode, a backward prediction mode, a bi-directional prediction mode and a direct prediction mode; calculating the SAD values of each mode; and selecting a mode having the smallest SAD value from the SAD values of the modes as a prediction mode for a B picture coding. A penalty value in consideration of the overhead bit of each mode is set as an SAD initial value, and especially, in order for the direct mode to be selected for the flat regions having the zero motions, the SAD initial value of the direct mode is set as -16QP₀(QP), so that the bit rate is considerably reduced and the coding performance of image information can be improved.