The present invention discloses a scalable video format conversion system, which utilizes a plurality of system resources to convert an interlaced video signal into a progressive video signal. The disclosed scalable video format conversion system contains a scalable motion-adaptive de-interlacing system and a mode control module. The mode control module determines a detection number dynamically according to the availability of the system resources and/or the status of the scalable video format conversion system. A variable-field motion detection apparatus of the scalable motion-adaptive de-interlacing system accesses a plurality of video fields to detect a motion situation of an image area, wherein the number of the plurality of accessed video is equal to the detection number determined by the mode control module. Then, the scalable motion-adaptive de-interlacing system choose a proper de-interlacing algorithm according to the detected motion situation, to convert the interlaced video signal into the progressive video signal.