A three-dimensional shape modeling of a real object that reduces the limitations on the target shape and color. The rough shape of the target object is determined by a silhouette method (which has limitation on the target shape) and, at the same time, the detailed shape of the target object is determined by a triangulation method or the like (which has limitation on the target color). By using both sets of information, the consistency between the rough and detailed shape information are checked. For the portions where the consistency is not satisfied (points where detailed shape information is absent or points where detailed shape section is present outside the rough shape), rough shape is assumed to be the final object information and for other portions, the detailed shape is taken as the final object information. In this manner, three-dimensional shape modeling with reduced limitations on the target shape and color can be performed.