To provide a composite for a building material capable of manufacturing a building material that is harmless to living environment and global environment and can be recycled. The composite for a building material contains diatomaceous earth, waste lumber, and inorganic hardener, or further contains field earth, and, as necessary, contains used paper and/or crushed pieces of used tatami mat and used tile, and crushed earth. The composite for a building material is prepared by homogeneously mixing the components, and then the building material is manufactured by molding and curing the composite. The manufactured building material has a high humidity conditioning property as in the case of the conventional mud wall made of coarse clay, and is easily recycled by being crushed. Moreover, this building material does not cause hypersensitiveness to chemical substances because it produces no chemical substances, and has a feel of earth so that earthen walls and flower beds made by using this building material integrate well with an appearance of ancient city and are in harmony with beautiful appearance of houses properly.