zai-org

/

LongWriter-glm4-9b

like 124

Follow

Z.ai 3.6k

Heya folks,
For text-generation-webui on cpu if you try downloading this model out of the box you will get an error. Through a little bit of time and searching chinese forums you need to add this to modeling_chatglm.py: (into line 971):

def _extract_past_from_model_output(self, outputs: ModelOutput, standardize_cache_format: bool = False):
     past_key_values = None
     cache_name = None
 if "past_key_values" in outputs:
    past_key_values = outputs.past_key_values
    cache_name = "past_key_values"
 elif "mems" in outputs:
    past_key_values = outputs.mems
    cache_name = "mems"
 elif "past_buckets_states" in outputs:
    past_key_values = outputs.past_buckets_states
    cache_name = "past_buckets_states"

if standardize_cache_format and hasattr(self, "_convert_to_standard_cache"):
    batch_size = outputs.logits.shape[0]
    past_key_values = self._convert_to_standard_cache(past_key_values, batch_size=batch_size)

return cache_name, past_key_values

This will probably be mostly for myself because I will forget I did this but if it helps anyone else then that would be great

source: https://blog.csdn.net/qq_43749831/article/details/146022858