model=TheBloke_Llama-2-13B-GPTQ/model.safetensors, I also tried: Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors, same problem
Loading model ...
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[36], line 5
1 # MODEL_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ'
2 # CHECKPOINT_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ/Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors'
4 DEVICE = torch.device('cuda:0')
----> 5 model, tokenizer = load_model_main(MODEL_PATH, CHECKPOINT_PATH, DEVICE)
File /mnt/data/generativeAgent_LLM/server/model.py:56, in load_model_main(model_para, checkpoint_para, device)
55 def load_model_main(model_para, checkpoint_para, device):
---> 56 model = load_quant(model_para, checkpoint_para, 4, 128)
57 model.to(device)
58 tokenizer = AutoTokenizer.from_pretrained(model_para)
File /mnt/data/generativeAgent_LLM/server/model.py:47, in load_quant(model, checkpoint, wbits, groupsize)
45 if checkpoint.endswith('.safetensors'):
46 from safetensors.torch import load_file as safe_load
---> 47 model.load_state_dict(safe_load(checkpoint))
48 else:
49 model.load_state_dict(torch.load(checkpoint))
File ~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1671, in Module.load_state_dict(self, state_dict, strict)
1666 error_msgs.insert(
1667 0, 'Missing key(s) in state_dict: {}. '.format(
1668 ', '.join('"{}"'.format(k) for k in missing_keys)))
1670 if len(error_msgs) > 0:
-> 1671 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
1672 self.__class__.__name__, "\n\t".join(error_msgs)))
1673 return _IncompatibleKeys(missing_keys, unexpected_keys)
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.0.self_attn.k_proj.g_idx", "model.layers.0.self_attn.o_proj.g_idx", "model.layers.0.self_attn.q_proj.g_idx", "model.layers.0.self_attn.v_proj.g_idx", "model.layers.0.mlp.down_proj.g_idx", "model.layers.0.mlp.gate_proj.g_idx", "model.layers.0.mlp.up_proj.g_idx", "model.layers.1.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn.k_proj.g_idx", "model.layers.1.self_attn.o_proj.g_idx", "model.layers.1.self_attn.q_proj.g_idx", "model.layers.1.self_attn.v_proj.g_idx", "model.layers.1.mlp.down_proj.g_idx", "model.layers.1.mlp.gate_proj.g_idx", "model.layers.1.mlp.up_proj.g_idx", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.2.self_attn.k_proj.g_idx", "model.layers.2.self_attn.o_proj.g_idx", "model.layers.2.self_attn.q_proj.g_idx", "model.layers.2.self_attn.v_proj.g_idx", "model.layers.2.mlp.down_proj.g_idx", "model.layers.2.mlp.gate_proj.g_idx",
model=TheBloke_Llama-2-13B-GPTQ/model.safetensors, I also tried: Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors, same problem