René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"convert-hf-to-ggml.py offload_weight() too much parameters","articleBody":"I tried to follow the _Quick start_.\r\n\r\n```\r\n$ git clone https://github.com/bigcode-project/starcoder.cpp\r\nClonage dans 'starcoder.cpp'...\r\nremote: Enumerating objects: 110, done.\r\nremote: Counting objects: 100% (110/110), done.\r\nremote: Compressing objects: 100% (78/78), done.\r\nremote: Total 110 (delta 34), reused 93 (delta 25), pack-reused 0 (from 0)\r\nRéception d'objets: 100% (110/110), 7.28 Mio | 28.89 Mio/s, fait.\r\nRésolution des deltas: 100% (34/34), fait.\r\n```\r\n```\r\n$ cd starcoder.cpp/\r\n```\r\n```\r\n$ python convert-hf-to-ggml.py bigcode/gpt_bigcode-santacoder\r\nLoading model:  bigcode/gpt_bigcode-santacoder\r\ntokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 159/159 [00:00\u003c00:00, 324kB/s]\r\ntokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.08M/2.08M [00:00\u003c00:00, 6.11MB/s]\r\nspecial_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 138/138 [00:00\u003c00:00, 387kB/s]\r\nSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\r\nconfig.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 812/812 [00:00\u003c00:00, 2.52MB/s]\r\nmodel.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.25G/2.25G [00:43\u003c00:00, 52.3MB/s]\r\nTraceback (most recent call last):\r\n  File \"/mnt/hdd/Data/ia/starcoder.cpp/convert-hf-to-ggml.py\", line 58, in \u003cmodule\u003e\r\n    model = AutoModelForCausalLM.from_pretrained(model_name, config=config, torch_dtype=torch.float16 if use_f16 else torch.float32, low_cpu_mem_usage=True, trust_remote_code=True, offload_state_dict=True)\r\n  File \"/home/dmeziere/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py\", line 563, in from_pretrained\r\n    return model_class.from_pretrained(\r\n  File \"/home/dmeziere/.local/lib/python3.10/site-packages/transformers/modeling_utils.py\", line 3507, in from_pretrained\r\n    ) = cls._load_pretrained_model(\r\n  File \"/home/dmeziere/.local/lib/python3.10/site-packages/transformers/modeling_utils.py\", line 3932, in _load_pretrained_model\r\n    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(\r\n  File \"/home/dmeziere/.local/lib/python3.10/site-packages/transformers/modeling_utils.py\", line 798, in _load_state_dict_into_meta_model\r\n    state_dict_index = offload_weight(param, param_name, model, state_dict_folder, state_dict_index)\r\nTypeError: offload_weight() takes from 3 to 4 positional arguments but 5 were given\r\n```\r\nThen i tried `make`, but it also returns a lot of warnings. But i think it has compiled.\r\n\r\n```\r\n$ ./quantize models/bigcode/gpt_bigcode-santacoder-ggml.bin models/bigcode/gpt_bigcode-santacoder-ggml-q4_1.bin 3\r\nstarcoder_model_quantize: loading model from 'models/bigcode/gpt_bigcode-santacoder-ggml.bin'\r\nstarcoder_model_quantize: failed to open 'models/bigcode/gpt_bigcode-santacoder-ggml.bin' for reading\r\nmain: failed to quantize model from 'models/bigcode/gpt_bigcode-santacoder-ggml.bin'\r\n```\r\n\r\nThe model seems to be missing, probably because of the errors of `convert-hf-to-ggml.py`.","author":{"url":"https://github.com/dmeziere","@type":"Person","name":"dmeziere"},"datePublished":"2025-01-06T16:01:27.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/36/starcoder.cpp/issues/36"}