René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"convert-hf-to-ggml.py CUDA out of memory","articleBody":"i've tried to convert model from HF to GGML format:\r\n```\r\npython3 convert-hf-to-ggml.py ../starcoderbase_int8\r\n```\r\nand got an error:\r\n```\r\nLoading model:  ../starcoderbase_int8\r\nLoading checkpoint shards:   ...\r\nTraceback (most recent call last):\r\n  File \"/home/alex/starcoder/starcoder.cpp/convert-hf-to-ggml.py\", line 58, in \u003cmodule\u003e\r\n    model = AutoModelForCausalLM.from_pretrained(model_name, config=config, torch_dtype=torch.float16 if use_f16 else torch.float32, low_cpu_mem_usage=True, trust_remote_code=True, offload_state_dict=True)\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py\", line 493, in from_pretrained\r\n    return model_class.from_pretrained(\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 2901, in from_pretrained\r\n    ) = cls._load_pretrained_model(\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 3258, in _load_pretrained_model\r\n    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 725, in _load_state_dict_into_meta_model\r\n    set_module_quantized_tensor_to_device(model, param_name, param_device, value=param, fp16_statistics=fp16_statistics)\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/utils/bitsandbytes.py\", line 109, in set_module_quantized_tensor_to_device\r\n    new_value = value.to(device)\r\ntorch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 576.00 MiB (GPU 0; 10.90 GiB total capacity; 9.21 GiB already allocated; 568.69 MiB free; 9.74 GiB reserved in total by PyTorch) If reserved memory is \u003e\u003e allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF\r\n```\r\nAnd next I've tried to force it to run on the CPU:\r\n```\r\nexport CUDA_VISIBLE_DEVICES=\"\"\r\npython3 convert-hf-to-ggml.py ../starcoderbase_int8\r\n```\r\nThen, I got this:\r\n```\r\nLoading model:  ../starcoderbase_int8\r\nTraceback (most recent call last):\r\n  File \"/home/alex/starcoder/starcoder.cpp/convert-hf-to-ggml.py\", line 58, in \u003cmodule\u003e\r\n    model = AutoModelForCausalLM.from_pretrained(model_name, config=config, torch_dtype=torch.float16 if use_f16 else torch.float32, low_cpu_mem_usage=True, trust_remote_code=True, offload_state_dict=True)\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py\", line 493, in from_pretrained\r\n    return model_class.from_pretrained(\r\n  File \"/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py\", line 2370, in from_pretrained\r\n    raise RuntimeError(\"No GPU found. A GPU is needed for quantization.\")\r\nRuntimeError: No GPU found. A GPU is needed for quantization.\r\n\r\n```\r\nFor me, the main reason to go with GGML implementation is that I can't fit the model in my GPU. I thought I could perform both the conversion and inference using only the CPU and system RAM. Am I doing something specific wrong or I got it wrong in general?","author":{"url":"https://github.com/Alex20129","@type":"Person","name":"Alex20129"},"datePublished":"2024-03-01T08:12:19.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/35/starcoder.cpp/issues/35"}