René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BitBLAS's compiling seems conflicting with vLLM's torch.compile integration?","articleBody":"Failed to run a GPTQ model https://huggingface.co/JunHowie/Qwen3-8B-GPTQ-Int4 with latest vLLM with `quantization=\"gptq_bitblas\"`.\n\nGot:\n```\n(EngineCore_DP0 pid=506800) torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped\n(EngineCore_DP0 pid=506800)   Explanation: Dynamo does not know how to trace the builtin `\u003cunknown module\u003e._SimpleCData.__new__.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).\n... (omitted)\n(APIServer pid=506583) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\n```\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\nComplete log (after tuning):\n\u003c/summary\u003e\n\n```\nLoading safetensors checkpoint shards:   0% Completed | 0/2 [00:00\u003c?, ?it/s]\nLoading safetensors checkpoint shards:  50% Completed | 1/2 [00:00\u003c00:00,  1.38it/s]\nLoading safetensors checkpoint shards: 100% Completed | 2/2 [00:01\u003c00:00,  1.58it/s]\nLoading safetensors checkpoint shards: 100% Completed | 2/2 [00:01\u003c00:00,  1.54it/s]\n(EngineCore_DP0 pid=506800) \n(EngineCore_DP0 pid=506800) INFO 10-14 21:31:21 [default_loader.py:314] Loading weights took 1.41 seconds\n(EngineCore_DP0 pid=506800) INFO 10-14 21:31:50 [gpu_model_runner.py:2910] Model loading took 5.6824 GiB and 489.310448 seconds\n(EngineCore_DP0 pid=506800) /home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/variables/functions.py:1481: UserWarning: Dynamo does not know how to trace the builtin `\u003cunknown module\u003e._SimpleCData.__new__.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).\n(EngineCore_DP0 pid=506800) If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.\n(EngineCore_DP0 pid=506800) If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.\n(EngineCore_DP0 pid=506800)   torch._dynamo.utils.warn_once(explanation + \"\\n\" + \"\\n\".join(hints))\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] EngineCore failed to start.\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] Traceback (most recent call last):\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 783, in run_engine_core\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     engine_core = EngineCoreProc(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 555, in __init__\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     super().__init__(\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 112, in __init__\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 223, in _initialize_kv_caches\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     available_gpu_memory = self.model_executor.determine_available_memory()\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/executor/abstract.py\", line 88, in determine_available_memory\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return self.collective_rpc(\"determine_available_memory\")\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/executor/uniproc_executor.py\", line 74, in collective_rpc\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return [run_method(self.driver_worker, method, args, kwargs)]\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/utils/__init__.py\", line 2977, in run_method\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return func(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 120, in decorate_context\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return func(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/worker/gpu_worker.py\", line 280, in determine_available_memory\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     self.model_runner.profile_run()\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/worker/gpu_model_runner.py\", line 3722, in profile_run\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     hidden_states, last_hidden_states = self._dummy_run(\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]                                         ^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 120, in decorate_context\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return func(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/worker/gpu_model_runner.py\", line 3455, in _dummy_run\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     outputs = self.model(\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]               ^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/compilation/cuda_graph.py\", line 126, in __call__\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return self.runnable(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return self._call_impl(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return forward_call(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen3.py\", line 321, in forward\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     hidden_states = self.model(\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]                     ^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/compilation/decorators.py\", line 407, in __call__\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     output = self.compiled_callable(*args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py\", line 745, in compile_wrapper\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     raise e.with_traceback(None) from e.__cause__  # User compiler error\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   Explanation: Dynamo does not know how to trace the builtin `\u003cunknown module\u003e._SimpleCData.__new__.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   Hint: If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   Hint: If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] \n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   Developer debug context: module: \u003cunknown module\u003e, qualname: _SimpleCData.__new__, skip reason: \u003cmissing reason\u003e\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] \n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] \n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] from user code:\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]    File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen2.py\", line 385, in forward\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     hidden_states, residual = layer(positions, hidden_states, residual)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen3.py\", line 225, in forward\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     hidden_states = self.self_attn(\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen3.py\", line 144, in forward\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     qkv, _ = self.qkv_proj(hidden_states)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/layers/linear.py\", line 582, in forward\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     output_parallel = self.quant_method.apply(self, input_, bias)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/layers/quantization/gptq_bitblas.py\", line 479, in apply\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     out = self.kernel.apply_gptq_bitblas_linear(layer, x)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/layers/quantization/kernels/mixed_precision/bitblas.py\", line 315, in apply_gptq_bitblas_linear\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     output = self.bitblas_matmul(*args)  # type: ignore[operator]\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/bitblas/ops/general_matmul/__init__.py\", line 756, in __call__\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     return self.forward(*args, **kwds)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/bitblas/ops/general_matmul/__init__.py\", line 751, in forward\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     self._forward_from_prebuild_lib(*args, stream=stream.cuda_stream)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/bitblas/ops/operator.py\", line 459, in _forward_from_prebuild_lib\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     ctypes.c_void_p(arr.data_ptr()) if not isinstance(arr, int) else arr for arr in args\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/polyfills/__init__.py\", line 204, in instantiate_user_defined_class_object\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792]     obj = cls.__new__(cls, *args, **kwargs)\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] \n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS=\"+dynamo\"\n(EngineCore_DP0 pid=506800) ERROR 10-14 21:31:51 [core.py:792] \n(EngineCore_DP0 pid=506800) Process EngineCore_DP0:\n(EngineCore_DP0 pid=506800) Traceback (most recent call last):\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py\", line 314, in _bootstrap\n(EngineCore_DP0 pid=506800)     self.run()\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py\", line 108, in run\n(EngineCore_DP0 pid=506800)     self._target(*self._args, **self._kwargs)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 796, in run_engine_core\n(EngineCore_DP0 pid=506800)     raise e\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 783, in run_engine_core\n(EngineCore_DP0 pid=506800)     engine_core = EngineCoreProc(*args, **kwargs)\n(EngineCore_DP0 pid=506800)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 555, in __init__\n(EngineCore_DP0 pid=506800)     super().__init__(\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 112, in __init__\n(EngineCore_DP0 pid=506800)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(\n(EngineCore_DP0 pid=506800)                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core.py\", line 223, in _initialize_kv_caches\n(EngineCore_DP0 pid=506800)     available_gpu_memory = self.model_executor.determine_available_memory()\n(EngineCore_DP0 pid=506800)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/executor/abstract.py\", line 88, in determine_available_memory\n(EngineCore_DP0 pid=506800)     return self.collective_rpc(\"determine_available_memory\")\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/executor/uniproc_executor.py\", line 74, in collective_rpc\n(EngineCore_DP0 pid=506800)     return [run_method(self.driver_worker, method, args, kwargs)]\n(EngineCore_DP0 pid=506800)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/utils/__init__.py\", line 2977, in run_method\n(EngineCore_DP0 pid=506800)     return func(*args, **kwargs)\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 120, in decorate_context\n(EngineCore_DP0 pid=506800)     return func(*args, **kwargs)\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/worker/gpu_worker.py\", line 280, in determine_available_memory\n(EngineCore_DP0 pid=506800)     self.model_runner.profile_run()\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/worker/gpu_model_runner.py\", line 3722, in profile_run\n(EngineCore_DP0 pid=506800)     hidden_states, last_hidden_states = self._dummy_run(\n(EngineCore_DP0 pid=506800)                                         ^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py\", line 120, in decorate_context\n(EngineCore_DP0 pid=506800)     return func(*args, **kwargs)\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/worker/gpu_model_runner.py\", line 3455, in _dummy_run\n(EngineCore_DP0 pid=506800)     outputs = self.model(\n(EngineCore_DP0 pid=506800)               ^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/compilation/cuda_graph.py\", line 126, in __call__\n(EngineCore_DP0 pid=506800)     return self.runnable(*args, **kwargs)\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl\n(EngineCore_DP0 pid=506800)     return self._call_impl(*args, **kwargs)\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl\n(EngineCore_DP0 pid=506800)     return forward_call(*args, **kwargs)\n(EngineCore_DP0 pid=506800)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen3.py\", line 321, in forward\n(EngineCore_DP0 pid=506800)     hidden_states = self.model(\n(EngineCore_DP0 pid=506800)                     ^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/compilation/decorators.py\", line 407, in __call__\n(EngineCore_DP0 pid=506800)     output = self.compiled_callable(*args, **kwargs)\n(EngineCore_DP0 pid=506800)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py\", line 745, in compile_wrapper\n(EngineCore_DP0 pid=506800)     raise e.with_traceback(None) from e.__cause__  # User compiler error\n(EngineCore_DP0 pid=506800)     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(EngineCore_DP0 pid=506800) torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped\n(EngineCore_DP0 pid=506800)   Explanation: Dynamo does not know how to trace the builtin `\u003cunknown module\u003e._SimpleCData.__new__.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).\n(EngineCore_DP0 pid=506800)   Hint: If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.\n(EngineCore_DP0 pid=506800)   Hint: If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.\n(EngineCore_DP0 pid=506800) \n(EngineCore_DP0 pid=506800)   Developer debug context: module: \u003cunknown module\u003e, qualname: _SimpleCData.__new__, skip reason: \u003cmissing reason\u003e\n(EngineCore_DP0 pid=506800) \n(EngineCore_DP0 pid=506800) \n(EngineCore_DP0 pid=506800) from user code:\n(EngineCore_DP0 pid=506800)    File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen2.py\", line 385, in forward\n(EngineCore_DP0 pid=506800)     hidden_states, residual = layer(positions, hidden_states, residual)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen3.py\", line 225, in forward\n(EngineCore_DP0 pid=506800)     hidden_states = self.self_attn(\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/models/qwen3.py\", line 144, in forward\n(EngineCore_DP0 pid=506800)     qkv, _ = self.qkv_proj(hidden_states)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/layers/linear.py\", line 582, in forward\n(EngineCore_DP0 pid=506800)     output_parallel = self.quant_method.apply(self, input_, bias)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/layers/quantization/gptq_bitblas.py\", line 479, in apply\n(EngineCore_DP0 pid=506800)     out = self.kernel.apply_gptq_bitblas_linear(layer, x)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/model_executor/layers/quantization/kernels/mixed_precision/bitblas.py\", line 315, in apply_gptq_bitblas_linear\n(EngineCore_DP0 pid=506800)     output = self.bitblas_matmul(*args)  # type: ignore[operator]\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/bitblas/ops/general_matmul/__init__.py\", line 756, in __call__\n(EngineCore_DP0 pid=506800)     return self.forward(*args, **kwds)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/bitblas/ops/general_matmul/__init__.py\", line 751, in forward\n(EngineCore_DP0 pid=506800)     self._forward_from_prebuild_lib(*args, stream=stream.cuda_stream)\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/bitblas/ops/operator.py\", line 459, in _forward_from_prebuild_lib\n(EngineCore_DP0 pid=506800)     ctypes.c_void_p(arr.data_ptr()) if not isinstance(arr, int) else arr for arr in args\n(EngineCore_DP0 pid=506800)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/torch/_dynamo/polyfills/__init__.py\", line 204, in instantiate_user_defined_class_object\n(EngineCore_DP0 pid=506800)     obj = cls.__new__(cls, *args, **kwargs)\n(EngineCore_DP0 pid=506800) \n(EngineCore_DP0 pid=506800) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS=\"+dynamo\"\n(EngineCore_DP0 pid=506800) \n[rank0]:[W1014 21:31:51.724849712 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())\n(APIServer pid=506583) Traceback (most recent call last):\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/bin/vllm\", line 10, in \u003cmodule\u003e\n(APIServer pid=506583)     sys.exit(main())\n(APIServer pid=506583)              ^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/entrypoints/cli/main.py\", line 73, in main\n(APIServer pid=506583)     args.dispatch_function(args)\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/entrypoints/cli/serve.py\", line 62, in cmd\n(APIServer pid=506583)     uvloop.run(run_server(args))\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/uvloop/__init__.py\", line 109, in run\n(APIServer pid=506583)     return __asyncio.run(\n(APIServer pid=506583)            ^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py\", line 195, in run\n(APIServer pid=506583)     return runner.run(main)\n(APIServer pid=506583)            ^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py\", line 118, in run\n(APIServer pid=506583)     return self._loop.run_until_complete(task)\n(APIServer pid=506583)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"uvloop/loop.pyx\", line 1518, in uvloop.loop.Loop.run_until_complete\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/.venv/lib/python3.12/site-packages/uvloop/__init__.py\", line 61, in wrapper\n(APIServer pid=506583)     return await main\n(APIServer pid=506583)            ^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/entrypoints/openai/api_server.py\", line 1917, in run_server\n(APIServer pid=506583)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/entrypoints/openai/api_server.py\", line 1933, in run_server_worker\n(APIServer pid=506583)     async with build_async_engine_client(\n(APIServer pid=506583)                ^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py\", line 210, in __aenter__\n(APIServer pid=506583)     return await anext(self.gen)\n(APIServer pid=506583)            ^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/entrypoints/openai/api_server.py\", line 191, in build_async_engine_client\n(APIServer pid=506583)     async with build_async_engine_client_from_engine_args(\n(APIServer pid=506583)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py\", line 210, in __aenter__\n(APIServer pid=506583)     return await anext(self.gen)\n(APIServer pid=506583)            ^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/entrypoints/openai/api_server.py\", line 238, in build_async_engine_client_from_engine_args\n(APIServer pid=506583)     async_llm = AsyncLLM.from_vllm_config(\n(APIServer pid=506583)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/utils/__init__.py\", line 1336, in inner\n(APIServer pid=506583)     return fn(*args, **kwargs)\n(APIServer pid=506583)            ^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/async_llm.py\", line 208, in from_vllm_config\n(APIServer pid=506583)     return cls(\n(APIServer pid=506583)            ^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/async_llm.py\", line 130, in __init__\n(APIServer pid=506583)     self.engine_core = EngineCoreClient.make_async_mp_client(\n(APIServer pid=506583)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core_client.py\", line 121, in make_async_mp_client\n(APIServer pid=506583)     return AsyncMPClient(*client_args)\n(APIServer pid=506583)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core_client.py\", line 807, in __init__\n(APIServer pid=506583)     super().__init__(\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/core_client.py\", line 468, in __init__\n(APIServer pid=506583)     with launch_core_engines(vllm_config, executor_class, log_stats) as (\n(APIServer pid=506583)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n(APIServer pid=506583)   File \"/home/lixiangyu/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/contextlib.py\", line 144, in __exit__\n(APIServer pid=506583)     next(self.gen)\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/utils.py\", line 816, in launch_core_engines\n(APIServer pid=506583)     wait_for_engine_startup(\n(APIServer pid=506583)   File \"/home/lixiangyu/repos/xxxxyu/vllm/vllm/v1/engine/utils.py\", line 873, in wait_for_engine_startup\n(APIServer pid=506583)     raise RuntimeError(\n(APIServer pid=506583) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}\n```\n\u003c/details\u003e\n\nWhen setting `enforce-eager=True`, there is no such error — it seems BitBLAS's compiling conflicts with vLLM's torch.compile integration?\n\nIf so, what is the best practice for running vLLM + BitBLAS?","author":{"url":"https://github.com/xxxxyu","@type":"Person","name":"xxxxyu"},"datePublished":"2025-10-14T14:05:23.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/315/BitBLAS/issues/315"}