René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Low GEMM Performance on Hopper GPU with Small M Shapes","articleBody":"Hi,\n\nThank you for the great library! I’m observing some unexpected performance with GEMM on Hopper GPUs when using small M dimensions. I followed the example in [example14_autotune.py](https://github.com/NVIDIA/nvmath-python/blob/main/examples/linalg/advanced/matmul/example14_autotune.py).\n\nCompared to the PyTorch implementation, the performance is significantly lower — around 30% of the expected TFLOPS and memory bandwidth utilization.\n\n\u003cimg width=\"565\" alt=\"Image\" src=\"https://github.com/user-attachments/assets/d5a87505-8f0b-4426-a522-a2dac97c9b96\" /\u003e\n\nNot sure I am correctly using the API — I would greatly appreciate any suggestions.\n\nEnvironment:\n\t•\tGPU: H200\n\t•\tCUDA: 12.8\n\t•\tPyTorch: 2.6.0\n\t•\tnvmath-python: 0.3.0\n\nBenchmark code:\n\n```python\nimport argparse\n\nimport torch\nimport nvmath\nfrom triton.testing import do_bench\n\n\ndef profile(m, n, k, dtype):\n    device = torch.device(\"cuda\")\n    assert isinstance(device, torch.device)\n\n    X = torch.randn(m, k, device=device, dtype=dtype)\n    Y = torch.randn(n, k, device=device, dtype=dtype)\n    \n    _torch_gemm = lambda: torch.matmul(X, Y)\n    \n    mm = nvmath.linalg.advanced.Matmul(X, Y)\n    \n    mm.plan(preferences={\"limit\":1000})\n    mm.autotune(iterations=1000)\n    \n    # print(mm.algorithms[0].capabilities)\n    _nvmath_gemm = lambda: mm.execute()\n    \n    t_torch = do_bench(_torch_gemm)\n    t_nvmath = do_bench(_nvmath_gemm)\n    \n    return t_torch, t_nvmath\n\n\nif __name__ == \"__main__\":\n    parser = argparse.ArgumentParser(description=\"GEMM profile\")\n    parser.add_argument(\"--m\", type=int, default=4096)\n    parser.add_argument(\"--n\", type=int, default=4096)\n    parser.add_argument(\"--k\", type=int, default=4096)\n    args = parser.parse_args()\n\n    print(\"Provider,Operation,dtype,m,n,k,Runtime,GB/s,GFLOPs\")\n    \n    for dtype in [torch.float16, torch.bfloat16]:\n        t_torch, t_nvmath = profile(args.m, args.n, args.k, dtype)\n\n        m = args.m\n        n = args.n\n        k = args.k\n\n        torch_mem_bd = 2 * (m * n + n * k + m * k) * 1e3 / t_torch / 1e9\n        torch_gflops = 2 * m * n * k * 1e3 / t_torch / 1e9\n        nv_mem_bd = 2 * (m * n + n * k + m * k) * 1e3 / t_nvmath / 1e9\n        nv_gflops = 2 * m * n * k * 1e3 / t_nvmath / 1e9\n        \n        print(f\"TORCH,0,{dtype},{args.m},{args.n},{args.k},{t_torch},{torch_mem_bd},{torch_gflops}\")\n        print(f\"NVMATH,0,{dtype},{args.m},{args.n},{args.k},{t_nvmath},{nv_mem_bd},{nv_gflops}\")\n```\n","author":{"url":"https://github.com/happierpig","@type":"Person","name":"happierpig"},"datePublished":"2025-03-28T00:22:34.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/21/nvmath-python/issues/21"}