René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Speed Comparison: BitLinear and nn.Linear","articleBody":"Hello,\r\n\r\nI measured the time of your BitLinear and BitLinearBitBLAS against nn.Linear, and it seems that the time for smaller input_features and out_features is slower than nn.Linear. Is there a solution for this?\r\n\r\nI used the quant_utils from your BitNet integration: https://github.com/microsoft/BitBLAS/blob/main/integration/BitNet/utils_quant.py\r\n\r\nMy GPU is NVIDIA GeForce RTX 3090\r\n\r\n```\r\nimport time\r\nimport torch\r\nimport torch.nn as nn\r\nfrom torch.autograd import Variable\r\n\r\n# from bitlinear import BitLinear, BitLinearBitBLAS\r\nfrom utils_quant import BitLinear, BitLinearBitBLAS\r\n\r\n# Function to measure computation time\r\ndef measure_time(layer, input_tensor, num_runs=100):\r\n    with torch.no_grad():\r\n        # Warm up\r\n        for _ in range(100): ## 100\r\n            _ = layer(input_tensor)\r\n        \r\n        start_time = time.time()\r\n        for _ in range(num_runs):\r\n            _ = layer(input_tensor)\r\n        torch.cuda.synchronize()  #### new\r\n        end_time = time.time()\r\n        \r\n        avg_time = (end_time - start_time) / num_runs\r\n    return avg_time\r\n\r\n# # # Test parameters\r\ninput_features = 512\r\noutput_features = 256\r\nbatch_size = 8\r\n\r\n# input_features = 1024\r\n# output_features = 512\r\n# batch_size = 32\r\n\r\n# input_features = 10240\r\n# output_features = 5120\r\n# batch_size = 32\r\n\r\n# input_features = 20480\r\n# output_features = 10240\r\n# batch_size = 32\r\n\r\n\r\n# Create random input tensor\r\ninput_tensor = torch.randn(batch_size, input_features).cuda()\r\n\r\n# Initialize layers\r\nnn_linear_layer = nn.Linear(input_features, output_features).cuda()\r\nbit_linear_layer = BitLinear(input_features, output_features).cuda()\r\n\r\nbitblas_linear_layer = BitLinearBitBLAS.from_bit_linear(bit_linear_layer)\r\n\r\n# Measure computation time\r\nnum_runs = 100\r\nnn_linear_time = measure_time(nn_linear_layer, input_tensor, num_runs)\r\nbit_linear_time = measure_time(bit_linear_layer, input_tensor, num_runs)\r\nbitblas_linear_time = measure_time(bitblas_linear_layer, input_tensor, num_runs)\r\n\r\nprint('input_features, output_features, batch_size: ', input_features, output_features, batch_size)\r\nprint(f\"Average computation time for nn.Linear: {nn_linear_time * 1000:.4f} ms\")\r\nprint(f\"Average computation time for fp32 simulated BitLinear: {bit_linear_time * 1000:.4f} ms\")\r\nprint(f\"Average computation time Bitblas BitLinear: {bitblas_linear_time * 1000:.4f} ms\")\r\n\r\n\r\n```\r\n\r\nHere are the testing results:\r\n\r\n```\r\ninput_features, output_features, batch_size:  512 256 8\r\nAverage computation time for nn.Linear: 0.0230 ms\r\nAverage computation time for fp32 simulated BitLinear: 0.3450 ms\r\nAverage computation time Bitblas BitLinear: 0.3091 ms\r\n\r\ninput_features, output_features, batch_size:  1024 512 32\r\nAverage computation time for nn.Linear: 0.0265 ms\r\nAverage computation time for fp32 simulated BitLinear: 0.3427 ms\r\nAverage computation time Bitblas BitLinear: 0.3137 ms\r\n\r\ninput_features, output_features, batch_size:  10240 5120 32\r\nAverage computation time for nn.Linear: 0.5421 ms\r\nAverage computation time for fp32 simulated BitLinear: 6.3314 ms\r\nAverage computation time Bitblas BitLinear: 0.3170 ms\r\n\r\ninput_features, output_features, batch_size:  20480 10240 32\r\nAverage computation time for nn.Linear: 2.1726 ms\r\nAverage computation time for fp32 simulated BitLinear: 25.2509 ms\r\nAverage computation time Bitblas BitLinear: 0.5633 ms\r\n```\r\n\r\nThanks for your reply in advance.","author":{"url":"https://github.com/ZiqingChang","@type":"Person","name":"ZiqingChang"},"datePublished":"2024-08-01T13:19:30.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":12},"url":"https://github.com/118/BitBLAS/issues/118"}