René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[Help Wanted] Optimize binary matmul kernel","articleBody":"The [current kernel](https://github.com/FasterDecoding/BitDelta/blob/main/bitdelta/binary_gemm_kernel.py#L297) is fairly slow compared to the theoretical optimum, considering the small memory footprint of weight deltas. So right now it functions more as as proof of concept (eg. it outperforms naive simultaneous inference). Can expect an additional 4-8x latency improvement if further optimized. \r\n\r\nI don't have much kernel optimization experience yet, though - if anyone in the OSS community is interested, would love some help! \r\n\r\n\r\nAfterwards, it'd be super interesting to run some benchmarks against LoRA-based multi-tenant systems like Punica/S-LoRA.","author":{"url":"https://github.com/chromecast56","@type":"Person","name":"chromecast56"},"datePublished":"2024-03-03T21:07:20.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/5/BitDelta/issues/5"}