René's URL Explorer Experiment


Title: [Dev] Potentially improve performance through block reduction by LeiWang1999 · Pull Request #63 · microsoft/BitBLAS · GitHub

Open Graph Title: [Dev] Potentially improve performance through block reduction by LeiWang1999 · Pull Request #63 · microsoft/BitBLAS

X Title: [Dev] Potentially improve performance through block reduction by LeiWang1999 · Pull Request #63 · microsoft/BitBLAS

Description: In our recent evaluations, we observed that batch inference on matrix multiplication shapes for 7b/13b models didn't achieve the expected theoretical speedup. This performance bottleneck appears to be linked to the absence of block reduction support within our schedule template and tvm. This pull request includes a variety of changes across multiple files, mainly focusing on introducing new features related to block reduction. Here are the most important changes: * python/bitblas/base/roller/policy/tensorcore.py: The block_reduction_depth field was added to the Policy class, and this value is now considered in several functions, including _check_small_tile, _enlarge, and _score. [1] [2] [3] [4] [5] * python/bitblas/gpu/matmul_analysis.py: The check_last_trait function was updated to set the block_reduction_depth to 2 for small M values. * python/bitblas/gpu/matmul_mma.py: The apply_config function was updated to call a new function apply_block_reduction_with_config if block_reduction_depth is not None. * 3rdparty/tvm: The subproject commit was updated.

Open Graph Description: In our recent evaluations, we observed that batch inference on matrix multiplication shapes for 7b/13b models didn't achieve the expected theoretical speedup. This performance bottleneck appear...

X Description: In our recent evaluations, we observed that batch inference on matrix multiplication shapes for 7b/13b models didn't achieve the expected theoretical speedup. This performance bottleneck ap...

Opengraph URL: https://github.com/microsoft/BitBLAS/pull/63

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/commits/:range(.:format)
route-controllerpull_requests
route-actioncommits
fetch-noncev2:2525f8d4-c77a-b3f6-d43a-d8c9e88cd432
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idE6CE:17A025:36D8F12:48F003C:6A655A4B
html-safe-noncef63b36c3f930ec2060afa4abaf8541d953bf44c35436769a407cb9a5423ac5da
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFNkNFOjE3QTAyNTozNkQ4RjEyOjQ4RjAwM0M6NkE2NTVBNEIiLCJ2aXNpdG9yX2lkIjoiNzczODcwNTYwMjIzMjUzOTcyMyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacf75e0fd257ab660bd40921e0c71ee8ce5461c47eee7719416361e6e4fae43aed
hovercard-subject-tagpull_request:1945905558
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/commits
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
twitter:imagehttps://avatars.githubusercontent.com/u/34334180?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/34334180?s=400&v=4
og:image:altIn our recent evaluations, we observed that batch inference on matrix multiplication shapes for 7b/13b models didn't achieve the expected theoretical speedup. This performance bottleneck appear...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None52c76df668885aaff23b50bdca1fa1ea44ac9c1553e888ebc70ff1e4daa4625b
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/microsoft/BitBLAS git https://github.com/microsoft/BitBLAS.git
octolytics-dimension-user_id6154722
octolytics-dimension-user_loginmicrosoft
octolytics-dimension-repository_id762181983
octolytics-dimension-repository_nwomicrosoft/BitBLAS
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id762181983
octolytics-dimension-repository_network_root_nwomicrosoft/BitBLAS
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release309153364422b3c499922d1a2a6404910a58ed8e
ui-targetcanary-1
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FBitBLAS%2Fpull%2F63%2Fcommits%2F00bfa319caa321c0e946d5f87d374982beea263e
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub Copilot appDirect agents from issue to mergehttps://github.com/features/ai/github-app
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
Code QualityEnforce quality at mergehttps://github.com/features/code-quality
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
View all resourceshttps://github.com/resources
GitHub SponsorsFund open source developershttps://github.com/open-source/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/open-source/accelerator
GitHub Starshttps://stars.github.com
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FBitBLAS%2Fpull%2F63%2Fcommits%2F00bfa319caa321c0e946d5f87d374982beea263e
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Fcommits&source=header-repo&source_repo=microsoft%2FBitBLAS
Reloadhttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
Reloadhttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
Reloadhttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
Please reload this pagehttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
microsoft https://github.com/microsoft
BitBLAShttps://github.com/microsoft/BitBLAS
Notifications https://github.com/login?return_to=%2Fmicrosoft%2FBitBLAS
Fork 60 https://github.com/login?return_to=%2Fmicrosoft%2FBitBLAS
Star 769 https://github.com/login?return_to=%2Fmicrosoft%2FBitBLAS
Code https://github.com/microsoft/BitBLAS
Issues 33 https://github.com/microsoft/BitBLAS/issues
Pull requests 6 https://github.com/microsoft/BitBLAS/pulls
Discussions https://github.com/microsoft/BitBLAS/discussions
Actions https://github.com/microsoft/BitBLAS/actions
Projects https://github.com/microsoft/BitBLAS/projects
Models https://github.com/microsoft/BitBLAS/models
Security and quality 0 https://github.com/microsoft/BitBLAS/security
Insights https://github.com/microsoft/BitBLAS/pulse
Code https://github.com/microsoft/BitBLAS
Issues https://github.com/microsoft/BitBLAS/issues
Pull requests https://github.com/microsoft/BitBLAS/pulls
Discussions https://github.com/microsoft/BitBLAS/discussions
Actions https://github.com/microsoft/BitBLAS/actions
Projects https://github.com/microsoft/BitBLAS/projects
Models https://github.com/microsoft/BitBLAS/models
Security and quality https://github.com/microsoft/BitBLAS/security
Insights https://github.com/microsoft/BitBLAS/pulse
LeiWang1999https://github.com/LeiWang1999
microsoft:mainhttps://github.com/microsoft/BitBLAS/tree/main
Conversation 1 https://github.com/microsoft/BitBLAS/pull/63
Commits 38 https://github.com/microsoft/BitBLAS/pull/63/commits
Checks 0 https://github.com/microsoft/BitBLAS/pull/63/checks
Files changed https://github.com/microsoft/BitBLAS/pull/63/files
Please reload this pagehttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
[Dev] Potentially improve performance through block reduction https://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e#top
Show all changes 38 commits https://github.com/microsoft/BitBLAS/pull/63/files
75d2f3d improve e4m3 decoding. May 21, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/75d2f3d5c2713064944fea9f6f0372d966b0e1d1
dd744d0 Merge branch 'main' of https://github.com/microsoft/BitBLAS into main May 23, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/dd744d0d0c455005661b4afe0dd4e4c466b2fcd9
00bfa31 append fp16xint1 May 25, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
8cd8b10 Update submodule commit reference Jun 1, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/8cd8b10c18e3f484a3273f6b6bda498bb022c783
9122ff7 chore: Update shared memory scope for float32 output dtype Jun 1, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/9122ff7ca0e07e53b120940dae0fee434a6b0e08
b508acc BUGFIX: UINT8/INT8 Decoding Jun 2, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/b508acc60fa650bac5eb4be2a9277a2c9bd8b08f
58d55b7 feat: Add rasterization options for roller module Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/58d55b7cb750bd0678985a30b532929196765d13
e7547ce Refactor tensorcore_legalization method to optimize tensor core usage Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/e7547ced6f222de7668a2963f190471791aa0be3
678a2e1 feat: Add function to collect variables from expression, improve for … Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/678a2e15fc10c2aeb0a2ae8e16e685cc5bb102f8
3088b35 chore: Update typing import in __init__.py Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/3088b35b90042b00fba9a5300f89405368606174
5d206b3 chore: Refactor CPU execution of operators Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/5d206b3dae4e240571b8bf161d410b52754f83e7
e06ce10 Refactor matmul implementation for splitk layout Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/e06ce10fe19d03321b80ccae6e6ee5bbc9b03dab
d67cc6d Refactor matmul implementation for splitk layout Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/d67cc6d58a592852f8f18e2f9fb062a52d43899b
9e36b6d Refactor matmul implementation for splitk layout Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/9e36b6dd4a62442c09674b76b49cc69607389d45
e1a0149 chore: Update version to 0.0.1.dev8 Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/e1a01496f6a59054d2904bcf51d70759b755b143
df0ed7a chore: Enable debug output in bitblas.set_debug_level() Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/df0ed7a34b76b717ed5366fa384568b15b6386ec
a0f651a Refactor Linear module matmul implementation for splitk layout Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/a0f651a3ba2c8146f06dc21b6aba48443fe3b956
88295a7 Refactor matmul implementation for splitk layout Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/88295a73deff9b5b61d8865627316c8ed33dd499
3366dce Merge branch 'main' of https://github.com/microsoft/BitBLAS into lei/… Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/3366dced1bee3e03410abc3f27e8aa678a9cad41
25b5c63 Refactor CUDA kernel launch string for dynamic symbolic set Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/25b5c63b6e4978f949aba7c87bdc3dc850024032
26a9f1b Bumpt version to v0.0.1.dev9 Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/26a9f1b195039ad36ad92f7ceffd1589911d4eba
251bf08 Merge branch 'main' of https://github.com/microsoft/BitBLAS into lei/… Jun 5, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/251bf0857a31d6879dccc216bf6f25a2c4dbd831
e0cf62c Refactor CUDA kernel launch string for dynamic symbolic set Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/e0cf62c6d71a955def7508ee5324d56683bc30da
2e4e8dd Bump version to v0.0.1.dev10 Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/2e4e8ddb00c498358913b928884ea902067cb617
0dec7d8 Merge branch 'main' into lei/splitk LeiWang1999 Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/0dec7d8ad13422c0063273d2e4213fe3fd874425
81f5b9a Refactor CUDA kernel launch string for dynamic symbolic set Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/81f5b9a6c92784195a387217498ce05dd865a6ea
ec64f91 Merge branch 'lei/splitk' of https://github.com/LeiWang1999/MSBitBLAS… Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/ec64f91acde273aca6de62d97e61fc9daf537d11
5e71163 Bump version to v0.0.1.dev12 and add MatmulConfigWithSplitK and Matmu… Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/5e71163af16a61d9c68f2624e330403eee7d39d1
d0e0726 Merge branch 'main' into lei/splitk LeiWang1999 Jun 6, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/d0e07269928f3fcd77ff6f900738e7df128cc27f
30c0ae7 fix the typo Jun 29, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/30c0ae7de79e03774015d729fb57a8e9944ec7bf
4bbccae Refactor CUDA kernel launch string for dynamic symbolic set Jun 29, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/4bbccaed2ce65dc9850cd9d59bc0a6aeb5be31a0
0d1b649 Refactor CUDA kernel launch string for dynamic symbolic set Jun 29, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/0d1b6491be046f41aacde78a5995d35fd72ff1ca
2ce41bb Refactor CUDA kernel launch string for dynamic symbolic set Jun 29, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/2ce41bba3a49945c0fc440920e03cb4f90d8fd49
866f561 Refactor CUDA kernel launch string for dynamic symbolic set Jun 30, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/866f561aa73344aa9ff15ed9bba9b626429c230c
8d2393c Merge branch 'main' of https://github.com/microsoft/BitBLAS into lei/… Jun 30, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/8d2393c76c7dec8dcf68685cd00a976e1e935267
22c12d7 Merge branch 'microsoft:main' into main LeiWang1999 Jun 30, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/22c12d79a00e0882711de6eca7e84798561f6d15
d9fdc21 Merge branch 'main' of https://github.com/LeiWang1999/MSBitBLAS into … Jun 30, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/d9fdc21238c1ad0d6765f8bdb18c3eecaa112da3
1e534f4 Merge branch 'lei/splitk' of https://github.com/LeiWang1999/MSBitBLAS… Jun 30, 2024 https://github.com/microsoft/BitBLAS/pull/63/commits/1e534f4f2ca0303950f060d600198ac9378635f5
Clear filters https://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
Please reload this pagehttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
Please reload this pagehttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
Prev https://github.com/microsoft/BitBLAS/pull/63/commits/dd744d0d0c455005661b4afe0dd4e4c466b2fcd9
Next https://github.com/microsoft/BitBLAS/pull/63/commits/8cd8b10c18e3f484a3273f6b6bda498bb022c783
Please reload this pagehttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e
python/bitblas/gpu/intrin/lop3.pyhttps://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e#diff-35154465a7819e5c425cf4ae4ee62bf48f98d44ee018e779c7621291858077e6
View file https://github.com/microsoft/BitBLAS/blob/00bfa319caa321c0e946d5f87d374982beea263e/python/bitblas/gpu/intrin/lop3.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/microsoft/BitBLAS/pull/63/commits/{{ revealButtonHref }}
https://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e#diff-35154465a7819e5c425cf4ae4ee62bf48f98d44ee018e779c7621291858077e6
https://github.com/microsoft/BitBLAS/pull/63/commits/00bfa319caa321c0e946d5f87d374982beea263e#diff-35154465a7819e5c425cf4ae4ee62bf48f98d44ee018e779c7621291858077e6
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.