René's URL Explorer Experiment

Title: GitHub - samadejacobs/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. · GitHub

Open Graph Title: GitHub - samadejacobs/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

X Title: GitHub - samadejacobs/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - samadejacobs/DeepSpeed

Open Graph Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - samadejacobs/DeepSpeed

X Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - samadejacobs/DeepSpeed

Mail addresses
opencode@microsoft.com

Opengraph URL: https://github.com/samadejacobs/DeepSpeed

X: @github

direct link

Domain: github.com

route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:26c74cee-227f-980c-5f11-6ac565fd73cf
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	9BD6:12297D:5058EF:6CB88B:6A4D3DCE
html-safe-nonce	7074ab3b84240247b79e410f78fc74f2ff602a3780c014adbe8e48b17f7f0a10
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5QkQ2OjEyMjk3RDo1MDU4RUY6NkNCODhCOjZBNEQzRENFIiwidmlzaXRvcl9pZCI6IjY0MjE1Mjc3NDIwMTgwNDMzNDIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac	cb1b84531eecc43504501c1aee951e49822edde30a4424e73a7defa56967d480
hovercard-subject-tag	repository:508411488
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/samadejacobs/DeepSpeed
twitter:image	https://opengraph.githubassets.com/833afe2fb91999facb09646f085f330e5b2b02dfc738744bc501c02e081cebb5/samadejacobs/DeepSpeed
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/833afe2fb91999facb09646f085f330e5b2b02dfc738744bc501c02e081cebb5/samadejacobs/DeepSpeed
og:image:alt	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - samadejacobs/DeepSpeed
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	92571a8944142227b7e19cd10918b1ddd06e5066c1ad5bc7e4769cf6140a87e6
turbo-cache-control	no-cache
go-import	github.com/samadejacobs/DeepSpeed git https://github.com/samadejacobs/DeepSpeed.git
octolytics-dimension-user_id	16696152
octolytics-dimension-user_login	samadejacobs
octolytics-dimension-repository_id	508411488
octolytics-dimension-repository_nwo	samadejacobs/DeepSpeed
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	235860204
octolytics-dimension-repository_parent_nwo	deepspeedai/DeepSpeed
octolytics-dimension-repository_network_root_id	235860204
octolytics-dimension-repository_network_root_nwo	deepspeedai/DeepSpeed
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	56fc8347865a14e2ec811533d68f929cf4e0ec19
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/samadejacobs/DeepSpeed#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsamadejacobs%2FDeepSpeed
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub Copilot appDirect agents from issue to merge	https://github.com/features/ai/github-app
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
View all resources	https://github.com/resources
GitHub SponsorsFund open source developers	https://github.com/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/accelerator
GitHub Stars	https://stars.github.com
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsamadejacobs%2FDeepSpeed
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=samadejacobs%2FDeepSpeed
Reload	https://github.com/samadejacobs/DeepSpeed
Reload	https://github.com/samadejacobs/DeepSpeed
Reload	https://github.com/samadejacobs/DeepSpeed
samadejacobs	https://github.com/samadejacobs
DeepSpeed	https://github.com/samadejacobs/DeepSpeed
deepspeedai/DeepSpeed	https://github.com/deepspeedai/DeepSpeed
Notifications	https://github.com/login?return_to=%2Fsamadejacobs%2FDeepSpeed
Fork 0	https://github.com/login?return_to=%2Fsamadejacobs%2FDeepSpeed
Star 1	https://github.com/login?return_to=%2Fsamadejacobs%2FDeepSpeed
Code	https://github.com/samadejacobs/DeepSpeed
Pull requests 0	https://github.com/samadejacobs/DeepSpeed/pulls
Actions	https://github.com/samadejacobs/DeepSpeed/actions
Projects	https://github.com/samadejacobs/DeepSpeed/projects
Security and quality 0	https://github.com/samadejacobs/DeepSpeed/security
Insights	https://github.com/samadejacobs/DeepSpeed/pulse
Code	https://github.com/samadejacobs/DeepSpeed
Pull requests	https://github.com/samadejacobs/DeepSpeed/pulls
Actions	https://github.com/samadejacobs/DeepSpeed/actions
Projects	https://github.com/samadejacobs/DeepSpeed/projects
Security and quality	https://github.com/samadejacobs/DeepSpeed/security
Insights	https://github.com/samadejacobs/DeepSpeed/pulse
	https://github.com/samadejacobs/DeepSpeed
Branches	https://github.com/samadejacobs/DeepSpeed/branches
Tags	https://github.com/samadejacobs/DeepSpeed/tags
	https://github.com/samadejacobs/DeepSpeed/branches
	https://github.com/samadejacobs/DeepSpeed/tags
2,409 Commits	https://github.com/samadejacobs/DeepSpeed/commits/master/
	https://github.com/samadejacobs/DeepSpeed/commits/master/
.github	https://github.com/samadejacobs/DeepSpeed/tree/master/.github
.github	https://github.com/samadejacobs/DeepSpeed/tree/master/.github
accelerator	https://github.com/samadejacobs/DeepSpeed/tree/master/accelerator
accelerator	https://github.com/samadejacobs/DeepSpeed/tree/master/accelerator
azure	https://github.com/samadejacobs/DeepSpeed/tree/master/azure
azure	https://github.com/samadejacobs/DeepSpeed/tree/master/azure
benchmarks	https://github.com/samadejacobs/DeepSpeed/tree/master/benchmarks
benchmarks	https://github.com/samadejacobs/DeepSpeed/tree/master/benchmarks
bin	https://github.com/samadejacobs/DeepSpeed/tree/master/bin
bin	https://github.com/samadejacobs/DeepSpeed/tree/master/bin
blogs	https://github.com/samadejacobs/DeepSpeed/tree/master/blogs
blogs	https://github.com/samadejacobs/DeepSpeed/tree/master/blogs
csrc	https://github.com/samadejacobs/DeepSpeed/tree/master/csrc
csrc	https://github.com/samadejacobs/DeepSpeed/tree/master/csrc
deepspeed	https://github.com/samadejacobs/DeepSpeed/tree/master/deepspeed
deepspeed	https://github.com/samadejacobs/DeepSpeed/tree/master/deepspeed
docker	https://github.com/samadejacobs/DeepSpeed/tree/master/docker
docker	https://github.com/samadejacobs/DeepSpeed/tree/master/docker
docs	https://github.com/samadejacobs/DeepSpeed/tree/master/docs
docs	https://github.com/samadejacobs/DeepSpeed/tree/master/docs
examples	https://github.com/samadejacobs/DeepSpeed/tree/master/examples
examples	https://github.com/samadejacobs/DeepSpeed/tree/master/examples
op_builder	https://github.com/samadejacobs/DeepSpeed/tree/master/op_builder
op_builder	https://github.com/samadejacobs/DeepSpeed/tree/master/op_builder
release	https://github.com/samadejacobs/DeepSpeed/tree/master/release
release	https://github.com/samadejacobs/DeepSpeed/tree/master/release
requirements	https://github.com/samadejacobs/DeepSpeed/tree/master/requirements
requirements	https://github.com/samadejacobs/DeepSpeed/tree/master/requirements
scripts	https://github.com/samadejacobs/DeepSpeed/tree/master/scripts
scripts	https://github.com/samadejacobs/DeepSpeed/tree/master/scripts
tests	https://github.com/samadejacobs/DeepSpeed/tree/master/tests
tests	https://github.com/samadejacobs/DeepSpeed/tree/master/tests
.clang-format	https://github.com/samadejacobs/DeepSpeed/blob/master/.clang-format
.clang-format	https://github.com/samadejacobs/DeepSpeed/blob/master/.clang-format
.flake8	https://github.com/samadejacobs/DeepSpeed/blob/master/.flake8
.flake8	https://github.com/samadejacobs/DeepSpeed/blob/master/.flake8
.gitignore	https://github.com/samadejacobs/DeepSpeed/blob/master/.gitignore
.gitignore	https://github.com/samadejacobs/DeepSpeed/blob/master/.gitignore
.gitmodules	https://github.com/samadejacobs/DeepSpeed/blob/master/.gitmodules
.gitmodules	https://github.com/samadejacobs/DeepSpeed/blob/master/.gitmodules
.pre-commit-config.yaml	https://github.com/samadejacobs/DeepSpeed/blob/master/.pre-commit-config.yaml
.pre-commit-config.yaml	https://github.com/samadejacobs/DeepSpeed/blob/master/.pre-commit-config.yaml
.pylintrc	https://github.com/samadejacobs/DeepSpeed/blob/master/.pylintrc
.pylintrc	https://github.com/samadejacobs/DeepSpeed/blob/master/.pylintrc
.readthedocs.yml	https://github.com/samadejacobs/DeepSpeed/blob/master/.readthedocs.yml
.readthedocs.yml	https://github.com/samadejacobs/DeepSpeed/blob/master/.readthedocs.yml
.style.yapf	https://github.com/samadejacobs/DeepSpeed/blob/master/.style.yapf
.style.yapf	https://github.com/samadejacobs/DeepSpeed/blob/master/.style.yapf
CODEOWNERS	https://github.com/samadejacobs/DeepSpeed/blob/master/CODEOWNERS
CODEOWNERS	https://github.com/samadejacobs/DeepSpeed/blob/master/CODEOWNERS
CODE_OF_CONDUCT.md	https://github.com/samadejacobs/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CODE_OF_CONDUCT.md	https://github.com/samadejacobs/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CONTRIBUTING.md	https://github.com/samadejacobs/DeepSpeed/blob/master/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/samadejacobs/DeepSpeed/blob/master/CONTRIBUTING.md
LICENSE	https://github.com/samadejacobs/DeepSpeed/blob/master/LICENSE
LICENSE	https://github.com/samadejacobs/DeepSpeed/blob/master/LICENSE
MANIFEST.in	https://github.com/samadejacobs/DeepSpeed/blob/master/MANIFEST.in
MANIFEST.in	https://github.com/samadejacobs/DeepSpeed/blob/master/MANIFEST.in
MANIFEST_win.in	https://github.com/samadejacobs/DeepSpeed/blob/master/MANIFEST_win.in
MANIFEST_win.in	https://github.com/samadejacobs/DeepSpeed/blob/master/MANIFEST_win.in
README.md	https://github.com/samadejacobs/DeepSpeed/blob/master/README.md
README.md	https://github.com/samadejacobs/DeepSpeed/blob/master/README.md
SECURITY.md	https://github.com/samadejacobs/DeepSpeed/blob/master/SECURITY.md
SECURITY.md	https://github.com/samadejacobs/DeepSpeed/blob/master/SECURITY.md
build_win.bat	https://github.com/samadejacobs/DeepSpeed/blob/master/build_win.bat
build_win.bat	https://github.com/samadejacobs/DeepSpeed/blob/master/build_win.bat
environment.yml	https://github.com/samadejacobs/DeepSpeed/blob/master/environment.yml
environment.yml	https://github.com/samadejacobs/DeepSpeed/blob/master/environment.yml
install.sh	https://github.com/samadejacobs/DeepSpeed/blob/master/install.sh
install.sh	https://github.com/samadejacobs/DeepSpeed/blob/master/install.sh
setup.cfg	https://github.com/samadejacobs/DeepSpeed/blob/master/setup.cfg
setup.cfg	https://github.com/samadejacobs/DeepSpeed/blob/master/setup.cfg
setup.py	https://github.com/samadejacobs/DeepSpeed/blob/master/setup.py
setup.py	https://github.com/samadejacobs/DeepSpeed/blob/master/setup.py
version.txt	https://github.com/samadejacobs/DeepSpeed/blob/master/version.txt
version.txt	https://github.com/samadejacobs/DeepSpeed/blob/master/version.txt
README	https://github.com/samadejacobs/DeepSpeed
Code of conduct	https://github.com/samadejacobs/DeepSpeed
Contributing	https://github.com/samadejacobs/DeepSpeed
Apache-2.0 license	https://github.com/samadejacobs/DeepSpeed
Security	https://github.com/samadejacobs/DeepSpeed
	https://github.com/Microsoft/DeepSpeed/blob/master/LICENSE
	https://pypi.org/project/deepspeed/
	https://pepy.tech/project/deepspeed
	https://github.com/samadejacobs/DeepSpeed#build-pipeline-status
	https://twitter.com/intent/follow?screen_name=MSFTDeepSpeed
	https://twitter.com/MSFTDeepSpeedJP
	https://www.zhihu.com/people/deepspeed
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only
	https://github.com/samadejacobs/DeepSpeed#latest-news
learn how	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat
DeepNVMe: Improving DL Applications through I/O Optimizations	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-gds/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-gds/japanese/README.md
DeepSpeed Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ucp/README.md
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ucp/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ucp/japanese/README.md
DeepSpeed-FP6:The power of FP6-Centric Serving for Large Language Models	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fp6/03-05-2024
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fp6/03-05-2024/README.md
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fp6/03-05-2024/README-Chinese.md
DeepSpeed-FastGen: Introducing Mixtral, Phi-2, and Falcon support with major performance and feature enhancements.	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/2024-01-19
Llama 2 Inference on 4th Gen Intel® Xeon® Scalable Processor with DeepSpeed	https://github.com/microsoft/DeepSpeed/tree/master/blogs/intel-inference
[Intel version]	https://www.intel.com/content/www/us/en/developer/articles/technical/xllama-2-on-xeon-scalable-processor-with-deepspeed.html
DeepSpeed ZeRO-Offload++: 6x Higher Training Throughput via Collaborative CPU/GPU Twin-Flow	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-offloadpp
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/japanese/README.md
DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-visualchat/10-03-2023/README.md
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-visualchat/10-03-2023/README.md
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README-Chinese.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README-Japanese.md
DeepSpeed4Science website	https://deepspeed4science.ai/
Tutorials	https://www.deepspeed.ai/deepspeed4science/
White paper	https://arxiv.org/abs/2310.04610
Blog	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/japanese/README.md
DeepSpeed ZeRO-Inference: 20x faster inference through weight quantization and KV cache offloading	https://github.com/microsoft/DeepSpeedExamples/blob/master/inference/huggingface/zero_inference/README.md
DeepSpeed-Chat: Llama/Llama-2 system support, efficiency boost, and training stability improvements	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat/ds-chat-release-8-31/README.md
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ulysses
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/japanese/README.md
ZeRO++: A leap in speed for LLM and chat model training with 4X less communication	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
English	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/japanese/README.md
	https://github.com/samadejacobs/DeepSpeed#extreme-speed-and-scale-for-dl-training-and-inference
DeepSpeed	https://www.deepspeed.ai/
MT-530B	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
BLOOM	https://huggingface.co/blog/bloom-megatron-deepspeed
	https://github.com/samadejacobs/DeepSpeed#deepspeeds-four-innovation-pillars
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/DeepSpeed-pillars.png
	https://github.com/samadejacobs/DeepSpeed#deepspeed-training
DeepSpeed-Training	https://www.deepspeed.ai/training/
	https://github.com/samadejacobs/DeepSpeed#deepspeed-inference
DeepSpeed-Inference	https://www.deepspeed.ai/inference
	https://github.com/samadejacobs/DeepSpeed#deepspeed-compression
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/samadejacobs/DeepSpeed#deepspeed4science
DeepSpeed4Science website	https://deepspeed4science.ai/
tutorials	https://www.deepspeed.ai/deepspeed4science/
	https://github.com/samadejacobs/DeepSpeed#deepspeed-software-suite
	https://github.com/samadejacobs/DeepSpeed#deepspeed-library
DeepSpeed	https://github.com/microsoft/deepspeed
DeepSpeed Adoption	https://github.com/samadejacobs/DeepSpeed#deepspeed-adoption
	https://github.com/samadejacobs/DeepSpeed#model-implementations-for-inference-mii
Model Implementations for Inference (MII)	https://github.com/microsoft/deepspeed-mii
	https://github.com/samadejacobs/DeepSpeed#deepspeed-on-azure
recipes	https://github.com/Azure/azureml-examples/tree/main/v1/python-sdk/workflows/train/deepspeed
here	https://github.com/microsoft/Megatron-DeepSpeed/tree/main/examples_deepspeed/azureml
Azure tutorial	https://www.deepspeed.ai/tutorials/azure/
	https://github.com/samadejacobs/DeepSpeed#deepspeed-adoption
AI at Scale	https://www.microsoft.com/en-us/research/project/ai-at-scale/
here	https://innovation.microsoft.com/en-us/exploring-ai-at-scale
Megatron-Turing NLG (530B)	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
Jurassic-1 (178B)	https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
BLOOM (176B)	https://huggingface.co/blog/bloom-megatron-deepspeed
GLM (130B)	https://github.com/THUDM/GLM-130B
xTrimoPGLM (100B)	https://www.biorxiv.org/content/10.1101/2023.07.05.547496v2
YaLM (100B)	https://github.com/yandex/YaLM-100B
GPT-NeoX (20B)	https://github.com/EleutherAI/gpt-neox
AlexaTM (20B)	https://www.amazon.science/blog/20b-parameter-alexa-model-sets-new-marks-in-few-shot-learning
Turing NLG (17B)	https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
METRO-LM (5.4B)	https://arxiv.org/pdf/2204.06644.pdf
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/transformers-light.png#gh-light-mode-only
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/transformers-dark.png#gh-dark-mode-only
Transformers with DeepSpeed	https://huggingface.co/docs/transformers/main/main_classes/deepspeed
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/accelerate-light.png#gh-light-mode-only
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/accelerate-dark.png#gh-dark-mode-only
Accelerate with DeepSpeed	https://huggingface.co/docs/accelerate/usage_guides/deepspeed
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/lightning-light.svg#gh-light-mode-only
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/lightning-dark.svg#gh-dark-mode-only
Lightning with DeepSpeed	https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#deepspeed
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/mosaicml.svg
MosaicML with DeepSpeed	https://docs.mosaicml.com/projects/composer/en/latest/trainer/using_the_trainer.html?highlight=deepspeed#deepspeed-integration
	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/images/determined.svg
Determined with DeepSpeed	https://docs.determined.ai/latest/training/apis-howto/deepspeed/overview.html
	https://user-images.githubusercontent.com/58739961/187154444-fce76639-ac8d-429b-9354-c6fac64b7ef8.jpg
MMEngine with DeepSpeed	https://mmengine.readthedocs.io/en/latest/common_usage/large_model_training.html#deepspeed
	https://github.com/samadejacobs/DeepSpeed#build-pipeline-status
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-p40.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-h100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-nightly.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd-mi200.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/cpu-torch-latest.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/cpu-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/hpu-gaudi2.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/xpu-max1100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-nightly-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-transformers-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-lightning-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-accelerate-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-mii.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-ds-chat.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-sd.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/formatting.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/pages/pages-build-deployment
	https://deepspeed.readthedocs.io/en/latest/?badge=latest
	https://github.com/microsoft/DeepSpeed/actions/workflows/python.yml
	https://github.com/samadejacobs/DeepSpeed#installation
torch's JIT C++ extension loader that relies on ninja	https://pytorch.org/docs/stable/cpp_extension.html
	https://github.com/samadejacobs/DeepSpeed#requirements
PyTorch	https://pytorch.org/
nvcc	https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#introduction
hipcc	https://github.com/ROCm-Developer-Tools/HIPCC
	https://github.com/samadejacobs/DeepSpeed#contributed-hw-support
	https://github.com/samadejacobs/DeepSpeed#pypi
PyPI	https://pypi.org/project/deepspeed/
advanced installation instructions	https://www.deepspeed.ai/tutorials/advanced-install/
	https://github.com/samadejacobs/DeepSpeed#windows
	https://github.com/samadejacobs/DeepSpeed#features
DeepSpeed-Training	https://www.deepspeed.ai/training
DeepSpeed-Inference	https://www.deepspeed.ai/inference
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/samadejacobs/DeepSpeed#further-reading
deepspeed.ai	https://www.deepspeed.ai/
Getting Started	https://www.deepspeed.ai/getting-started/
DeepSpeed JSON Configuration	https://www.deepspeed.ai/docs/config-json/
API Documentation	https://deepspeed.readthedocs.io/en/latest/
Tutorials	https://www.deepspeed.ai/tutorials/
Blogs	https://www.deepspeed.ai/posts/
	https://github.com/samadejacobs/DeepSpeed#contributing
contributing	https://github.com/samadejacobs/DeepSpeed/blob/master/CONTRIBUTING.md
	https://github.com/microsoft/DeepSpeed/graphs/contributors
	https://github.com/samadejacobs/DeepSpeed#contributor-license-agreement
https://cla.opensource.microsoft.com	https://cla.opensource.microsoft.com
	https://github.com/samadejacobs/DeepSpeed#code-of-conduct
Microsoft Open Source Code of Conduct	https://opensource.microsoft.com/codeofconduct/
Code of Conduct FAQ	https://opensource.microsoft.com/codeofconduct/faq/
	https://github.com/samadejacobs/DeepSpeed#publications
arXiv:1910.02054	https://arxiv.org/abs/1910.02054
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20)	https://dl.acm.org/doi/10.5555/3433701.3433727
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial)	https://dl.acm.org/doi/10.1145/3394486.3406703
arXiv:2010.13369	https://arxiv.org/abs/2010.13369
NeurIPS 2020	https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html
arXiv:2101.06840	https://arxiv.org/abs/2101.06840
USENIX ATC 2021	https://www.usenix.org/conference/atc21/presentation/ren-jie
[paper]	https://arxiv.org/abs/2101.06840
[slides]	https://www.usenix.org/system/files/atc21_slides_ren-jie.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/
arXiv:2102.02888	https://arxiv.org/abs/2102.02888
ICML 2021	http://proceedings.mlr.press/v139/tang21a.html
arXiv:2104.07857	https://arxiv.org/abs/2104.07857
SC 2021	https://dl.acm.org/doi/abs/10.1145/3458817.3476205
[paper]	https://arxiv.org/abs/2104.07857
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/SC21-ZeRO-Infinity.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/
arXiv:2104.06069	https://arxiv.org/abs/2104.06069
HiPC 2022	https://hipc.org/advance-program/
arXiv:2108.06084	https://arxiv.org/abs/2108.06084
NeurIPS 2022	https://openreview.net/forum?id=JpZ5du_Kdh
arXiv:2202.06009	https://arxiv.org/abs/2202.06009
arXiv:2201.05596	https://arxiv.org/abs/2201.05596
ICML 2022	https://proceedings.mlr.press/v162/rajbhandari22a.html
[pdf]	https://arxiv.org/abs/2201.05596
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/ICML-5mins.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale/
arXiv:2201.11990	https://arxiv.org/abs/2201.11990
arXiv:2206.01859	https://arxiv.org/abs/2206.01859
NeurIPS 2022	https://openreview.net/forum?id=xNeAhc2CNAl
arXiv:2206.01861	https://arxiv.org/abs/2206.01861
NeurIPS 2022	https://openreview.net/forum?id=f-fVCElZ-G1
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/
arXiv:2207.00032	https://arxiv.org/abs/2207.00032
SC 2022	https://dl.acm.org/doi/abs/10.5555/3571885.3571946
[paper]	https://arxiv.org/abs/2207.00032
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/sc22-ds-inference.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/
arXiv:2211.11586	https://arxiv.org/abs/2211.11586
arXiv:2212.03597	https://arxiv.org/abs/2212.03597
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
arXiv:2301.12017	https://arxiv.org/abs/2301.12017
ICML2023	https://icml.cc/Conferences/2023
ICLR:2023	https://openreview.net/forum?id=Pgtn4l6eKjv
arXiv:2303.07226	https://arxiv.org/abs/2303.07226
Finding at EMNLP2023	https://2023.emnlp.org/
arXiv:2303.08374	https://arxiv.org/abs/2303.08374
arXiv:2303.06318	https://arxiv.org/abs/2303.06318
arXiv:2306.10209	https://arxiv.org/abs/2306.10209
ML for Sys Workshop at NeurIPS2023	http://mlforsystems.org/
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
arXiv:2303.08302	https://arxiv.org/abs/2303.08302
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
arXiv:2305.09847	https://arxiv.org/abs/2305.09847
arXiv:2308.01320	https://arxiv.org/abs/2308.01320
arXiv:2307.09782	https://arxiv.org/abs/2307.09782
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
arXiv:2309.14327	https://arxiv.org/pdf/2309.14327.pdf
arXiv:2310.04610	https://arxiv.org/abs/2310.04610
[blog]	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
arXiv:2310.17723	https://arxiv.org/abs/2310.17723
arXiv:2312.08583	https://arxiv.org/abs/2312.08583
arXiv:2401.14112	https://arxiv.org/abs/2401.14112
System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	https://dl.acm.org/doi/10.1145/3662158.3662806
arXiv:2406.18820	https://arxiv.org/abs/2406.18820
	https://github.com/samadejacobs/DeepSpeed#videos
Overview	https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29
ZeRO + large model training	https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28
17B T-NLG demo	https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27
Fastest BERT training + RScan tuning	https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26
part 1	https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92
part 2	https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94
part 3	https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93
FAQ	https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24
ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed	https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html
DeepSpeed on AzureML	https://youtu.be/yBVXR8G8Bg8
Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference	https://www.youtube.com/watch?v=cntxC3g22oU
[slides]	https://github.com/samadejacobs/DeepSpeed/blob/master/docs/assets/files/presentation-mlops.pdf
DeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim)	https://www.youtube.com/watch?v=pDGI668pNg0
Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher)	https://www.youtube.com/watch?v=tC01FRB0M7w
Ultimate Guide To Scaling ML Models (The AI Epiphany)	https://www.youtube.com/watch?v=hc0u4avAkuM
www.deepspeed.ai/	https://www.deepspeed.ai/
Readme	https://github.com/samadejacobs/DeepSpeed#readme-ov-file
Apache-2.0 license	https://github.com/samadejacobs/DeepSpeed#Apache-2.0-1-ov-file
Code of conduct	https://github.com/samadejacobs/DeepSpeed#coc-ov-file
Contributing	https://github.com/samadejacobs/DeepSpeed#contributing-ov-file
Security policy	https://github.com/samadejacobs/DeepSpeed#security-ov-file
Please reload this page	https://github.com/samadejacobs/DeepSpeed
Activity	https://github.com/samadejacobs/DeepSpeed/activity
0 forks	https://github.com/samadejacobs/DeepSpeed/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fsamadejacobs%2FDeepSpeed&report=samadejacobs+%28user%29
Releases	https://github.com/samadejacobs/DeepSpeed/releases
43 tags	https://github.com/samadejacobs/DeepSpeed/tags
Packages 0	https://github.com/users/samadejacobs/packages?repo_name=DeepSpeed
Please reload this page	https://github.com/samadejacobs/DeepSpeed
Contributors	https://github.com/samadejacobs/DeepSpeed/graphs/contributors
Please reload this page	https://github.com/samadejacobs/DeepSpeed
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.