René's URL Explorer Experiment

Title: GitHub - DeepDriving/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Open Graph Title: GitHub - DeepDriving/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

X Title: GitHub - DeepDriving/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepDriving/DeepSpeed

Open Graph Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepDriving/DeepSpeed

X Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepDriving/DeepSpeed

Mail addresses
opencode@microsoft.com

Opengraph URL: https://github.com/DeepDriving/DeepSpeed

X: @github

direct link

Domain: patch-diff.githubusercontent.com

route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:22c527e5-6c07-e710-6e68-01f5e1dd090d
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	9BA2:3A8D90:9371E75:BECB717:6976176E
html-safe-nonce	04b900024a8b11a1b1e9ae2d07c1ad2ee8da63ac888851ebd88de25fd68e1c7b
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5QkEyOjNBOEQ5MDo5MzcxRTc1OkJFQ0I3MTc6Njk3NjE3NkUiLCJ2aXNpdG9yX2lkIjoiMjE1MzIwMDUxMzIzMzMyNzk4MyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac	48ef3dc87fe6907e10d1cf1f1fa5c1f867a1a02593c228fa065713342d90ca36
hovercard-subject-tag	repository:717911001
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/DeepDriving/DeepSpeed
twitter:image	https://opengraph.githubassets.com/afae274d5bef23767892a17c9e2f3f2de47a8beae6535c0e4584ff4b56bb7c38/DeepDriving/DeepSpeed
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/afae274d5bef23767892a17c9e2f3f2de47a8beae6535c0e4584ff4b56bb7c38/DeepDriving/DeepSpeed
og:image:alt	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepDriving/DeepSpeed
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	c6814b4cc7afd45cd6e64525d0cff0e76dd802f315a5b0e55a7abda1d1d070d0
turbo-cache-control	no-preview
go-import	github.com/DeepDriving/DeepSpeed git https://github.com/DeepDriving/DeepSpeed.git
octolytics-dimension-user_id	2389926
octolytics-dimension-user_login	DeepDriving
octolytics-dimension-repository_id	717911001
octolytics-dimension-repository_nwo	DeepDriving/DeepSpeed
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	235860204
octolytics-dimension-repository_parent_nwo	deepspeedai/DeepSpeed
octolytics-dimension-repository_network_root_id	235860204
octolytics-dimension-repository_network_root_nwo	deepspeedai/DeepSpeed
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	4ea235bfed58ef16c8a5642b3ac64b74f10c9f52
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#start-of-content
	https://patch-diff.githubusercontent.com/
Sign in	https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2FDeepDriving%2FDeepSpeed
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub SparkBuild and deploy intelligent apps	https://github.com/features/spark
GitHub ModelsManage and compare prompts	https://github.com/features/models
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
GitHub SponsorsFund open source developers	https://github.com/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/accelerator
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://patch-diff.githubusercontent.com/login?return_to=https%3A%2F%2Fgithub.com%2FDeepDriving%2FDeepSpeed
Sign up	https://patch-diff.githubusercontent.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=DeepDriving%2FDeepSpeed
Reload	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Reload	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Reload	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
DeepDriving	https://patch-diff.githubusercontent.com/DeepDriving
DeepSpeed	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
deepspeedai/DeepSpeed	https://patch-diff.githubusercontent.com/deepspeedai/DeepSpeed
Notifications	https://patch-diff.githubusercontent.com/login?return_to=%2FDeepDriving%2FDeepSpeed
Fork 0	https://patch-diff.githubusercontent.com/login?return_to=%2FDeepDriving%2FDeepSpeed
Star 0	https://patch-diff.githubusercontent.com/login?return_to=%2FDeepDriving%2FDeepSpeed
www.deepspeed.ai/	https://www.deepspeed.ai/
Apache-2.0 license	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/LICENSE
0 stars	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/stargazers
4.7k forks	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/forks
Branches	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/branches
Tags	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tags
Activity	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/activity
Star	https://patch-diff.githubusercontent.com/login?return_to=%2FDeepDriving%2FDeepSpeed
Notifications	https://patch-diff.githubusercontent.com/login?return_to=%2FDeepDriving%2FDeepSpeed
Code	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Pull requests 0	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/pulls
Actions	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/actions
Projects 0	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/projects
Security 0	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/security
Insights	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/pulse
Code	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Pull requests	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/pulls
Actions	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/actions
Projects	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/projects
Security	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/security
Insights	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/pulse
Branches	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/branches
Tags	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tags
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/branches
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tags
1,942 Commits	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/commits/master/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/commits/master/
.github	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/.github
.github	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/.github
accelerator	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/accelerator
accelerator	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/accelerator
azure	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/azure
azure	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/azure
benchmarks	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/benchmarks
benchmarks	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/benchmarks
bin	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/bin
bin	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/bin
blogs	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/blogs
blogs	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/blogs
csrc	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/csrc
csrc	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/csrc
deepspeed	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/deepspeed
deepspeed	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/deepspeed
docker	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/docker
docker	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/docker
docs	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/docs
docs	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/docs
examples	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/examples
examples	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/examples
op_builder	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/op_builder
op_builder	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/op_builder
release	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/release
release	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/release
requirements	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/requirements
requirements	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/requirements
scripts	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/scripts
scripts	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/scripts
tests	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/tests
tests	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/tree/master/tests
.clang-format	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.clang-format
.clang-format	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.clang-format
.flake8	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.flake8
.flake8	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.flake8
.gitignore	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.gitignore
.gitignore	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.gitignore
.gitmodules	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.gitmodules
.gitmodules	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.gitmodules
.pre-commit-config.yaml	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.pre-commit-config.yaml
.pre-commit-config.yaml	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.pre-commit-config.yaml
.pylintrc	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.pylintrc
.pylintrc	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.pylintrc
.readthedocs.yml	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.readthedocs.yml
.readthedocs.yml	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.readthedocs.yml
.style.yapf	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.style.yapf
.style.yapf	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/.style.yapf
CODEOWNERS	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CODEOWNERS
CODEOWNERS	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CODEOWNERS
CODE_OF_CONDUCT.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CODE_OF_CONDUCT.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CONTRIBUTING.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CONTRIBUTING.md
CONTRIBUTING.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CONTRIBUTING.md
LICENSE	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/LICENSE
LICENSE	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/LICENSE
MANIFEST.in	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/MANIFEST.in
MANIFEST.in	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/MANIFEST.in
MANIFEST_win.in	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/MANIFEST_win.in
MANIFEST_win.in	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/MANIFEST_win.in
README.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/README.md
README.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/README.md
SECURITY.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/SECURITY.md
SECURITY.md	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/SECURITY.md
build_win.bat	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/build_win.bat
build_win.bat	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/build_win.bat
environment.yml	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/environment.yml
environment.yml	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/environment.yml
install.sh	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/install.sh
install.sh	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/install.sh
setup.cfg	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/setup.cfg
setup.cfg	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/setup.cfg
setup.py	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/setup.py
setup.py	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/setup.py
version.txt	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/version.txt
version.txt	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/version.txt
README	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Code of conduct	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Contributing	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
License	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Security	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
	https://github.com/Microsoft/DeepSpeed/blob/master/LICENSE
	https://pypi.org/project/deepspeed/
	https://pepy.tech/project/deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#build-pipeline-status
	https://twitter.com/intent/follow?screen_name=MSFTDeepSpeed
	https://twitter.com/MSFTDeepSpeedJP
	https://www.zhihu.com/people/deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#latest-news
learn how	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat
DeepSpeed ZeRO-Offload++: 6x Higher Training Throughput via Collaborative CPU/GPU Twin-Flow	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-offloadpp
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/japanese/README.md
DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-visualchat/10-03-2023/README.md
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-visualchat/10-03-2023/README.md
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README-Chinese.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README-Japanese.md
DeepSpeed4Science website	https://deepspeed4science.ai/
Tutorials	https://www.deepspeed.ai/deepspeed4science/
White paper	https://arxiv.org/abs/2310.04610
Blog	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/japanese/README.md
DeepSpeed ZeRO-Inference: 20x faster inference through weight quantization and KV cache offloading	https://github.com/microsoft/DeepSpeedExamples/blob/master/inference/huggingface/zero_inference/README.md
DeepSpeed-Chat: Llama/Llama-2 system support, efficiency boost, and training stability improvements	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat/ds-chat-release-8-31/README.md
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ulysses
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/japanese/README.md
ZeRO++: A leap in speed for LLM and chat model training with 4X less communication	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
English	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/japanese/README.md
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#extreme-speed-and-scale-for-dl-training-and-inference
DeepSpeed	https://www.deepspeed.ai/
MT-530B	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
BLOOM	https://huggingface.co/blog/bloom-megatron-deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeeds-four-innovation-pillars
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/DeepSpeed-pillars.png
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-training
DeepSpeed-Training	https://www.deepspeed.ai/training/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-inference
DeepSpeed-Inference	https://www.deepspeed.ai/inference
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-compression
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed4science
DeepSpeed4Science website	https://deepspeed4science.ai/
tutorials	https://www.deepspeed.ai/deepspeed4science/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-software-suite
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-library
DeepSpeed	https://github.com/microsoft/deepspeed
DeepSpeed Adoption	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-adoption
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#model-implementations-for-inference-mii
Model Implementations for Inference (MII)	https://github.com/microsoft/deepspeed-mii
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-on-azure
recipes	https://github.com/Azure/azureml-examples/tree/main/v1/python-sdk/workflows/train/deepspeed
here	https://github.com/microsoft/Megatron-DeepSpeed/tree/main/examples_deepspeed/azureml
Azure tutorial	https://www.deepspeed.ai/tutorials/azure/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#deepspeed-adoption
AI at Scale	https://www.microsoft.com/en-us/research/project/ai-at-scale/
here	https://innovation.microsoft.com/en-us/exploring-ai-at-scale
Megatron-Turing NLG (530B)	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
Jurassic-1 (178B)	https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
BLOOM (176B)	https://huggingface.co/blog/bloom-megatron-deepspeed
GLM (130B)	https://github.com/THUDM/GLM-130B
xTrimoPGLM (100B)	https://www.biorxiv.org/content/10.1101/2023.07.05.547496v2
YaLM (100B)	https://github.com/yandex/YaLM-100B
GPT-NeoX (20B)	https://github.com/EleutherAI/gpt-neox
AlexaTM (20B)	https://www.amazon.science/blog/20b-parameter-alexa-model-sets-new-marks-in-few-shot-learning
Turing NLG (17B)	https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
METRO-LM (5.4B)	https://arxiv.org/pdf/2204.06644.pdf
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/transformers-light.png#gh-light-mode-only
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/transformers-dark.png#gh-dark-mode-only
Transformers with DeepSpeed	https://huggingface.co/docs/transformers/main/main_classes/deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/accelerate-light.png#gh-light-mode-only
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/accelerate-dark.png#gh-dark-mode-only
Accelerate with DeepSpeed	https://huggingface.co/docs/accelerate/usage_guides/deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/lightning-light.svg#gh-light-mode-only
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/lightning-dark.svg#gh-dark-mode-only
Lightning with DeepSpeed	https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/mosaicml.svg
MosaicML with DeepSpeed	https://docs.mosaicml.com/projects/composer/en/latest/trainer/using_the_trainer.html?highlight=deepspeed#deepspeed-integration
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/images/determined.svg
Determined with DeepSpeed	https://docs.determined.ai/latest/training/apis-howto/deepspeed/overview.html
	https://user-images.githubusercontent.com/58739961/187154444-fce76639-ac8d-429b-9354-c6fac64b7ef8.jpg
MMEngine with DeepSpeed	https://mmengine.readthedocs.io/en/latest/common_usage/large_model_training.html#deepspeed
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#build-pipeline-status
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-p40.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-h100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-nightly.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd-mi100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd-mi200.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-cpu.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-nightly-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-transformers-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-lightning-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-accelerate-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-megatron.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-mii.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-ds-chat.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/formatting.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/pages/pages-build-deployment
	https://deepspeed.readthedocs.io/en/latest/?badge=latest
	https://github.com/microsoft/DeepSpeed/actions/workflows/python.yml
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#installation
torch's JIT C++ extension loader that relies on ninja	https://pytorch.org/docs/stable/cpp_extension.html
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#requirements
PyTorch	https://pytorch.org/
nvcc	https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#introduction
hipcc	https://github.com/ROCm-Developer-Tools/HIPCC
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#pypi
PyPI	https://pypi.org/project/deepspeed/
advanced installation instructions	https://www.deepspeed.ai/tutorials/advanced-install/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#windows
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#features
DeepSpeed-Training	https://www.deepspeed.ai/training
DeepSpeed-Inference	https://www.deepspeed.ai/inference
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#further-reading
deepspeed.ai	https://www.deepspeed.ai/
Getting Started	https://www.deepspeed.ai/getting-started/
DeepSpeed JSON Configuration	https://www.deepspeed.ai/docs/config-json/
API Documentation	https://deepspeed.readthedocs.io/en/latest/
Tutorials	https://www.deepspeed.ai/tutorials/
Blogs	https://www.deepspeed.ai/posts/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#contributing
contributing	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/CONTRIBUTING.md
	https://github.com/microsoft/DeepSpeed/graphs/contributors
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#contributor-license-agreement
https://cla.opensource.microsoft.com	https://cla.opensource.microsoft.com
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#code-of-conduct
Microsoft Open Source Code of Conduct	https://opensource.microsoft.com/codeofconduct/
Code of Conduct FAQ	https://opensource.microsoft.com/codeofconduct/faq/
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#publications
arXiv:1910.02054	https://arxiv.org/abs/1910.02054
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20)	https://dl.acm.org/doi/10.5555/3433701.3433727
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial)	https://dl.acm.org/doi/10.1145/3394486.3406703
arXiv:2010.13369	https://arxiv.org/abs/2010.13369
NeurIPS 2020	https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html
arXiv:2101.06840	https://arxiv.org/abs/2101.06840
USENIX ATC 2021	https://www.usenix.org/conference/atc21/presentation/ren-jie
[paper]	https://arxiv.org/abs/2101.06840
[slides]	https://www.usenix.org/system/files/atc21_slides_ren-jie.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/
arXiv:2102.02888	https://arxiv.org/abs/2102.02888
ICML 2021	http://proceedings.mlr.press/v139/tang21a.html
arXiv:2104.07857	https://arxiv.org/abs/2104.07857
SC 2021	https://dl.acm.org/doi/abs/10.1145/3458817.3476205
[paper]	https://arxiv.org/abs/2104.07857
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/SC21-ZeRO-Infinity.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/
arXiv:2104.06069	https://arxiv.org/abs/2104.06069
HiPC 2022	https://hipc.org/advance-program/
arXiv:2108.06084	https://arxiv.org/abs/2108.06084
NeurIPS 2022	https://openreview.net/forum?id=JpZ5du_Kdh
arXiv:2202.06009	https://arxiv.org/abs/2202.06009
arXiv:2201.05596	https://arxiv.org/abs/2201.05596
ICML 2022	https://proceedings.mlr.press/v162/rajbhandari22a.html
[pdf]	https://arxiv.org/abs/2201.05596
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/ICML-5mins.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale/
arXiv:2201.11990	https://arxiv.org/abs/2201.11990
arXiv:2206.01859	https://arxiv.org/abs/2206.01859
NeurIPS 2022	https://openreview.net/forum?id=xNeAhc2CNAl
arXiv:2206.01861	https://arxiv.org/abs/2206.01861
NeurIPS 2022	https://openreview.net/forum?id=f-fVCElZ-G1
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/
arXiv:2207.00032	https://arxiv.org/abs/2207.00032
SC 2022	https://dl.acm.org/doi/abs/10.5555/3571885.3571946
[paper]	https://arxiv.org/abs/2207.00032
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/sc22-ds-inference.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/
arXiv:2211.11586	https://arxiv.org/abs/2211.11586
arXiv:2212.03597	https://arxiv.org/abs/2212.03597
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
arXiv:2301.12017	https://arxiv.org/abs/2301.12017
ICML2023	https://icml.cc/Conferences/2023
ICLR:2023	https://openreview.net/forum?id=Pgtn4l6eKjv
arXiv:2303.07226	https://arxiv.org/abs/2303.07226
Finding at EMNLP2023	https://2023.emnlp.org/
arXiv:2303.08374	https://arxiv.org/abs/2303.08374
arXiv:2303.06318	https://arxiv.org/abs/2303.06318
arXiv:2306.10209	https://arxiv.org/abs/2306.10209
ML for Sys Workshop at NeurIPS2023	http://mlforsystems.org/
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
arXiv:2303.08302	https://arxiv.org/abs/2303.08302
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
arXiv:2305.09847	https://arxiv.org/abs/2305.09847
arXiv:2308.01320	https://arxiv.org/abs/2308.01320
arXiv:2307.09782	https://arxiv.org/abs/2307.09782
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
arXiv:2309.14327	https://arxiv.org/pdf/2309.14327.pdf
arXiv:2310.04610	https://arxiv.org/abs/2310.04610
[blog]	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
arXiv:2310.17723	https://arxiv.org/abs/2310.17723
	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#videos
Overview	https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29
ZeRO + large model training	https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28
17B T-NLG demo	https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27
Fastest BERT training + RScan tuning	https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26
part 1	https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92
part 2	https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94
part 3	https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93
FAQ	https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24
ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed	https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html
DeepSpeed on AzureML	https://youtu.be/yBVXR8G8Bg8
Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference	https://www.youtube.com/watch?v=cntxC3g22oU
[slides]	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/blob/master/docs/assets/files/presentation-mlops.pdf
DeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim)	https://www.youtube.com/watch?v=pDGI668pNg0
Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher)	https://www.youtube.com/watch?v=tC01FRB0M7w
Ultimate Guide To Scaling ML Models (The AI Epiphany)	https://www.youtube.com/watch?v=hc0u4avAkuM
www.deepspeed.ai/	https://www.deepspeed.ai/
Readme	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#readme-ov-file
Apache-2.0 license	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed#Apache-2.0-1-ov-file
Please reload this page	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed
Activity	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/activity
0 stars	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/stargazers
0 watching	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/watchers
0 forks	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/forks
Report repository	https://patch-diff.githubusercontent.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FDeepDriving%2FDeepSpeed&report=DeepDriving+%28user%29
Releases	https://patch-diff.githubusercontent.com/DeepDriving/DeepSpeed/releases
Packages 0	https://patch-diff.githubusercontent.com/users/DeepDriving/packages?repo_name=DeepSpeed
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.