René's URL Explorer Experiment

Title: GitHub - tensorwavecloud/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Open Graph Title: GitHub - tensorwavecloud/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

X Title: GitHub - tensorwavecloud/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - tensorwavecloud/DeepSpeed

Open Graph Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - tensorwavecloud/DeepSpeed

X Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - tensorwavecloud/DeepSpeed

Mail addresses
opencode@microsoft.com

Opengraph URL: https://github.com/tensorwavecloud/DeepSpeed

X: @github

direct link

Domain: github.com

route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:11831178-d02f-1ff9-3ff2-3988919f1c1f
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	A1A0:16D2A2:940A7:BF3D1:698E7E71
html-safe-nonce	af45235d5038e0588527486601c816de73a9a967c3918cb1b46e57c1a762e800
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBMUEwOjE2RDJBMjo5NDBBNzpCRjNEMTo2OThFN0U3MSIsInZpc2l0b3JfaWQiOiI0MzU4MzUwMDU0MzAzMzY2NzcwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac	ed93ff275c2a268c1f8948f831b58dd7866e6a96c682327c8081a8a3a9eddb7f
hovercard-subject-tag	repository:800656951
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/tensorwavecloud/DeepSpeed
twitter:image	https://opengraph.githubassets.com/f8ec550b808ea09e1cdfd0d1535e3ae46de2d0fddfa19813673efd9457d1b1b7/tensorwavecloud/DeepSpeed
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/f8ec550b808ea09e1cdfd0d1535e3ae46de2d0fddfa19813673efd9457d1b1b7/tensorwavecloud/DeepSpeed
og:image:alt	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - tensorwavecloud/DeepSpeed
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	cb2828a801ee6b7be618f3ac76fbf55def35bbc30f053a9c41bf90210b8b72ba
turbo-cache-control	no-preview
go-import	github.com/tensorwavecloud/DeepSpeed git https://github.com/tensorwavecloud/DeepSpeed.git
octolytics-dimension-user_id	155877601
octolytics-dimension-user_login	tensorwavecloud
octolytics-dimension-repository_id	800656951
octolytics-dimension-repository_nwo	tensorwavecloud/DeepSpeed
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	235860204
octolytics-dimension-repository_parent_nwo	deepspeedai/DeepSpeed
octolytics-dimension-repository_network_root_id	235860204
octolytics-dimension-repository_network_root_nwo	deepspeedai/DeepSpeed
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	f7a12861f7011eecb14bcc4388a767d829a52dfc
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/tensorwavecloud/DeepSpeed#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftensorwavecloud%2FDeepSpeed
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub SparkBuild and deploy intelligent apps	https://github.com/features/spark
GitHub ModelsManage and compare prompts	https://github.com/features/models
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
GitHub SponsorsFund open source developers	https://github.com/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/accelerator
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftensorwavecloud%2FDeepSpeed
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=tensorwavecloud%2FDeepSpeed
Reload	https://github.com/tensorwavecloud/DeepSpeed
Reload	https://github.com/tensorwavecloud/DeepSpeed
Reload	https://github.com/tensorwavecloud/DeepSpeed
tensorwavecloud	https://github.com/tensorwavecloud
DeepSpeed	https://github.com/tensorwavecloud/DeepSpeed
deepspeedai/DeepSpeed	https://github.com/deepspeedai/DeepSpeed
Notifications	https://github.com/login?return_to=%2Ftensorwavecloud%2FDeepSpeed
Fork 0	https://github.com/login?return_to=%2Ftensorwavecloud%2FDeepSpeed
Star 0	https://github.com/login?return_to=%2Ftensorwavecloud%2FDeepSpeed
www.deepspeed.ai/	https://www.deepspeed.ai/
Apache-2.0 license	https://github.com/tensorwavecloud/DeepSpeed/blob/master/LICENSE
0 stars	https://github.com/tensorwavecloud/DeepSpeed/stargazers
4.7k forks	https://github.com/tensorwavecloud/DeepSpeed/forks
Branches	https://github.com/tensorwavecloud/DeepSpeed/branches
Tags	https://github.com/tensorwavecloud/DeepSpeed/tags
Activity	https://github.com/tensorwavecloud/DeepSpeed/activity
Star	https://github.com/login?return_to=%2Ftensorwavecloud%2FDeepSpeed
Notifications	https://github.com/login?return_to=%2Ftensorwavecloud%2FDeepSpeed
Code	https://github.com/tensorwavecloud/DeepSpeed
Pull requests 0	https://github.com/tensorwavecloud/DeepSpeed/pulls
Actions	https://github.com/tensorwavecloud/DeepSpeed/actions
Projects 0	https://github.com/tensorwavecloud/DeepSpeed/projects
Security 0	https://github.com/tensorwavecloud/DeepSpeed/security
Insights	https://github.com/tensorwavecloud/DeepSpeed/pulse
Code	https://github.com/tensorwavecloud/DeepSpeed
Pull requests	https://github.com/tensorwavecloud/DeepSpeed/pulls
Actions	https://github.com/tensorwavecloud/DeepSpeed/actions
Projects	https://github.com/tensorwavecloud/DeepSpeed/projects
Security	https://github.com/tensorwavecloud/DeepSpeed/security
Insights	https://github.com/tensorwavecloud/DeepSpeed/pulse
Branches	https://github.com/tensorwavecloud/DeepSpeed/branches
Tags	https://github.com/tensorwavecloud/DeepSpeed/tags
	https://github.com/tensorwavecloud/DeepSpeed/branches
	https://github.com/tensorwavecloud/DeepSpeed/tags
2,450 Commits	https://github.com/tensorwavecloud/DeepSpeed/commits/master/
	https://github.com/tensorwavecloud/DeepSpeed/commits/master/
.github	https://github.com/tensorwavecloud/DeepSpeed/tree/master/.github
.github	https://github.com/tensorwavecloud/DeepSpeed/tree/master/.github
accelerator	https://github.com/tensorwavecloud/DeepSpeed/tree/master/accelerator
accelerator	https://github.com/tensorwavecloud/DeepSpeed/tree/master/accelerator
azure	https://github.com/tensorwavecloud/DeepSpeed/tree/master/azure
azure	https://github.com/tensorwavecloud/DeepSpeed/tree/master/azure
benchmarks	https://github.com/tensorwavecloud/DeepSpeed/tree/master/benchmarks
benchmarks	https://github.com/tensorwavecloud/DeepSpeed/tree/master/benchmarks
bin	https://github.com/tensorwavecloud/DeepSpeed/tree/master/bin
bin	https://github.com/tensorwavecloud/DeepSpeed/tree/master/bin
blogs	https://github.com/tensorwavecloud/DeepSpeed/tree/master/blogs
blogs	https://github.com/tensorwavecloud/DeepSpeed/tree/master/blogs
csrc	https://github.com/tensorwavecloud/DeepSpeed/tree/master/csrc
csrc	https://github.com/tensorwavecloud/DeepSpeed/tree/master/csrc
deepspeed	https://github.com/tensorwavecloud/DeepSpeed/tree/master/deepspeed
deepspeed	https://github.com/tensorwavecloud/DeepSpeed/tree/master/deepspeed
docker	https://github.com/tensorwavecloud/DeepSpeed/tree/master/docker
docker	https://github.com/tensorwavecloud/DeepSpeed/tree/master/docker
docs	https://github.com/tensorwavecloud/DeepSpeed/tree/master/docs
docs	https://github.com/tensorwavecloud/DeepSpeed/tree/master/docs
examples	https://github.com/tensorwavecloud/DeepSpeed/tree/master/examples
examples	https://github.com/tensorwavecloud/DeepSpeed/tree/master/examples
op_builder	https://github.com/tensorwavecloud/DeepSpeed/tree/master/op_builder
op_builder	https://github.com/tensorwavecloud/DeepSpeed/tree/master/op_builder
release	https://github.com/tensorwavecloud/DeepSpeed/tree/master/release
release	https://github.com/tensorwavecloud/DeepSpeed/tree/master/release
requirements	https://github.com/tensorwavecloud/DeepSpeed/tree/master/requirements
requirements	https://github.com/tensorwavecloud/DeepSpeed/tree/master/requirements
scripts	https://github.com/tensorwavecloud/DeepSpeed/tree/master/scripts
scripts	https://github.com/tensorwavecloud/DeepSpeed/tree/master/scripts
tests	https://github.com/tensorwavecloud/DeepSpeed/tree/master/tests
tests	https://github.com/tensorwavecloud/DeepSpeed/tree/master/tests
.clang-format	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.clang-format
.clang-format	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.clang-format
.flake8	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.flake8
.flake8	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.flake8
.gitignore	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.gitignore
.gitignore	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.gitignore
.gitmodules	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.gitmodules
.gitmodules	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.gitmodules
.pre-commit-config.yaml	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.pre-commit-config.yaml
.pre-commit-config.yaml	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.pre-commit-config.yaml
.pylintrc	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.pylintrc
.pylintrc	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.pylintrc
.readthedocs.yml	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.readthedocs.yml
.readthedocs.yml	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.readthedocs.yml
.style.yapf	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.style.yapf
.style.yapf	https://github.com/tensorwavecloud/DeepSpeed/blob/master/.style.yapf
CODEOWNERS	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CODEOWNERS
CODEOWNERS	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CODEOWNERS
CODE_OF_CONDUCT.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CODE_OF_CONDUCT.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CONTRIBUTING.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CONTRIBUTING.md
LICENSE	https://github.com/tensorwavecloud/DeepSpeed/blob/master/LICENSE
LICENSE	https://github.com/tensorwavecloud/DeepSpeed/blob/master/LICENSE
MANIFEST.in	https://github.com/tensorwavecloud/DeepSpeed/blob/master/MANIFEST.in
MANIFEST.in	https://github.com/tensorwavecloud/DeepSpeed/blob/master/MANIFEST.in
MANIFEST_win.in	https://github.com/tensorwavecloud/DeepSpeed/blob/master/MANIFEST_win.in
MANIFEST_win.in	https://github.com/tensorwavecloud/DeepSpeed/blob/master/MANIFEST_win.in
README.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/README.md
README.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/README.md
SECURITY.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/SECURITY.md
SECURITY.md	https://github.com/tensorwavecloud/DeepSpeed/blob/master/SECURITY.md
build_win.bat	https://github.com/tensorwavecloud/DeepSpeed/blob/master/build_win.bat
build_win.bat	https://github.com/tensorwavecloud/DeepSpeed/blob/master/build_win.bat
environment.yml	https://github.com/tensorwavecloud/DeepSpeed/blob/master/environment.yml
environment.yml	https://github.com/tensorwavecloud/DeepSpeed/blob/master/environment.yml
install.sh	https://github.com/tensorwavecloud/DeepSpeed/blob/master/install.sh
install.sh	https://github.com/tensorwavecloud/DeepSpeed/blob/master/install.sh
setup.cfg	https://github.com/tensorwavecloud/DeepSpeed/blob/master/setup.cfg
setup.cfg	https://github.com/tensorwavecloud/DeepSpeed/blob/master/setup.cfg
setup.py	https://github.com/tensorwavecloud/DeepSpeed/blob/master/setup.py
setup.py	https://github.com/tensorwavecloud/DeepSpeed/blob/master/setup.py
version.txt	https://github.com/tensorwavecloud/DeepSpeed/blob/master/version.txt
version.txt	https://github.com/tensorwavecloud/DeepSpeed/blob/master/version.txt
README	https://github.com/tensorwavecloud/DeepSpeed
Code of conduct	https://github.com/tensorwavecloud/DeepSpeed
Contributing	https://github.com/tensorwavecloud/DeepSpeed
License	https://github.com/tensorwavecloud/DeepSpeed
Security	https://github.com/tensorwavecloud/DeepSpeed
	https://github.com/Microsoft/DeepSpeed/blob/master/LICENSE
	https://pypi.org/project/deepspeed/
	https://pepy.tech/project/deepspeed
	https://github.com/tensorwavecloud/DeepSpeed#build-pipeline-status
	https://twitter.com/intent/follow?screen_name=MSFTDeepSpeed
	https://twitter.com/MSFTDeepSpeedJP
	https://www.zhihu.com/people/deepspeed
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only
	https://github.com/tensorwavecloud/DeepSpeed#latest-news
learn how	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat
DeepSpeed on Windows	https://github.com/microsoft/DeepSpeed/tree/master/blogs/windows/08-2024/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/windows/08-2024/japanese/README.md
DeepNVMe: Improving DL Applications through I/O Optimizations	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-gds/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-gds/japanese/README.md
DeepSpeed Universal Checkpointing: Efficient and Flexible Checkpointing for Large Scale Distributed Training	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ucp/README.md
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ucp/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ucp/japanese/README.md
DeepSpeed-FP6:The power of FP6-Centric Serving for Large Language Models	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fp6/03-05-2024
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fp6/03-05-2024/README.md
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fp6/03-05-2024/README-Chinese.md
DeepSpeed-FastGen: Introducing Mixtral, Phi-2, and Falcon support with major performance and feature enhancements.	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/2024-01-19
Llama 2 Inference on 4th Gen Intel® Xeon® Scalable Processor with DeepSpeed	https://github.com/microsoft/DeepSpeed/tree/master/blogs/intel-inference
[Intel version]	https://www.intel.com/content/www/us/en/developer/articles/technical/xllama-2-on-xeon-scalable-processor-with-deepspeed.html
DeepSpeed ZeRO-Offload++: 6x Higher Training Throughput via Collaborative CPU/GPU Twin-Flow	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-offloadpp
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen
中文	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen/japanese/README.md
DeepSpeed-VisualChat: Improve Your Chat Experience with Multi-Round Multi-Image Inputs	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-visualchat/10-03-2023/README.md
English	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-visualchat/10-03-2023/README.md
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README-Chinese.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README-Japanese.md
DeepSpeed4Science website	https://deepspeed4science.ai/
Tutorials	https://www.deepspeed.ai/deepspeed4science/
White paper	https://arxiv.org/abs/2310.04610
Blog	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/japanese/README.md
DeepSpeed ZeRO-Inference: 20x faster inference through weight quantization and KV cache offloading	https://github.com/microsoft/DeepSpeedExamples/blob/master/inference/huggingface/zero_inference/README.md
DeepSpeed-Chat: Llama/Llama-2 system support, efficiency boost, and training stability improvements	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat/ds-chat-release-8-31/README.md
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ulysses
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/japanese/README.md
ZeRO++: A leap in speed for LLM and chat model training with 4X less communication	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
English	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/japanese/README.md
	https://github.com/tensorwavecloud/DeepSpeed#extreme-speed-and-scale-for-dl-training-and-inference
DeepSpeed	https://www.deepspeed.ai/
MT-530B	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
BLOOM	https://huggingface.co/blog/bloom-megatron-deepspeed
	https://github.com/tensorwavecloud/DeepSpeed#deepspeeds-four-innovation-pillars
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/DeepSpeed-pillars.png
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-training
DeepSpeed-Training	https://www.deepspeed.ai/training/
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-inference
DeepSpeed-Inference	https://www.deepspeed.ai/inference
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-compression
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed4science
DeepSpeed4Science website	https://deepspeed4science.ai/
tutorials	https://www.deepspeed.ai/deepspeed4science/
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-software-suite
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-library
DeepSpeed	https://github.com/microsoft/deepspeed
DeepSpeed Adoption	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-adoption
	https://github.com/tensorwavecloud/DeepSpeed#model-implementations-for-inference-mii
Model Implementations for Inference (MII)	https://github.com/microsoft/deepspeed-mii
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-on-azure
recipes	https://github.com/Azure/azureml-examples/tree/main/v1/python-sdk/workflows/train/deepspeed
here	https://github.com/microsoft/Megatron-DeepSpeed/tree/main/examples_deepspeed/azureml
Azure tutorial	https://www.deepspeed.ai/tutorials/azure/
	https://github.com/tensorwavecloud/DeepSpeed#deepspeed-adoption
AI at Scale	https://www.microsoft.com/en-us/research/project/ai-at-scale/
here	https://innovation.microsoft.com/en-us/exploring-ai-at-scale
Megatron-Turing NLG (530B)	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
Jurassic-1 (178B)	https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
BLOOM (176B)	https://huggingface.co/blog/bloom-megatron-deepspeed
GLM (130B)	https://github.com/THUDM/GLM-130B
xTrimoPGLM (100B)	https://www.biorxiv.org/content/10.1101/2023.07.05.547496v2
YaLM (100B)	https://github.com/yandex/YaLM-100B
GPT-NeoX (20B)	https://github.com/EleutherAI/gpt-neox
AlexaTM (20B)	https://www.amazon.science/blog/20b-parameter-alexa-model-sets-new-marks-in-few-shot-learning
Turing NLG (17B)	https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
METRO-LM (5.4B)	https://arxiv.org/pdf/2204.06644.pdf
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/transformers-light.png#gh-light-mode-only
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/transformers-dark.png#gh-dark-mode-only
Transformers with DeepSpeed	https://huggingface.co/docs/transformers/main/main_classes/deepspeed
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/accelerate-light.png#gh-light-mode-only
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/accelerate-dark.png#gh-dark-mode-only
Accelerate with DeepSpeed	https://huggingface.co/docs/accelerate/usage_guides/deepspeed
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/lightning-light.svg#gh-light-mode-only
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/lightning-dark.svg#gh-dark-mode-only
Lightning with DeepSpeed	https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#deepspeed
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/mosaicml.svg
MosaicML with DeepSpeed	https://docs.mosaicml.com/projects/composer/en/latest/trainer/using_the_trainer.html?highlight=deepspeed#deepspeed-integration
	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/images/determined.svg
Determined with DeepSpeed	https://docs.determined.ai/latest/training/apis-howto/deepspeed/overview.html
	https://user-images.githubusercontent.com/58739961/187154444-fce76639-ac8d-429b-9354-c6fac64b7ef8.jpg
MMEngine with DeepSpeed	https://mmengine.readthedocs.io/en/latest/common_usage/large_model_training.html#deepspeed
	https://github.com/tensorwavecloud/DeepSpeed#build-pipeline-status
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-p40.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-h100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-nightly.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd-mi200.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/cpu-torch-latest.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/cpu-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/hpu-gaudi2.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/xpu-max1100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-nightly-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-transformers-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-lightning-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-accelerate-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-mii.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-ds-chat.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-sd.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/formatting.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/pages/pages-build-deployment
	https://deepspeed.readthedocs.io/en/latest/?badge=latest
	https://github.com/microsoft/DeepSpeed/actions/workflows/python.yml
	https://github.com/tensorwavecloud/DeepSpeed#installation
torch's JIT C++ extension loader that relies on ninja	https://pytorch.org/docs/stable/cpp_extension.html
	https://github.com/tensorwavecloud/DeepSpeed#requirements
PyTorch	https://pytorch.org/
nvcc	https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#introduction
hipcc	https://github.com/ROCm-Developer-Tools/HIPCC
	https://github.com/tensorwavecloud/DeepSpeed#contributed-hw-support
	https://github.com/tensorwavecloud/DeepSpeed#pypi
PyPI	https://pypi.org/project/deepspeed/
advanced installation instructions	https://www.deepspeed.ai/tutorials/advanced-install/
	https://github.com/tensorwavecloud/DeepSpeed#windows
	https://github.com/tensorwavecloud/DeepSpeed#features
DeepSpeed-Training	https://www.deepspeed.ai/training
DeepSpeed-Inference	https://www.deepspeed.ai/inference
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/tensorwavecloud/DeepSpeed#further-reading
deepspeed.ai	https://www.deepspeed.ai/
Getting Started	https://www.deepspeed.ai/getting-started/
DeepSpeed JSON Configuration	https://www.deepspeed.ai/docs/config-json/
API Documentation	https://deepspeed.readthedocs.io/en/latest/
Tutorials	https://www.deepspeed.ai/tutorials/
Blogs	https://www.deepspeed.ai/posts/
	https://github.com/tensorwavecloud/DeepSpeed#contributing
contributing	https://github.com/tensorwavecloud/DeepSpeed/blob/master/CONTRIBUTING.md
	https://github.com/microsoft/DeepSpeed/graphs/contributors
	https://github.com/tensorwavecloud/DeepSpeed#contributor-license-agreement
https://cla.opensource.microsoft.com	https://cla.opensource.microsoft.com
	https://github.com/tensorwavecloud/DeepSpeed#code-of-conduct
Microsoft Open Source Code of Conduct	https://opensource.microsoft.com/codeofconduct/
Code of Conduct FAQ	https://opensource.microsoft.com/codeofconduct/faq/
	https://github.com/tensorwavecloud/DeepSpeed#publications
arXiv:1910.02054	https://arxiv.org/abs/1910.02054
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20)	https://dl.acm.org/doi/10.5555/3433701.3433727
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial)	https://dl.acm.org/doi/10.1145/3394486.3406703
arXiv:2010.13369	https://arxiv.org/abs/2010.13369
NeurIPS 2020	https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html
arXiv:2101.06840	https://arxiv.org/abs/2101.06840
USENIX ATC 2021	https://www.usenix.org/conference/atc21/presentation/ren-jie
[paper]	https://arxiv.org/abs/2101.06840
[slides]	https://www.usenix.org/system/files/atc21_slides_ren-jie.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/
arXiv:2102.02888	https://arxiv.org/abs/2102.02888
ICML 2021	http://proceedings.mlr.press/v139/tang21a.html
arXiv:2104.07857	https://arxiv.org/abs/2104.07857
SC 2021	https://dl.acm.org/doi/abs/10.1145/3458817.3476205
[paper]	https://arxiv.org/abs/2104.07857
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/SC21-ZeRO-Infinity.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/
arXiv:2104.06069	https://arxiv.org/abs/2104.06069
HiPC 2022	https://hipc.org/advance-program/
arXiv:2108.06084	https://arxiv.org/abs/2108.06084
NeurIPS 2022	https://openreview.net/forum?id=JpZ5du_Kdh
arXiv:2202.06009	https://arxiv.org/abs/2202.06009
arXiv:2201.05596	https://arxiv.org/abs/2201.05596
ICML 2022	https://proceedings.mlr.press/v162/rajbhandari22a.html
[pdf]	https://arxiv.org/abs/2201.05596
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/ICML-5mins.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale/
arXiv:2201.11990	https://arxiv.org/abs/2201.11990
arXiv:2206.01859	https://arxiv.org/abs/2206.01859
NeurIPS 2022	https://openreview.net/forum?id=xNeAhc2CNAl
arXiv:2206.01861	https://arxiv.org/abs/2206.01861
NeurIPS 2022	https://openreview.net/forum?id=f-fVCElZ-G1
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/
arXiv:2207.00032	https://arxiv.org/abs/2207.00032
SC 2022	https://dl.acm.org/doi/abs/10.5555/3571885.3571946
[paper]	https://arxiv.org/abs/2207.00032
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/sc22-ds-inference.pdf
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/
arXiv:2211.11586	https://arxiv.org/abs/2211.11586
arXiv:2212.03597	https://arxiv.org/abs/2212.03597
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
arXiv:2301.12017	https://arxiv.org/abs/2301.12017
ICML2023	https://icml.cc/Conferences/2023
ICLR:2023	https://openreview.net/forum?id=Pgtn4l6eKjv
arXiv:2303.07226	https://arxiv.org/abs/2303.07226
Finding at EMNLP2023	https://2023.emnlp.org/
arXiv:2303.08374	https://arxiv.org/abs/2303.08374
arXiv:2303.06318	https://arxiv.org/abs/2303.06318
arXiv:2306.10209	https://arxiv.org/abs/2306.10209
ML for Sys Workshop at NeurIPS2023	http://mlforsystems.org/
[blog]	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
arXiv:2303.08302	https://arxiv.org/abs/2303.08302
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
arXiv:2305.09847	https://arxiv.org/abs/2305.09847
arXiv:2308.01320	https://arxiv.org/abs/2308.01320
arXiv:2307.09782	https://arxiv.org/abs/2307.09782
ENLSP2023 Workshop at NeurIPS2023	https://neurips2023-enlsp.github.io/
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf
arXiv:2309.14327	https://arxiv.org/pdf/2309.14327.pdf
arXiv:2310.04610	https://arxiv.org/abs/2310.04610
[blog]	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
arXiv:2310.17723	https://arxiv.org/abs/2310.17723
arXiv:2312.08583	https://arxiv.org/abs/2312.08583
arXiv:2401.14112	https://arxiv.org/abs/2401.14112
System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	https://dl.acm.org/doi/10.1145/3662158.3662806
arXiv:2406.18820	https://arxiv.org/abs/2406.18820
	https://github.com/tensorwavecloud/DeepSpeed#videos
Overview	https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29
ZeRO + large model training	https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28
17B T-NLG demo	https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27
Fastest BERT training + RScan tuning	https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26
part 1	https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92
part 2	https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94
part 3	https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93
FAQ	https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24
ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed	https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html
DeepSpeed on AzureML	https://youtu.be/yBVXR8G8Bg8
Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference	https://www.youtube.com/watch?v=cntxC3g22oU
[slides]	https://github.com/tensorwavecloud/DeepSpeed/blob/master/docs/assets/files/presentation-mlops.pdf
DeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim)	https://www.youtube.com/watch?v=pDGI668pNg0
Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher)	https://www.youtube.com/watch?v=tC01FRB0M7w
Ultimate Guide To Scaling ML Models (The AI Epiphany)	https://www.youtube.com/watch?v=hc0u4avAkuM
www.deepspeed.ai/	https://www.deepspeed.ai/
Readme	https://github.com/tensorwavecloud/DeepSpeed#readme-ov-file
Apache-2.0 license	https://github.com/tensorwavecloud/DeepSpeed#Apache-2.0-1-ov-file
Code of conduct	https://github.com/tensorwavecloud/DeepSpeed#coc-ov-file
Contributing	https://github.com/tensorwavecloud/DeepSpeed#contributing-ov-file
Security policy	https://github.com/tensorwavecloud/DeepSpeed#security-ov-file
Please reload this page	https://github.com/tensorwavecloud/DeepSpeed
Activity	https://github.com/tensorwavecloud/DeepSpeed/activity
Custom properties	https://github.com/tensorwavecloud/DeepSpeed/custom-properties
0 stars	https://github.com/tensorwavecloud/DeepSpeed/stargazers
0 watching	https://github.com/tensorwavecloud/DeepSpeed/watchers
0 forks	https://github.com/tensorwavecloud/DeepSpeed/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Ftensorwavecloud%2FDeepSpeed&report=tensorwavecloud+%28user%29
Releases	https://github.com/tensorwavecloud/DeepSpeed/releases
Packages 0	https://github.com/orgs/tensorwavecloud/packages?repo_name=DeepSpeed
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.