René's URL Explorer Experiment

Title: GitHub - DistributedDeepLearning/DeepSpeed at workaround-zero3 · GitHub

Open Graph Title: GitHub - DistributedDeepLearning/DeepSpeed at workaround-zero3

X Title: GitHub - DistributedDeepLearning/DeepSpeed at workaround-zero3

Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - GitHub - DistributedDeepLearning/DeepSpeed at workaround-zero3

Open Graph Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DistributedDeepLearning/DeepSpeed

X Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DistributedDeepLearning/DeepSpeed

Mail addresses
opencode@microsoft.com

Opengraph URL: https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3

X: @github

direct link

Domain: github.com

route-pattern	/:user_id/:repository/tree/name(/path)
route-controller	files
route-action	disambiguate
fetch-nonce	v2:9590637a-afd9-3221-6376-e1feb283ccd5
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	A4B0:2371E7:307E610:42F6FA8:6A507DC1
html-safe-nonce	f078f4637d83af222cbd5a7f6199df367d8c6bafcf58dd47d34209691dc339e7
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNEIwOjIzNzFFNzozMDdFNjEwOjQyRjZGQTg6NkE1MDdEQzEiLCJ2aXNpdG9yX2lkIjoiNTg0NjgxNDkwMzk2NTM1MTM2MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac	abbde819256148698cfb2cb83bb97d6ffc657be85994a005c333c07ee6011e6f
hovercard-subject-tag	repository:510079179
github-keyboard-shortcuts	repository,source-code,file-tree,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	///files/disambiguate
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
twitter:image	https://opengraph.githubassets.com/833afe2fb91999facb09646f085f330e5b2b02dfc738744bc501c02e081cebb5/DistributedDeepLearning/DeepSpeed
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/833afe2fb91999facb09646f085f330e5b2b02dfc738744bc501c02e081cebb5/DistributedDeepLearning/DeepSpeed
og:image:alt	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DistributedDeepLearning/DeepSpeed
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	d6dc8294eb500fa36bbc57472d61fe87c522f9c3c1d64f70f4926f66a66a7efb
turbo-cache-control	no-cache
go-import	github.com/DistributedDeepLearning/DeepSpeed git https://github.com/DistributedDeepLearning/DeepSpeed.git
octolytics-dimension-user_id	108561606
octolytics-dimension-user_login	DistributedDeepLearning
octolytics-dimension-repository_id	510079179
octolytics-dimension-repository_nwo	DistributedDeepLearning/DeepSpeed
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	235860204
octolytics-dimension-repository_parent_nwo	deepspeedai/DeepSpeed
octolytics-dimension-repository_network_root_id	235860204
octolytics-dimension-repository_network_root_nwo	deepspeedai/DeepSpeed
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	7ac0ad2f2c7e4b9056617355fd9e33e22b0c8df9
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FDistributedDeepLearning%2FDeepSpeed%2Ftree%2Fworkaround-zero3
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub Copilot appDirect agents from issue to merge	https://github.com/features/ai/github-app
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
View all resources	https://github.com/resources
GitHub SponsorsFund open source developers	https://github.com/open-source/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/open-source/accelerator
GitHub Stars	https://stars.github.com
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/enterprise/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FDistributedDeepLearning%2FDeepSpeed%2Ftree%2Fworkaround-zero3
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Ffiles%2Fdisambiguate&source=header-repo&source_repo=DistributedDeepLearning%2FDeepSpeed
Reload	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Reload	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Reload	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Please reload this page	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
DistributedDeepLearning	https://github.com/DistributedDeepLearning
DeepSpeed	https://github.com/DistributedDeepLearning/DeepSpeed
deepspeedai/DeepSpeed	https://github.com/deepspeedai/DeepSpeed
Notifications	https://github.com/login?return_to=%2FDistributedDeepLearning%2FDeepSpeed
Fork 0	https://github.com/login?return_to=%2FDistributedDeepLearning%2FDeepSpeed
Star 0	https://github.com/login?return_to=%2FDistributedDeepLearning%2FDeepSpeed
Code	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Pull requests 0	https://github.com/DistributedDeepLearning/DeepSpeed/pulls
Actions	https://github.com/DistributedDeepLearning/DeepSpeed/actions
Projects	https://github.com/DistributedDeepLearning/DeepSpeed/projects
Security and quality 0	https://github.com/DistributedDeepLearning/DeepSpeed/security
Insights	https://github.com/DistributedDeepLearning/DeepSpeed/pulse
Code	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Pull requests	https://github.com/DistributedDeepLearning/DeepSpeed/pulls
Actions	https://github.com/DistributedDeepLearning/DeepSpeed/actions
Projects	https://github.com/DistributedDeepLearning/DeepSpeed/projects
Security and quality	https://github.com/DistributedDeepLearning/DeepSpeed/security
Insights	https://github.com/DistributedDeepLearning/DeepSpeed/pulse
	https://github.com/DistributedDeepLearning/DeepSpeed
Branches	https://github.com/DistributedDeepLearning/DeepSpeed/branches
Tags	https://github.com/DistributedDeepLearning/DeepSpeed/tags
	https://github.com/DistributedDeepLearning/DeepSpeed/branches
	https://github.com/DistributedDeepLearning/DeepSpeed/tags
941 Commits	https://github.com/DistributedDeepLearning/DeepSpeed/commits/workaround-zero3/
	https://github.com/DistributedDeepLearning/DeepSpeed/commits/workaround-zero3/
.github	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/.github
.github	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/.github
DeepSpeedExamples @ 36212dd	https://github.com/microsoft/DeepSpeedExamples/tree/36212dd59cb3eb342c39bc8965aaba04d5491933
DeepSpeedExamples @ 36212dd	https://github.com/microsoft/DeepSpeedExamples/tree/36212dd59cb3eb342c39bc8965aaba04d5491933
azure	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/azure
azure	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/azure
bin	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/bin
bin	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/bin
csrc	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/csrc
csrc	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/csrc
deepspeed	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/deepspeed
deepspeed	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/deepspeed
docker	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/docker
docker	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/docker
docs	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/docs
docs	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/docs
op_builder	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/op_builder
op_builder	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/op_builder
release	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/release
release	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/release
requirements	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/requirements
requirements	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/requirements
tests	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/tests
tests	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3/tests
.clang-format	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.clang-format
.clang-format	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.clang-format
.gitignore	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.gitignore
.gitignore	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.gitignore
.gitmodules	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.gitmodules
.gitmodules	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.gitmodules
.pre-commit-config.yaml	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.pre-commit-config.yaml
.pre-commit-config.yaml	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.pre-commit-config.yaml
.pylintrc	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.pylintrc
.pylintrc	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.pylintrc
.readthedocs.yml	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.readthedocs.yml
.readthedocs.yml	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.readthedocs.yml
.style.yapf	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.style.yapf
.style.yapf	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/.style.yapf
CODEOWNERS	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CODEOWNERS
CODEOWNERS	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CODEOWNERS
CODE_OF_CONDUCT.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CODE_OF_CONDUCT.md
CODE_OF_CONDUCT.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CODE_OF_CONDUCT.md
CONTRIBUTING.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CONTRIBUTING.md
LICENSE	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/LICENSE
LICENSE	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/LICENSE
MANIFEST.in	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/MANIFEST.in
MANIFEST.in	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/MANIFEST.in
MANIFEST_win.in	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/MANIFEST_win.in
MANIFEST_win.in	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/MANIFEST_win.in
README.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/README.md
README.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/README.md
SECURITY.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/SECURITY.md
SECURITY.md	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/SECURITY.md
install.sh	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/install.sh
install.sh	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/install.sh
setup.cfg	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/setup.cfg
setup.cfg	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/setup.cfg
setup.py	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/setup.py
setup.py	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/setup.py
version.txt	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/version.txt
version.txt	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/version.txt
README	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Code of conduct	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Contributing	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
License	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Security	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
	https://github.com/microsoft/DeepSpeed/actions
	https://pypi.org/project/deepspeed/
	https://deepspeed.readthedocs.io/en/latest/?badge=latest
	https://github.com/Microsoft/DeepSpeed/blob/master/LICENSE
	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only
	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#latest-news
Supporting efficient large model training on AMD Instinct GPUs with DeepSpeed	https://cloudblogs.microsoft.com/opensource/2022/03/21/supporting-efficient-large-model-training-on-amd-instinct-gpus-with-deepspeed/
Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam	https://www.deepspeed.ai/tutorials/zero-one-adam/
DeepSpeed: Advancing MoE inference and training to power next-generation AI scale	https://www.microsoft.com/en-us/research/blog/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale/
Mixture of Experts (MoE) for NLG tutorial	https://www.deepspeed.ai/tutorials/mixture-of-experts-nlg/
Mixture of Experts (MoE) Inference tutorial	https://www.deepspeed.ai/tutorials/moe-inference-tutorial
Autotuning: Automatically discover the optimal DeepSpeed configuration that delivers good training speed	https://www.deepspeed.ai/news/2021/11/15/autotuning.html
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World’s Largest and Most Powerful Generative Language Model	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
train large models with DeepSpeed	https://www.deepspeed.ai/tutorials/large-models-w-deepspeed/
come join us!	https://careers.microsoft.com/us/en/search-results?keywords=http:%2F%2Fdeepspeed.ai
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#deepspeed-is-hiring-come-join-us
DeepSpeed	https://www.deepspeed.ai/
Turing-NLG	https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft
AI at Scale	https://www.microsoft.com/en-us/research/project/ai-at-scale/
here	https://innovation.microsoft.com/en-us/exploring-ai-at-scale
deepspeed.ai	https://www.deepspeed.ai/
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#table-of-contents
Why DeepSpeed?	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#why-deepspeed
Install	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#installation
Features	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#features
Further Reading	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#further-reading
Contributing	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#contributing
Publications	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#publications
Videos	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#videos
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#why-deepspeed
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#installation
torch's JIT C++ extension loader that relies on ninja	https://pytorch.org/docs/stable/cpp_extension.html
PyTorch	https://pytorch.org/
advanced installation instructions	https://www.deepspeed.ai/tutorials/advanced-install/
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#features
feature overview	https://www.deepspeed.ai/features/
Distributed Training with Mixed Precision	https://www.deepspeed.ai/features/#distributed-training-with-mixed-precision
Model Parallelism	https://www.deepspeed.ai/features/#model-parallelism
Pipeline Parallelism	https://www.deepspeed.ai/tutorials/pipeline/
The Zero Redundancy Optimizer (ZeRO)	https://www.deepspeed.ai/tutorials/zero/
ZeRO-Offload	https://www.deepspeed.ai/tutorials/zero-offload/
Ultra-fast dense transformer kernels	https://www.deepspeed.ai/2020/05/18/bert-record.html
Sparse attention	https://www.deepspeed.ai/2020/09/08/sparse-attention-news.html
1-bit Adam	https://www.deepspeed.ai/2020/09/08/onebit-adam-blog-post.html
0/1 Adam	https://www.deepspeed.ai/tutorials/zero-one-adam/
1-bit LAMB	https://www.deepspeed.ai/tutorials/onebit-lamb/
Additional Memory and Bandwidth Optimizations	https://www.deepspeed.ai/features/#additional-memory-and-bandwidth-optimizations
Training Features	https://www.deepspeed.ai/features/#training-features
Training Optimizers	https://www.deepspeed.ai/features/#training-optimizers
Training Agnostic Checkpointing	https://www.deepspeed.ai/features/#training-agnostic-checkpointing
Advanced Parameter Search	https://www.deepspeed.ai/features/#advanced-parameter-search
Simplified Data Loader	https://www.deepspeed.ai/features/#simplified-data-loader
Curriculum Learning	https://www.deepspeed.ai/tutorials/curriculum-learning/
Performance Analysis and Debugging	https://www.deepspeed.ai/features/#performance-analysis-and-debugging
Mixture of Experts (MoE)	https://www.deepspeed.ai/tutorials/mixture-of-experts/
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#further-reading
deepspeed.ai	https://www.deepspeed.ai/
DeepSpeed Features	https://www.deepspeed.ai/features/
Getting Started	https://www.deepspeed.ai/getting-started/
DeepSpeed JSON Configuration	https://www.deepspeed.ai/docs/config-json/
API Documentation	https://deepspeed.readthedocs.io/en/latest/
CIFAR-10 Tutorial	https://www.deepspeed.ai/tutorials/cifar-10
Megatron-LM Tutorial	https://www.deepspeed.ai/tutorials/megatron/
BERT Pre-training Tutorial	https://www.deepspeed.ai/tutorials/bert-pretraining/
Learning Rate Range Test Tutorial	https://www.deepspeed.ai/tutorials/lrrt/
1Cycle Tutorial	https://www.deepspeed.ai/tutorials/one-cycle/
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#contributing
contributing	https://github.com/DistributedDeepLearning/DeepSpeed/blob/workaround-zero3/CONTRIBUTING.md
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#contributor-license-agreement
https://cla.opensource.microsoft.com	https://cla.opensource.microsoft.com
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#code-of-conduct
Microsoft Open Source Code of Conduct	https://opensource.microsoft.com/codeofconduct/
Code of Conduct FAQ	https://opensource.microsoft.com/codeofconduct/faq/
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#publications
arXiv:1910.02054	https://arxiv.org/abs/1910.02054
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20)	https://dl.acm.org/doi/10.5555/3433701.3433727
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial)	https://dl.acm.org/doi/10.1145/3394486.3406703
arXiv:2010.13369	https://arxiv.org/abs/2010.13369
NeurIPS 2020	https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html
arXiv:2101.06840	https://arxiv.org/abs/2101.06840
arXiv:2102.02888	https://arxiv.org/abs/2102.02888
ICML 2021	http://proceedings.mlr.press/v139/tang21a.html
arXiv:2104.07857	https://arxiv.org/abs/2104.07857
arXiv:2104.06069	https://arxiv.org/abs/2104.06069
arXiv:2108.06084	https://arxiv.org/abs/2108.06084
arXiv:2202.06009	https://arxiv.org/abs/2202.06009
arXiv:2201.05596	https://arxiv.org/abs/2201.05596
	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#videos
Overview	https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29
ZeRO + large model training	https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28
17B T-NLG demo	https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27
Fastest BERT training + RScan tuning	https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26
part 1	https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92
part 2	https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94
part 3	https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93
FAQ	https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24
ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed	https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html
DeepSpeed on AzureML	https://youtu.be/yBVXR8G8Bg8
DeepSpeed: All the tricks to scale to gigantic models	https://www.youtube.com/watch?v=pDGI668pNg0
Turing-NLG, DeepSpeed and the ZeRO optimizer	https://www.youtube.com/watch?v=tC01FRB0M7w
www.deepspeed.ai/	https://www.deepspeed.ai/
Readme	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#readme-ov-file
MIT license	https://github.com/DistributedDeepLearning/DeepSpeed/blob/master/LICENSE
Code of conduct	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#coc-ov-file
Contributing	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#contributing-ov-file
Security policy	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3#security-ov-file
Please reload this page	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Activity	https://github.com/DistributedDeepLearning/DeepSpeed/activity
Custom properties	https://github.com/DistributedDeepLearning/DeepSpeed/custom-properties
0 forks	https://github.com/DistributedDeepLearning/DeepSpeed/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FDistributedDeepLearning%2FDeepSpeed&report=DistributedDeepLearning+%28user%29
Releases	https://github.com/DistributedDeepLearning/DeepSpeed/releases
43 tags	https://github.com/DistributedDeepLearning/DeepSpeed/tags
Packages 0	https://github.com/orgs/DistributedDeepLearning/packages?repo_name=DeepSpeed
Please reload this page	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
Contributors	https://github.com/DistributedDeepLearning/DeepSpeed/graphs/contributors
Please reload this page	https://github.com/DistributedDeepLearning/DeepSpeed/tree/workaround-zero3
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.