René's URL Explorer Experiment

Title: GitHub - sakogan/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. · GitHub

Open Graph Title: GitHub - sakogan/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

X Title: GitHub - sakogan/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - sakogan/DeepSpeed

Open Graph Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - sakogan/DeepSpeed

X Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - sakogan/DeepSpeed

Mail addresses
opencode@microsoft.com

Opengraph URL: https://github.com/sakogan/DeepSpeed

X: @github

direct link

Domain: github.com

route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:9f67ef32-66b7-07d0-5669-f4de0dff4c6e
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	D548:12297D:21DB01:2DD2B3:6A4D3384
html-safe-nonce	b50a1bba8bf54706829ca59ef1283f7ffbed7598197ecdb9510ae0a0556b3308
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTQ4OjEyMjk3RDoyMURCMDE6MkREMkIzOjZBNEQzMzg0IiwidmlzaXRvcl9pZCI6IjU4Nzk1NzU2NzMyNTM0NzUyMDQiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac	1ec1d452da96b2d5ce625867c01cc365b884ee21dcbee6322baf1d18a38dccaf
hovercard-subject-tag	repository:635034611
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/sakogan/DeepSpeed
twitter:image	https://opengraph.githubassets.com/54493d87227def25ea2a87ae6f3631ca3c16e5a48cc3ace9e68fdbb4c4f0d7e4/sakogan/DeepSpeed
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/54493d87227def25ea2a87ae6f3631ca3c16e5a48cc3ace9e68fdbb4c4f0d7e4/sakogan/DeepSpeed
og:image:alt	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - sakogan/DeepSpeed
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	92571a8944142227b7e19cd10918b1ddd06e5066c1ad5bc7e4769cf6140a87e6
turbo-cache-control	no-cache
go-import	github.com/sakogan/DeepSpeed git https://github.com/sakogan/DeepSpeed.git
octolytics-dimension-user_id	82225080
octolytics-dimension-user_login	sakogan
octolytics-dimension-repository_id	635034611
octolytics-dimension-repository_nwo	sakogan/DeepSpeed
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	235860204
octolytics-dimension-repository_parent_nwo	deepspeedai/DeepSpeed
octolytics-dimension-repository_network_root_id	235860204
octolytics-dimension-repository_network_root_nwo	deepspeedai/DeepSpeed
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	56fc8347865a14e2ec811533d68f929cf4e0ec19
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/sakogan/DeepSpeed#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsakogan%2FDeepSpeed
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub Copilot appDirect agents from issue to merge	https://github.com/features/ai/github-app
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
View all resources	https://github.com/resources
GitHub SponsorsFund open source developers	https://github.com/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/accelerator
GitHub Stars	https://stars.github.com
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsakogan%2FDeepSpeed
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=sakogan%2FDeepSpeed
Reload	https://github.com/sakogan/DeepSpeed
Reload	https://github.com/sakogan/DeepSpeed
Reload	https://github.com/sakogan/DeepSpeed
sakogan	https://github.com/sakogan
DeepSpeed	https://github.com/sakogan/DeepSpeed
deepspeedai/DeepSpeed	https://github.com/deepspeedai/DeepSpeed
Notifications	https://github.com/login?return_to=%2Fsakogan%2FDeepSpeed
Fork 0	https://github.com/login?return_to=%2Fsakogan%2FDeepSpeed
Star 0	https://github.com/login?return_to=%2Fsakogan%2FDeepSpeed
Code	https://github.com/sakogan/DeepSpeed
Pull requests 0	https://github.com/sakogan/DeepSpeed/pulls
Actions	https://github.com/sakogan/DeepSpeed/actions
Projects	https://github.com/sakogan/DeepSpeed/projects
Security and quality 0	https://github.com/sakogan/DeepSpeed/security
Insights	https://github.com/sakogan/DeepSpeed/pulse
Code	https://github.com/sakogan/DeepSpeed
Pull requests	https://github.com/sakogan/DeepSpeed/pulls
Actions	https://github.com/sakogan/DeepSpeed/actions
Projects	https://github.com/sakogan/DeepSpeed/projects
Security and quality	https://github.com/sakogan/DeepSpeed/security
Insights	https://github.com/sakogan/DeepSpeed/pulse
	https://github.com/sakogan/DeepSpeed
Branches	https://github.com/sakogan/DeepSpeed/branches
Tags	https://github.com/sakogan/DeepSpeed/tags
	https://github.com/sakogan/DeepSpeed/branches
	https://github.com/sakogan/DeepSpeed/tags
1,828 Commits	https://github.com/sakogan/DeepSpeed/commits/master/
	https://github.com/sakogan/DeepSpeed/commits/master/
.github	https://github.com/sakogan/DeepSpeed/tree/master/.github
.github	https://github.com/sakogan/DeepSpeed/tree/master/.github
accelerator	https://github.com/sakogan/DeepSpeed/tree/master/accelerator
accelerator	https://github.com/sakogan/DeepSpeed/tree/master/accelerator
azure	https://github.com/sakogan/DeepSpeed/tree/master/azure
azure	https://github.com/sakogan/DeepSpeed/tree/master/azure
benchmarks	https://github.com/sakogan/DeepSpeed/tree/master/benchmarks
benchmarks	https://github.com/sakogan/DeepSpeed/tree/master/benchmarks
bin	https://github.com/sakogan/DeepSpeed/tree/master/bin
bin	https://github.com/sakogan/DeepSpeed/tree/master/bin
blogs	https://github.com/sakogan/DeepSpeed/tree/master/blogs
blogs	https://github.com/sakogan/DeepSpeed/tree/master/blogs
csrc	https://github.com/sakogan/DeepSpeed/tree/master/csrc
csrc	https://github.com/sakogan/DeepSpeed/tree/master/csrc
deepspeed	https://github.com/sakogan/DeepSpeed/tree/master/deepspeed
deepspeed	https://github.com/sakogan/DeepSpeed/tree/master/deepspeed
docker	https://github.com/sakogan/DeepSpeed/tree/master/docker
docker	https://github.com/sakogan/DeepSpeed/tree/master/docker
docs	https://github.com/sakogan/DeepSpeed/tree/master/docs
docs	https://github.com/sakogan/DeepSpeed/tree/master/docs
examples	https://github.com/sakogan/DeepSpeed/tree/master/examples
examples	https://github.com/sakogan/DeepSpeed/tree/master/examples
op_builder	https://github.com/sakogan/DeepSpeed/tree/master/op_builder
op_builder	https://github.com/sakogan/DeepSpeed/tree/master/op_builder
release	https://github.com/sakogan/DeepSpeed/tree/master/release
release	https://github.com/sakogan/DeepSpeed/tree/master/release
requirements	https://github.com/sakogan/DeepSpeed/tree/master/requirements
requirements	https://github.com/sakogan/DeepSpeed/tree/master/requirements
scripts	https://github.com/sakogan/DeepSpeed/tree/master/scripts
scripts	https://github.com/sakogan/DeepSpeed/tree/master/scripts
tests	https://github.com/sakogan/DeepSpeed/tree/master/tests
tests	https://github.com/sakogan/DeepSpeed/tree/master/tests
.clang-format	https://github.com/sakogan/DeepSpeed/blob/master/.clang-format
.clang-format	https://github.com/sakogan/DeepSpeed/blob/master/.clang-format
.flake8	https://github.com/sakogan/DeepSpeed/blob/master/.flake8
.flake8	https://github.com/sakogan/DeepSpeed/blob/master/.flake8
.gitignore	https://github.com/sakogan/DeepSpeed/blob/master/.gitignore
.gitignore	https://github.com/sakogan/DeepSpeed/blob/master/.gitignore
.pre-commit-config.yaml	https://github.com/sakogan/DeepSpeed/blob/master/.pre-commit-config.yaml
.pre-commit-config.yaml	https://github.com/sakogan/DeepSpeed/blob/master/.pre-commit-config.yaml
.pylintrc	https://github.com/sakogan/DeepSpeed/blob/master/.pylintrc
.pylintrc	https://github.com/sakogan/DeepSpeed/blob/master/.pylintrc
.readthedocs.yml	https://github.com/sakogan/DeepSpeed/blob/master/.readthedocs.yml
.readthedocs.yml	https://github.com/sakogan/DeepSpeed/blob/master/.readthedocs.yml
.style.yapf	https://github.com/sakogan/DeepSpeed/blob/master/.style.yapf
.style.yapf	https://github.com/sakogan/DeepSpeed/blob/master/.style.yapf
CODEOWNERS	https://github.com/sakogan/DeepSpeed/blob/master/CODEOWNERS
CODEOWNERS	https://github.com/sakogan/DeepSpeed/blob/master/CODEOWNERS
CODE_OF_CONDUCT.md	https://github.com/sakogan/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CODE_OF_CONDUCT.md	https://github.com/sakogan/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CONTRIBUTING.md	https://github.com/sakogan/DeepSpeed/blob/master/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/sakogan/DeepSpeed/blob/master/CONTRIBUTING.md
LICENSE	https://github.com/sakogan/DeepSpeed/blob/master/LICENSE
LICENSE	https://github.com/sakogan/DeepSpeed/blob/master/LICENSE
MANIFEST.in	https://github.com/sakogan/DeepSpeed/blob/master/MANIFEST.in
MANIFEST.in	https://github.com/sakogan/DeepSpeed/blob/master/MANIFEST.in
MANIFEST_win.in	https://github.com/sakogan/DeepSpeed/blob/master/MANIFEST_win.in
MANIFEST_win.in	https://github.com/sakogan/DeepSpeed/blob/master/MANIFEST_win.in
README.md	https://github.com/sakogan/DeepSpeed/blob/master/README.md
README.md	https://github.com/sakogan/DeepSpeed/blob/master/README.md
SECURITY.md	https://github.com/sakogan/DeepSpeed/blob/master/SECURITY.md
SECURITY.md	https://github.com/sakogan/DeepSpeed/blob/master/SECURITY.md
build_win.bat	https://github.com/sakogan/DeepSpeed/blob/master/build_win.bat
build_win.bat	https://github.com/sakogan/DeepSpeed/blob/master/build_win.bat
environment.yml	https://github.com/sakogan/DeepSpeed/blob/master/environment.yml
environment.yml	https://github.com/sakogan/DeepSpeed/blob/master/environment.yml
install.sh	https://github.com/sakogan/DeepSpeed/blob/master/install.sh
install.sh	https://github.com/sakogan/DeepSpeed/blob/master/install.sh
setup.cfg	https://github.com/sakogan/DeepSpeed/blob/master/setup.cfg
setup.cfg	https://github.com/sakogan/DeepSpeed/blob/master/setup.cfg
setup.py	https://github.com/sakogan/DeepSpeed/blob/master/setup.py
setup.py	https://github.com/sakogan/DeepSpeed/blob/master/setup.py
version.txt	https://github.com/sakogan/DeepSpeed/blob/master/version.txt
version.txt	https://github.com/sakogan/DeepSpeed/blob/master/version.txt
README	https://github.com/sakogan/DeepSpeed
Code of conduct	https://github.com/sakogan/DeepSpeed
Contributing	https://github.com/sakogan/DeepSpeed
License	https://github.com/sakogan/DeepSpeed
Security	https://github.com/sakogan/DeepSpeed
	https://github.com/Microsoft/DeepSpeed/blob/master/LICENSE
	https://pypi.org/project/deepspeed/
	https://pepy.tech/project/deepspeed
	https://github.com/sakogan/DeepSpeed#build-pipeline-status
	https://twitter.com/intent/follow?screen_name=MSFTDeepSpeed
	https://twitter.com/MSFTDeepSpeedJP
	https://www.zhihu.com/people/deepspeed
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only
	https://github.com/sakogan/DeepSpeed#latest-news
learn how	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat
DeepSpeed4Science website	https://deepspeed4science.ai/
Tutorials	https://www.deepspeed.ai/deepspeed4science/
Blog	https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed4science/japanese/README.md
DeepSpeed ZeRO-Inference: 20X faster inference through weight quantization and KV cache offloading	https://github.com/microsoft/DeepSpeedExamples/blob/master/inference/huggingface/zero_inference/README.md
DeepSpeed-Chat: Llama/Llama-2 system support, efficiency boost, and training stability improvements	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat/ds-chat-release-8-31/README.md
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-ulysses
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ulysses/japanese/README.md
ZeRO++: A leap in speed for LLM and chat model training with 4X less communication	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
English	https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/
中文	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/chinese/README.md
日本語	https://github.com/microsoft/DeepSpeed/blob/master/blogs/zeropp/japanese/README.md
	https://github.com/sakogan/DeepSpeed#extreme-speed-and-scale-for-dl-training-and-inference
DeepSpeed	https://www.deepspeed.ai/
MT-530B	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
BLOOM	https://huggingface.co/blog/bloom-megatron-deepspeed
	https://github.com/sakogan/DeepSpeed#deepspeeds-four-innovation-pillars
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/DeepSpeed-pillars.png
	https://github.com/sakogan/DeepSpeed#deepspeed-training
DeepSpeed-Training	https://www.deepspeed.ai/training/
	https://github.com/sakogan/DeepSpeed#deepspeed-inference
DeepSpeed-Inference	https://www.deepspeed.ai/inference
	https://github.com/sakogan/DeepSpeed#deepspeed-compression
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/sakogan/DeepSpeed#deepspeed4science
DeepSpeed4Science website	https://deepspeed4science.ai/
tutorials	https://www.deepspeed.ai/deepspeed4science/
	https://github.com/sakogan/DeepSpeed#deepspeed-software-suite
	https://github.com/sakogan/DeepSpeed#deepspeed-library
DeepSpeed	https://github.com/microsoft/deepspeed
DeepSpeed Adoption	https://github.com/sakogan/DeepSpeed#deepspeed-adoption
	https://github.com/sakogan/DeepSpeed#model-implementations-for-inference-mii
Model Implementations for Inference (MII)	https://github.com/microsoft/deepspeed-mii
	https://github.com/sakogan/DeepSpeed#deepspeed-on-azure
recipes	https://github.com/Azure/azureml-examples/tree/main/v1/python-sdk/workflows/train/deepspeed
here	https://github.com/microsoft/Megatron-DeepSpeed/tree/main/examples_deepspeed/azureml
Azure tutorial	https://www.deepspeed.ai/tutorials/azure/
	https://github.com/sakogan/DeepSpeed#deepspeed-adoption
AI at Scale	https://www.microsoft.com/en-us/research/project/ai-at-scale/
here	https://innovation.microsoft.com/en-us/exploring-ai-at-scale
Megatron-Turing NLG (530B)	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
Jurassic-1 (178B)	https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
BLOOM (176B)	https://huggingface.co/blog/bloom-megatron-deepspeed
GLM (130B)	https://github.com/THUDM/GLM-130B
xTrimoPGLM (100B)	https://www.biorxiv.org/content/10.1101/2023.07.05.547496v2
YaLM (100B)	https://github.com/yandex/YaLM-100B
GPT-NeoX (20B)	https://github.com/EleutherAI/gpt-neox
AlexaTM (20B)	https://www.amazon.science/blog/20b-parameter-alexa-model-sets-new-marks-in-few-shot-learning
Turing NLG (17B)	https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
METRO-LM (5.4B)	https://arxiv.org/pdf/2204.06644.pdf
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/transformers-light.png#gh-light-mode-only
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/transformers-dark.png#gh-dark-mode-only
Transformers with DeepSpeed	https://huggingface.co/docs/transformers/main/main_classes/deepspeed
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/accelerate-light.png#gh-light-mode-only
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/accelerate-dark.png#gh-dark-mode-only
Accelerate with DeepSpeed	https://huggingface.co/docs/accelerate/usage_guides/deepspeed
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/lightning-light.svg#gh-light-mode-only
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/lightning-dark.svg#gh-dark-mode-only
Lightning with DeepSpeed	https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#deepspeed
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/mosaicml.svg
MosaicML with DeepSpeed	https://docs.mosaicml.com/projects/composer/en/latest/trainer/using_the_trainer.html?highlight=deepspeed#deepspeed-integration
	https://github.com/sakogan/DeepSpeed/blob/master/docs/assets/images/determined.svg
Determined with DeepSpeed	https://docs.determined.ai/latest/training/apis-howto/deepspeed/overview.html
	https://user-images.githubusercontent.com/58739961/187154444-fce76639-ac8d-429b-9354-c6fac64b7ef8.jpg
MMEngine with DeepSpeed	https://mmengine.readthedocs.io/en/latest/common_usage/large_model_training.html#deepspeed
	https://github.com/sakogan/DeepSpeed#build-pipeline-status
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-p40.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch110-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-h100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-nightly.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd-mi100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd-mi200.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-cpu.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-nightly-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-transformers-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-lightning-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-accelerate-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-megatron.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-mii.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-ds-chat.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/formatting.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/pages/pages-build-deployment
	https://deepspeed.readthedocs.io/en/latest/?badge=latest
	https://github.com/microsoft/DeepSpeed/actions/workflows/python.yml
	https://github.com/sakogan/DeepSpeed#installation
torch's JIT C++ extension loader that relies on ninja	https://pytorch.org/docs/stable/cpp_extension.html
	https://github.com/sakogan/DeepSpeed#requirements
PyTorch	https://pytorch.org/
nvcc	https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#introduction
hipcc	https://github.com/ROCm-Developer-Tools/HIPCC
	https://github.com/sakogan/DeepSpeed#pypi
PyPI	https://pypi.org/project/deepspeed/
advanced installation instructions	https://www.deepspeed.ai/tutorials/advanced-install/
	https://github.com/sakogan/DeepSpeed#windows
	https://github.com/sakogan/DeepSpeed#features
DeepSpeed-Training	https://www.deepspeed.ai/training
DeepSpeed-Inference	https://www.deepspeed.ai/inference
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/sakogan/DeepSpeed#further-reading
deepspeed.ai	https://www.deepspeed.ai/
Getting Started	https://www.deepspeed.ai/getting-started/
DeepSpeed JSON Configuration	https://www.deepspeed.ai/docs/config-json/
API Documentation	https://deepspeed.readthedocs.io/en/latest/
Tutorials	https://www.deepspeed.ai/tutorials/
Blogs	https://www.deepspeed.ai/posts/
	https://github.com/sakogan/DeepSpeed#contributing
contributing	https://github.com/sakogan/DeepSpeed/blob/master/CONTRIBUTING.md
	https://github.com/microsoft/DeepSpeed/graphs/contributors
	https://github.com/sakogan/DeepSpeed#contributor-license-agreement
https://cla.opensource.microsoft.com	https://cla.opensource.microsoft.com
	https://github.com/sakogan/DeepSpeed#code-of-conduct
Microsoft Open Source Code of Conduct	https://opensource.microsoft.com/codeofconduct/
Code of Conduct FAQ	https://opensource.microsoft.com/codeofconduct/faq/
	https://github.com/sakogan/DeepSpeed#publications
arXiv:1910.02054	https://arxiv.org/abs/1910.02054
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20)	https://dl.acm.org/doi/10.5555/3433701.3433727
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial)	https://dl.acm.org/doi/10.1145/3394486.3406703
arXiv:2010.13369	https://arxiv.org/abs/2010.13369
NeurIPS 2020	https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html
arXiv:2101.06840	https://arxiv.org/abs/2101.06840
USENIX ATC 2021	https://www.usenix.org/conference/atc21/presentation/ren-jie
arXiv:2102.02888	https://arxiv.org/abs/2102.02888
ICML 2021	http://proceedings.mlr.press/v139/tang21a.html
arXiv:2104.07857	https://arxiv.org/abs/2104.07857
SC 2021	https://dl.acm.org/doi/abs/10.1145/3458817.3476205
arXiv:2104.06069	https://arxiv.org/abs/2104.06069
HiPC 2022	https://hipc.org/advance-program/
arXiv:2108.06084	https://arxiv.org/abs/2108.06084
NeurIPS 2022	https://openreview.net/forum?id=JpZ5du_Kdh
arXiv:2202.06009	https://arxiv.org/abs/2202.06009
arXiv:2201.05596	https://arxiv.org/abs/2201.05596
ICML 2022	https://proceedings.mlr.press/v162/rajbhandari22a.html
arXiv:2201.11990	https://arxiv.org/abs/2201.11990
arXiv:2206.01859	https://arxiv.org/abs/2206.01859
NeurIPS 2022	https://openreview.net/forum?id=xNeAhc2CNAl
arXiv:2206.01861	https://arxiv.org/abs/2206.01861
NeurIPS 2022	https://openreview.net/forum?id=f-fVCElZ-G1
arXiv:2207.00032	https://arxiv.org/abs/2207.00032
SC 2022	https://dl.acm.org/doi/abs/10.5555/3571885.3571946
arXiv:2211.11586	https://arxiv.org/abs/2211.11586
arXiv:2212.03597	https://arxiv.org/abs/2212.03597
arXiv:2301.12017	https://arxiv.org/abs/2301.12017
ICLR:2023	https://openreview.net/forum?id=Pgtn4l6eKjv
arXiv:2303.07226	https://arxiv.org/abs/2303.07226
arXiv:2303.08374	https://arxiv.org/abs/2303.08374
arXiv:2303.06318	https://arxiv.org/abs/2303.06318
arXiv:2306.10209	https://arxiv.org/abs/2306.10209
arXiv:2303.08302	https://arxiv.org/abs/2303.08302
arXiv:2305.09847	https://arxiv.org/abs/2305.09847
arXiv:2308.01320	https://arxiv.org/abs/2308.01320
arXiv:2307.09782	https://arxiv.org/abs/2307.09782
	https://github.com/sakogan/DeepSpeed#videos
Overview	https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29
ZeRO + large model training	https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28
17B T-NLG demo	https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27
Fastest BERT training + RScan tuning	https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26
part 1	https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92
part 2	https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94
part 3	https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93
FAQ	https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24
ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed	https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html
DeepSpeed on AzureML	https://youtu.be/yBVXR8G8Bg8
DeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim)	https://www.youtube.com/watch?v=pDGI668pNg0
Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher)	https://www.youtube.com/watch?v=tC01FRB0M7w
Ultimate Guide To Scaling ML Models (The AI Epiphany)	https://www.youtube.com/watch?v=hc0u4avAkuM
www.deepspeed.ai/	https://www.deepspeed.ai/
Readme	https://github.com/sakogan/DeepSpeed#readme-ov-file
Apache-2.0 license	https://github.com/sakogan/DeepSpeed#Apache-2.0-1-ov-file
Code of conduct	https://github.com/sakogan/DeepSpeed#coc-ov-file
Contributing	https://github.com/sakogan/DeepSpeed#contributing-ov-file
Security policy	https://github.com/sakogan/DeepSpeed#security-ov-file
Please reload this page	https://github.com/sakogan/DeepSpeed
Activity	https://github.com/sakogan/DeepSpeed/activity
0 forks	https://github.com/sakogan/DeepSpeed/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fsakogan%2FDeepSpeed&report=sakogan+%28user%29
Releases	https://github.com/sakogan/DeepSpeed/releases
Packages 0	https://github.com/users/sakogan/packages?repo_name=DeepSpeed
Please reload this page	https://github.com/sakogan/DeepSpeed
Contributors	https://github.com/sakogan/DeepSpeed/graphs/contributors
Please reload this page	https://github.com/sakogan/DeepSpeed
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.