René's URL Explorer Experiment

Title: GitHub - dnuang/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. · GitHub

Open Graph Title: GitHub - dnuang/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

X Title: GitHub - dnuang/DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - dnuang/DeepSpeed

Open Graph Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - dnuang/DeepSpeed

X Description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - dnuang/DeepSpeed

Mail addresses
opencode@microsoft.com

Opengraph URL: https://github.com/dnuang/DeepSpeed

X: @github

direct link

Domain: github.com

route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:2088853b-65f0-d508-5752-55a35a0f42d6
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	C616:13657F:124B5D:1AC54B:6A4DFC62
html-safe-nonce	7c7bceaed16d877f2507466d11899a078e80082c9a2d2c1335c1d3d7face03e0
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNjE2OjEzNjU3RjoxMjRCNUQ6MUFDNTRCOjZBNERGQzYyIiwidmlzaXRvcl9pZCI6IjEzODI3MTIzNzEzNzQzMjQ4MzQiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac	34e7c4e370c87140067774a04fbfb5411f10a37b36c843f2da6aba5b0bd53f35
hovercard-subject-tag	repository:616788792
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/dnuang/DeepSpeed
twitter:image	https://opengraph.githubassets.com/cc23f6ea8c5bb3615356b4d44648a326b22d671ba15094d4a65d5ebe03563f13/dnuang/DeepSpeed
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/cc23f6ea8c5bb3615356b4d44648a326b22d671ba15094d4a65d5ebe03563f13/dnuang/DeepSpeed
og:image:alt	DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - dnuang/DeepSpeed
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	5818716c93c6a2925b815402541a32814e43a7b1261c322b0c2df75224289566
turbo-cache-control	no-cache
go-import	github.com/dnuang/DeepSpeed git https://github.com/dnuang/DeepSpeed.git
octolytics-dimension-user_id	18747061
octolytics-dimension-user_login	dnuang
octolytics-dimension-repository_id	616788792
octolytics-dimension-repository_nwo	dnuang/DeepSpeed
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	235860204
octolytics-dimension-repository_parent_nwo	deepspeedai/DeepSpeed
octolytics-dimension-repository_network_root_id	235860204
octolytics-dimension-repository_network_root_nwo	deepspeedai/DeepSpeed
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	4314b1df11fa8a565684f3a72dc971e3785da365
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/dnuang/DeepSpeed#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdnuang%2FDeepSpeed
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub Copilot appDirect agents from issue to merge	https://github.com/features/ai/github-app
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
View all resources	https://github.com/resources
GitHub SponsorsFund open source developers	https://github.com/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/accelerator
GitHub Stars	https://stars.github.com
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdnuang%2FDeepSpeed
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=dnuang%2FDeepSpeed
Reload	https://github.com/dnuang/DeepSpeed
Reload	https://github.com/dnuang/DeepSpeed
Reload	https://github.com/dnuang/DeepSpeed
dnuang	https://github.com/dnuang
DeepSpeed	https://github.com/dnuang/DeepSpeed
deepspeedai/DeepSpeed	https://github.com/deepspeedai/DeepSpeed
Notifications	https://github.com/login?return_to=%2Fdnuang%2FDeepSpeed
Fork 0	https://github.com/login?return_to=%2Fdnuang%2FDeepSpeed
Star 0	https://github.com/login?return_to=%2Fdnuang%2FDeepSpeed
Code	https://github.com/dnuang/DeepSpeed
Pull requests 0	https://github.com/dnuang/DeepSpeed/pulls
Actions	https://github.com/dnuang/DeepSpeed/actions
Projects	https://github.com/dnuang/DeepSpeed/projects
Security and quality 0	https://github.com/dnuang/DeepSpeed/security
Insights	https://github.com/dnuang/DeepSpeed/pulse
Code	https://github.com/dnuang/DeepSpeed
Pull requests	https://github.com/dnuang/DeepSpeed/pulls
Actions	https://github.com/dnuang/DeepSpeed/actions
Projects	https://github.com/dnuang/DeepSpeed/projects
Security and quality	https://github.com/dnuang/DeepSpeed/security
Insights	https://github.com/dnuang/DeepSpeed/pulse
	https://github.com/dnuang/DeepSpeed
Branches	https://github.com/dnuang/DeepSpeed/branches
Tags	https://github.com/dnuang/DeepSpeed/tags
	https://github.com/dnuang/DeepSpeed/branches
	https://github.com/dnuang/DeepSpeed/tags
1,382 Commits	https://github.com/dnuang/DeepSpeed/commits/master/
	https://github.com/dnuang/DeepSpeed/commits/master/
.github	https://github.com/dnuang/DeepSpeed/tree/master/.github
.github	https://github.com/dnuang/DeepSpeed/tree/master/.github
accelerator	https://github.com/dnuang/DeepSpeed/tree/master/accelerator
accelerator	https://github.com/dnuang/DeepSpeed/tree/master/accelerator
azure	https://github.com/dnuang/DeepSpeed/tree/master/azure
azure	https://github.com/dnuang/DeepSpeed/tree/master/azure
benchmarks	https://github.com/dnuang/DeepSpeed/tree/master/benchmarks
benchmarks	https://github.com/dnuang/DeepSpeed/tree/master/benchmarks
bin	https://github.com/dnuang/DeepSpeed/tree/master/bin
bin	https://github.com/dnuang/DeepSpeed/tree/master/bin
csrc	https://github.com/dnuang/DeepSpeed/tree/master/csrc
csrc	https://github.com/dnuang/DeepSpeed/tree/master/csrc
deepspeed	https://github.com/dnuang/DeepSpeed/tree/master/deepspeed
deepspeed	https://github.com/dnuang/DeepSpeed/tree/master/deepspeed
docker	https://github.com/dnuang/DeepSpeed/tree/master/docker
docker	https://github.com/dnuang/DeepSpeed/tree/master/docker
docs	https://github.com/dnuang/DeepSpeed/tree/master/docs
docs	https://github.com/dnuang/DeepSpeed/tree/master/docs
examples	https://github.com/dnuang/DeepSpeed/tree/master/examples
examples	https://github.com/dnuang/DeepSpeed/tree/master/examples
op_builder	https://github.com/dnuang/DeepSpeed/tree/master/op_builder
op_builder	https://github.com/dnuang/DeepSpeed/tree/master/op_builder
release	https://github.com/dnuang/DeepSpeed/tree/master/release
release	https://github.com/dnuang/DeepSpeed/tree/master/release
requirements	https://github.com/dnuang/DeepSpeed/tree/master/requirements
requirements	https://github.com/dnuang/DeepSpeed/tree/master/requirements
scripts	https://github.com/dnuang/DeepSpeed/tree/master/scripts
scripts	https://github.com/dnuang/DeepSpeed/tree/master/scripts
tests	https://github.com/dnuang/DeepSpeed/tree/master/tests
tests	https://github.com/dnuang/DeepSpeed/tree/master/tests
.clang-format	https://github.com/dnuang/DeepSpeed/blob/master/.clang-format
.clang-format	https://github.com/dnuang/DeepSpeed/blob/master/.clang-format
.gitignore	https://github.com/dnuang/DeepSpeed/blob/master/.gitignore
.gitignore	https://github.com/dnuang/DeepSpeed/blob/master/.gitignore
.pre-commit-config.yaml	https://github.com/dnuang/DeepSpeed/blob/master/.pre-commit-config.yaml
.pre-commit-config.yaml	https://github.com/dnuang/DeepSpeed/blob/master/.pre-commit-config.yaml
.pylintrc	https://github.com/dnuang/DeepSpeed/blob/master/.pylintrc
.pylintrc	https://github.com/dnuang/DeepSpeed/blob/master/.pylintrc
.readthedocs.yml	https://github.com/dnuang/DeepSpeed/blob/master/.readthedocs.yml
.readthedocs.yml	https://github.com/dnuang/DeepSpeed/blob/master/.readthedocs.yml
.style.yapf	https://github.com/dnuang/DeepSpeed/blob/master/.style.yapf
.style.yapf	https://github.com/dnuang/DeepSpeed/blob/master/.style.yapf
CODEOWNERS	https://github.com/dnuang/DeepSpeed/blob/master/CODEOWNERS
CODEOWNERS	https://github.com/dnuang/DeepSpeed/blob/master/CODEOWNERS
CODE_OF_CONDUCT.md	https://github.com/dnuang/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CODE_OF_CONDUCT.md	https://github.com/dnuang/DeepSpeed/blob/master/CODE_OF_CONDUCT.md
CONTRIBUTING.md	https://github.com/dnuang/DeepSpeed/blob/master/CONTRIBUTING.md
CONTRIBUTING.md	https://github.com/dnuang/DeepSpeed/blob/master/CONTRIBUTING.md
LICENSE	https://github.com/dnuang/DeepSpeed/blob/master/LICENSE
LICENSE	https://github.com/dnuang/DeepSpeed/blob/master/LICENSE
MANIFEST.in	https://github.com/dnuang/DeepSpeed/blob/master/MANIFEST.in
MANIFEST.in	https://github.com/dnuang/DeepSpeed/blob/master/MANIFEST.in
MANIFEST_win.in	https://github.com/dnuang/DeepSpeed/blob/master/MANIFEST_win.in
MANIFEST_win.in	https://github.com/dnuang/DeepSpeed/blob/master/MANIFEST_win.in
README.md	https://github.com/dnuang/DeepSpeed/blob/master/README.md
README.md	https://github.com/dnuang/DeepSpeed/blob/master/README.md
SECURITY.md	https://github.com/dnuang/DeepSpeed/blob/master/SECURITY.md
SECURITY.md	https://github.com/dnuang/DeepSpeed/blob/master/SECURITY.md
build_win.bat	https://github.com/dnuang/DeepSpeed/blob/master/build_win.bat
build_win.bat	https://github.com/dnuang/DeepSpeed/blob/master/build_win.bat
install.sh	https://github.com/dnuang/DeepSpeed/blob/master/install.sh
install.sh	https://github.com/dnuang/DeepSpeed/blob/master/install.sh
setup.cfg	https://github.com/dnuang/DeepSpeed/blob/master/setup.cfg
setup.cfg	https://github.com/dnuang/DeepSpeed/blob/master/setup.cfg
setup.py	https://github.com/dnuang/DeepSpeed/blob/master/setup.py
setup.py	https://github.com/dnuang/DeepSpeed/blob/master/setup.py
version.txt	https://github.com/dnuang/DeepSpeed/blob/master/version.txt
version.txt	https://github.com/dnuang/DeepSpeed/blob/master/version.txt
README	https://github.com/dnuang/DeepSpeed
Code of conduct	https://github.com/dnuang/DeepSpeed
Contributing	https://github.com/dnuang/DeepSpeed
License	https://github.com/dnuang/DeepSpeed
Security	https://github.com/dnuang/DeepSpeed
	https://github.com/Microsoft/DeepSpeed/blob/master/LICENSE
	https://pypi.org/project/deepspeed/
	https://pepy.tech/project/deepspeed
	https://github.com/dnuang/DeepSpeed#build-pipeline-status
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only
	https://github.com/dnuang/DeepSpeed#latest-news
MT-530B	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
BLOOM	https://huggingface.co/blog/bloom-megatron-deepspeed
learn how	https://www.deepspeed.ai/tutorials/large-models-w-deepspeed/
Automatic Tensor Parallelism: Enables tensor parallelism by default without providing an injection policy	https://www.deepspeed.ai/tutorials/automatic-tensor-parallelism/
DeepSpeed Data Efficiency: A composable library that makes better use of data, increases training efficiency, and improves model quality	https://www.deepspeed.ai/2022/12/11/data-efficiency.html
Stable Diffusion Image Generation under 1 second w. DeepSpeed MII	https://github.com/microsoft/DeepSpeed-MII/tree/main/examples/benchmark/txt2img
DeepSpeed-MII: instant speedup on 24,000+ open-source DL models with up to 40x cheaper inference	https://www.deepspeed.ai/2022/10/10/mii.html
ZeRO-Inference: Democratizing massive model inference	https://www.deepspeed.ai/2022/09/09/zero-inference.html
Azure and DeepSpeed empower easy-to-use and high-performance model training	https://azure.microsoft.com/en-us/blog/azure-empowers-easytouse-highperformance-and-hyperscale-model-training-using-deepspeed/
	https://github.com/dnuang/DeepSpeed#extreme-speed-and-scale-for-dl-training-and-inference
DeepSpeed	https://www.deepspeed.ai/
	https://github.com/dnuang/DeepSpeed#deepspeeds-three-innovation-pillars
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/3pillars.png
	https://github.com/dnuang/DeepSpeed#deepspeed-training
DeepSpeed-Training	https://www.deepspeed.ai/training/
	https://github.com/dnuang/DeepSpeed#deepspeed-inference
DeepSpeed-Inference	https://www.deepspeed.ai/inference
	https://github.com/dnuang/DeepSpeed#deepspeed-compression
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/dnuang/DeepSpeed#deepspeed-software-suite
	https://github.com/dnuang/DeepSpeed#deepspeed-library
DeepSpeed	https://github.com/microsoft/deepspeed
DeepSpeed Adoption	https://github.com/dnuang/DeepSpeed#deepspeed-adoption
	https://github.com/dnuang/DeepSpeed#model-implementations-for-inference-mii
Model Implementations for Inference (MII)	https://github.com/microsoft/deepspeed-mii
	https://github.com/dnuang/DeepSpeed#deepspeed-on-azure
recipes	https://github.com/Azure/azureml-examples/tree/main/v1/python-sdk/workflows/train/deepspeed
here	https://github.com/microsoft/Megatron-DeepSpeed/tree/main/examples/azureml
Azure tutorial	https://www.deepspeed.ai/tutorials/azure/
	https://github.com/dnuang/DeepSpeed#deepspeed-adoption
AI at Scale	https://www.microsoft.com/en-us/research/project/ai-at-scale/
here	https://innovation.microsoft.com/en-us/exploring-ai-at-scale
Megatron-Turing NLG (530B)	https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/
Jurassic-1 (178B)	https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
BLOOM (176B)	https://huggingface.co/blog/bloom-megatron-deepspeed
GLM (130B)	https://github.com/THUDM/GLM-130B
YaLM (100B)	https://github.com/yandex/YaLM-100B
GPT-NeoX (20B)	https://github.com/EleutherAI/gpt-neox
AlexaTM (20B)	https://www.amazon.science/blog/20b-parameter-alexa-model-sets-new-marks-in-few-shot-learning
Turing NLG (17B)	https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/
METRO-LM (5.4B)	https://arxiv.org/pdf/2204.06644.pdf
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/transformers-light.png#gh-light-mode-only
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/transformers-dark.png#gh-dark-mode-only
Transformers with DeepSpeed	https://huggingface.co/docs/transformers/main/main_classes/deepspeed
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/accelerate-light.png#gh-light-mode-only
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/accelerate-dark.png#gh-dark-mode-only
Accelerate with DeepSpeed	https://huggingface.co/docs/accelerate/usage_guides/deepspeed
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/lightning-light.svg#gh-light-mode-only
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/lightning-dark.svg#gh-dark-mode-only
Lightning with DeepSpeed	https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.strategies.DeepSpeedStrategy.html
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/mosaicml.svg
MosaicML with DeepSpeed	https://docs.mosaicml.com/en/latest/trainer/using_the_trainer.html?highlight=deepspeed#deepspeed-integration
	https://github.com/dnuang/DeepSpeed/blob/master/docs/assets/images/determined.svg
Determined with DeepSpeed	https://docs.determined.ai/latest/training/apis-howto/deepspeed/overview.html
	https://github.com/dnuang/DeepSpeed#build-pipeline-status
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch12-p40.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch18-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-latest-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-inference.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-nightly.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/amd.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-torch-nightly-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-transformers-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-lightning-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/nv-accelerate-v100.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/formatting.yml
	https://github.com/microsoft/DeepSpeed/actions/workflows/pages/pages-build-deployment
	https://deepspeed.readthedocs.io/en/latest/?badge=latest
	https://github.com/dnuang/DeepSpeed#installation
torch's JIT C++ extension loader that relies on ninja	https://pytorch.org/docs/stable/cpp_extension.html
	https://github.com/dnuang/DeepSpeed#requirements
PyTorch	https://pytorch.org/
nvcc	https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#introduction
hipcc	https://github.com/ROCm-Developer-Tools/HIPCC
	https://github.com/dnuang/DeepSpeed#pypi
PyPI	https://pypi.org/project/deepspeed/
advanced installation instructions	https://www.deepspeed.ai/tutorials/advanced-install/
	https://github.com/dnuang/DeepSpeed#windows
	https://github.com/dnuang/DeepSpeed#features
DeepSpeed-Training	https://www.deepspeed.ai/training
DeepSpeed-Inference	https://www.deepspeed.ai/inference
DeepSpeed-Compression	https://www.deepspeed.ai/compression
	https://github.com/dnuang/DeepSpeed#further-reading
deepspeed.ai	https://www.deepspeed.ai/
Getting Started	https://www.deepspeed.ai/getting-started/
DeepSpeed JSON Configuration	https://www.deepspeed.ai/docs/config-json/
API Documentation	https://deepspeed.readthedocs.io/en/latest/
Tutorials	https://www.deepspeed.ai/tutorials/
Blogs	https://www.deepspeed.ai/posts/
	https://github.com/dnuang/DeepSpeed#contributing
contributing	https://github.com/dnuang/DeepSpeed/blob/master/CONTRIBUTING.md
	https://github.com/dnuang/DeepSpeed#contributor-license-agreement
https://cla.opensource.microsoft.com	https://cla.opensource.microsoft.com
	https://github.com/dnuang/DeepSpeed#code-of-conduct
Microsoft Open Source Code of Conduct	https://opensource.microsoft.com/codeofconduct/
Code of Conduct FAQ	https://opensource.microsoft.com/codeofconduct/faq/
	https://github.com/dnuang/DeepSpeed#publications
arXiv:1910.02054	https://arxiv.org/abs/1910.02054
In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20)	https://dl.acm.org/doi/10.5555/3433701.3433727
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial)	https://dl.acm.org/doi/10.1145/3394486.3406703
arXiv:2010.13369	https://arxiv.org/abs/2010.13369
NeurIPS 2020	https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html
arXiv:2101.06840	https://arxiv.org/abs/2101.06840
USENIX ATC 2021	https://www.usenix.org/conference/atc21/presentation/ren-jie
arXiv:2102.02888	https://arxiv.org/abs/2102.02888
ICML 2021	http://proceedings.mlr.press/v139/tang21a.html
arXiv:2104.07857	https://arxiv.org/abs/2104.07857
SC 2021	https://dl.acm.org/doi/abs/10.1145/3458817.3476205
arXiv:2104.06069	https://arxiv.org/abs/2104.06069
HiPC 2022	https://hipc.org/advance-program/
arXiv:2108.06084	https://arxiv.org/abs/2108.06084
NeurIPS 2022	https://openreview.net/forum?id=JpZ5du_Kdh
arXiv:2202.06009	https://arxiv.org/abs/2202.06009
arXiv:2201.05596	https://arxiv.org/abs/2201.05596
ICML 2022	https://proceedings.mlr.press/v162/rajbhandari22a.html
arXiv:2201.11990	https://arxiv.org/abs/2201.11990
arXiv:2206.01859	https://arxiv.org/abs/2206.01859
NeurIPS 2022	https://openreview.net/forum?id=xNeAhc2CNAl
arXiv:2206.01861	https://arxiv.org/abs/2206.01861
NeurIPS 2022	https://openreview.net/forum?id=f-fVCElZ-G1
arXiv:2207.00032	https://arxiv.org/abs/2207.00032
SC 2022	https://dl.acm.org/doi/abs/10.5555/3571885.3571946
arXiv:2211.11586	https://arxiv.org/abs/2211.11586
arXiv:2212.03597	https://arxiv.org/abs/2212.03597
arXiv:2301.12017	https://arxiv.org/abs/2301.12017
ICLR:2023	https://openreview.net/forum?id=Pgtn4l6eKjv
arXiv:2303.07226	https://arxiv.org/abs/2303.07226
	https://github.com/dnuang/DeepSpeed#videos
Overview	https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29
ZeRO + large model training	https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28
17B T-NLG demo	https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27
Fastest BERT training + RScan tuning	https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26
part 1	https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92
part 2	https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94
part 3	https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93
FAQ	https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24
ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed	https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html
DeepSpeed on AzureML	https://youtu.be/yBVXR8G8Bg8
DeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim)	https://www.youtube.com/watch?v=pDGI668pNg0
Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher)	https://www.youtube.com/watch?v=tC01FRB0M7w
Ultimate Guide To Scaling ML Models (The AI Epiphany)	https://www.youtube.com/watch?v=hc0u4avAkuM
www.deepspeed.ai/	https://www.deepspeed.ai/
Readme	https://github.com/dnuang/DeepSpeed#readme-ov-file
MIT license	https://github.com/dnuang/DeepSpeed#MIT-1-ov-file
Please reload this page	https://github.com/dnuang/DeepSpeed
Activity	https://github.com/dnuang/DeepSpeed/activity
0 forks	https://github.com/dnuang/DeepSpeed/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fdnuang%2FDeepSpeed&report=dnuang+%28user%29
Releases	https://github.com/dnuang/DeepSpeed/releases
Packages 0	https://github.com/users/dnuang/packages?repo_name=DeepSpeed
Please reload this page	https://github.com/dnuang/DeepSpeed
Contributors	https://github.com/dnuang/DeepSpeed/graphs/contributors
Please reload this page	https://github.com/dnuang/DeepSpeed
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.