| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository |
| route-controller | files |
| route-action | disambiguate |
| fetch-nonce | v2:5aaa7c35-54e2-1875-20f8-767650aeb9c2 |
| current-catalog-service-hash | f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb |
| request-id | BF1C:6991B:B83DE:EA9A7:6AFF8E96 |
| html-safe-nonce | ada33eef4f52c2faccee47648f8c5c69b667dfa1759bad7286577ffd476ce187 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRjFDOjY5OTFCOkI4M0RFOkVBOUE3OjZBRkY4RTk2IiwidmlzaXRvcl9pZCI6IjMxNjg2MTYyMjYxMTI5MDA3NTgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 34e734faf63ef444fb26b0629ab76b319bcd27807c110ad45e807e4a9d593bd4 |
| hovercard-subject-tag | repository:1206656085 |
| github-keyboard-shortcuts | repository,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | // |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/LLAI-lab/DeepSpeed |
| twitter:image | https://opengraph.githubassets.com/85e4ed4236c3e9e5cc7b67334b9e8ca49d8d16f63960d180166c0ba283abdf09/LLAI-lab/DeepSpeed |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/85e4ed4236c3e9e5cc7b67334b9e8ca49d8d16f63960d180166c0ba283abdf09/LLAI-lab/DeepSpeed |
| og:image:alt | DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - LLAI-lab/DeepSpeed |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 587ad99a3e443b2e1cb0d3df059ec49ab54389861f13310766dfdeb615fed90d |
| turbo-cache-control | no-cache |
| go-import | github.com/LLAI-lab/DeepSpeed git https://github.com/LLAI-lab/DeepSpeed.git |
| octolytics-dimension-user_id | 29222665 |
| octolytics-dimension-user_login | LLAI-lab |
| octolytics-dimension-repository_id | 1206656085 |
| octolytics-dimension-repository_nwo | LLAI-lab/DeepSpeed |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | true |
| octolytics-dimension-repository_parent_id | 235860204 |
| octolytics-dimension-repository_parent_nwo | deepspeedai/DeepSpeed |
| octolytics-dimension-repository_network_root_id | 235860204 |
| octolytics-dimension-repository_network_root_nwo | deepspeedai/DeepSpeed |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 5bb0fb68d64881d1344f5863d809515d496801e5 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| Skip to content | https://github.com/LLAI-lab/DeepSpeed#start-of-content |
| https://github.com/ |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FLLAI-lab%2FDeepSpeed |
| GitHubRepository hosting, code review, and collaboration | https://github.com/platform |
| CopilotAI coding agent for tasks, projects, and automated work | https://github.com/features/copilot |
| ActionsCI/CD and automation | https://github.com/features/actions |
| Code QualityQuality analysis, fixes, and enforcement | https://github.com/features/code-quality |
| CodespacesCloud development environments | https://github.com/features/codespaces |
| Code SecurityVulnerability detection and remediation | https://github.com/security/advanced-security/code-security |
| Secret ProtectionSecret scanning and leak prevention | https://github.com/security/advanced-security/secret-protection |
| All Features | https://github.com/features |
| Documentation | https://docs.github.com/en |
| Training | https://learn.github.com/skills |
| Certifications | https://learn.github.com/credentials |
| Build with Copilot | https://github.com/features/ai |
| Customize Copilot | https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview |
| Integrations | https://github.com/marketplace |
| Trending repos | https://github.com/trending |
| GitHub Sponsors | https://github.com/open-source/sponsors |
| Developer articles | https://github.com/resources/articles |
| ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place. | https://github.blog/changelog/ |
| Enterprises | https://github.com/enterprise |
| Team | https://github.com/team |
| Startups | https://github.com/enterprise/startups |
| Nonprofits | https://github.com/solutions/industry/nonprofits |
| Healthcare | https://github.com/solutions/industry/healthcare |
| Financial services | https://github.com/solutions/industry/financial-services |
| Manufacturing | https://github.com/solutions/industry/manufacturing |
| Government | https://github.com/solutions/industry/government |
| See all industries | https://github.com/solutions/industry |
| Why GitHub | https://github.com/why-github |
| Customer Stories | https://github.com/customer-stories |
| Business insights | https://github.com/solutions/executive-insights |
| Trust Center | https://github.com/trust-center |
| GitHub Advanced SecurityEnterprise-grade code security | https://github.com/security/advanced-security |
| Copilot for businessEnterprise-grade AI features | https://github.com/features/copilot/copilot-business |
| Premium SupportEnterprise-grade 24/7 support | https://github.com/enterprise/premium-support |
| Documentation | https://docs.github.com/en |
| Resource center | https://github.com/resources |
| Blog | https://github.blog/ |
| Events & webinars | https://github.com/resources/events |
| Community Forum | https://github.com/orgs/community/discussions |
| Customer Stories | https://github.com/customer-stories |
| Integrations | https://github.com/marketplace |
| Customer support | https://help.github.com/ |
| Training | https://learn.github.com/skills |
| GitHub Status | https://www.githubstatus.com/ |
| About | https://github.com/about |
| Partners | https://github.com/partners |
| Changelog | https://github.blog/changelog/ |
| GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub. | https://githubuniverse.com/ |
| Pricing | https://github.com/pricing |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FLLAI-lab%2FDeepSpeed |
| Sign up | https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=LLAI-lab%2FDeepSpeed |
| Reload | https://github.com/LLAI-lab/DeepSpeed |
| Reload | https://github.com/LLAI-lab/DeepSpeed |
| Reload | https://github.com/LLAI-lab/DeepSpeed |
| LLAI-lab | https://github.com/LLAI-lab |
| DeepSpeed | https://github.com/LLAI-lab/DeepSpeed |
| Code | https://github.com/LLAI-lab/DeepSpeed |
| Pull requests | https://github.com/LLAI-lab/DeepSpeed/pulls |
| Actions | https://github.com/LLAI-lab/DeepSpeed/actions |
| Projects | https://github.com/LLAI-lab/DeepSpeed/projects |
| Security and quality | https://github.com/LLAI-lab/DeepSpeed/security |
| Insights | https://github.com/LLAI-lab/DeepSpeed/pulse |
| https://github.com/LLAI-lab/DeepSpeed |
| DeepSpeed | https://github.com/LLAI-lab/DeepSpeed |
| deepspeedai/DeepSpeed | https://github.com/deepspeedai/DeepSpeed |
| Notifications | https://github.com/login?return_to=%2FLLAI-lab%2FDeepSpeed |
| Fork0 (0) | https://github.com/login?return_to=%2FLLAI-lab%2FDeepSpeed |
| Star1 (1) | https://github.com/login?return_to=%2FLLAI-lab%2FDeepSpeed |
| www.deepspeed.ai/ | https://www.deepspeed.ai/ |
| Readme | https://github.com/LLAI-lab/DeepSpeed#readme-ov-file |
| License | https://github.com/LLAI-lab/DeepSpeed#License-1-ov-file |
| Code of conduct | https://github.com/LLAI-lab/DeepSpeed#coc-ov-file |
| Contributing | https://github.com/LLAI-lab/DeepSpeed#contributing-ov-file |
| Security policy | https://github.com/LLAI-lab/DeepSpeed#security-ov-file |
| Activity | https://github.com/LLAI-lab/DeepSpeed/activity |
| 0 forks | https://github.com/LLAI-lab/DeepSpeed/forks |
| Report repository | https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FLLAI-lab%2FDeepSpeed&report=LLAI-lab+%28user%29 |
| Branches | https://github.com/LLAI-lab/DeepSpeed/branches |
| Tags | https://github.com/LLAI-lab/DeepSpeed/tags |
| https://github.com/LLAI-lab/DeepSpeed/branches |
| https://github.com/LLAI-lab/DeepSpeed/tags |
| 3,117 Commits | https://github.com/LLAI-lab/DeepSpeed/commits/master/ |
| https://github.com/LLAI-lab/DeepSpeed/commits/master/ |
| .github | https://github.com/LLAI-lab/DeepSpeed/tree/master/.github |
| .github | https://github.com/LLAI-lab/DeepSpeed/tree/master/.github |
| accelerator | https://github.com/LLAI-lab/DeepSpeed/tree/master/accelerator |
| accelerator | https://github.com/LLAI-lab/DeepSpeed/tree/master/accelerator |
| azure | https://github.com/LLAI-lab/DeepSpeed/tree/master/azure |
| azure | https://github.com/LLAI-lab/DeepSpeed/tree/master/azure |
| benchmarks | https://github.com/LLAI-lab/DeepSpeed/tree/master/benchmarks |
| benchmarks | https://github.com/LLAI-lab/DeepSpeed/tree/master/benchmarks |
| bin | https://github.com/LLAI-lab/DeepSpeed/tree/master/bin |
| bin | https://github.com/LLAI-lab/DeepSpeed/tree/master/bin |
| blogs | https://github.com/LLAI-lab/DeepSpeed/tree/master/blogs |
| blogs | https://github.com/LLAI-lab/DeepSpeed/tree/master/blogs |
| ci | https://github.com/LLAI-lab/DeepSpeed/tree/master/ci |
| ci | https://github.com/LLAI-lab/DeepSpeed/tree/master/ci |
| csrc | https://github.com/LLAI-lab/DeepSpeed/tree/master/csrc |
| csrc | https://github.com/LLAI-lab/DeepSpeed/tree/master/csrc |
| deepspeed | https://github.com/LLAI-lab/DeepSpeed/tree/master/deepspeed |
| deepspeed | https://github.com/LLAI-lab/DeepSpeed/tree/master/deepspeed |
| docker | https://github.com/LLAI-lab/DeepSpeed/tree/master/docker |
| docker | https://github.com/LLAI-lab/DeepSpeed/tree/master/docker |
| docs | https://github.com/LLAI-lab/DeepSpeed/tree/master/docs |
| docs | https://github.com/LLAI-lab/DeepSpeed/tree/master/docs |
| examples | https://github.com/LLAI-lab/DeepSpeed/tree/master/examples |
| examples | https://github.com/LLAI-lab/DeepSpeed/tree/master/examples |
| op_builder | https://github.com/LLAI-lab/DeepSpeed/tree/master/op_builder |
| op_builder | https://github.com/LLAI-lab/DeepSpeed/tree/master/op_builder |
| release | https://github.com/LLAI-lab/DeepSpeed/tree/master/release |
| release | https://github.com/LLAI-lab/DeepSpeed/tree/master/release |
| requirements | https://github.com/LLAI-lab/DeepSpeed/tree/master/requirements |
| requirements | https://github.com/LLAI-lab/DeepSpeed/tree/master/requirements |
| scripts | https://github.com/LLAI-lab/DeepSpeed/tree/master/scripts |
| scripts | https://github.com/LLAI-lab/DeepSpeed/tree/master/scripts |
| tests | https://github.com/LLAI-lab/DeepSpeed/tree/master/tests |
| tests | https://github.com/LLAI-lab/DeepSpeed/tree/master/tests |
| .clang-format | https://github.com/LLAI-lab/DeepSpeed/blob/master/.clang-format |
| .clang-format | https://github.com/LLAI-lab/DeepSpeed/blob/master/.clang-format |
| .flake8 | https://github.com/LLAI-lab/DeepSpeed/blob/master/.flake8 |
| .flake8 | https://github.com/LLAI-lab/DeepSpeed/blob/master/.flake8 |
| .gitignore | https://github.com/LLAI-lab/DeepSpeed/blob/master/.gitignore |
| .gitignore | https://github.com/LLAI-lab/DeepSpeed/blob/master/.gitignore |
| .gitmodules | https://github.com/LLAI-lab/DeepSpeed/blob/master/.gitmodules |
| .gitmodules | https://github.com/LLAI-lab/DeepSpeed/blob/master/.gitmodules |
| .pre-commit-config.yaml | https://github.com/LLAI-lab/DeepSpeed/blob/master/.pre-commit-config.yaml |
| .pre-commit-config.yaml | https://github.com/LLAI-lab/DeepSpeed/blob/master/.pre-commit-config.yaml |
| .pylintrc | https://github.com/LLAI-lab/DeepSpeed/blob/master/.pylintrc |
| .pylintrc | https://github.com/LLAI-lab/DeepSpeed/blob/master/.pylintrc |
| .readthedocs.yml | https://github.com/LLAI-lab/DeepSpeed/blob/master/.readthedocs.yml |
| .readthedocs.yml | https://github.com/LLAI-lab/DeepSpeed/blob/master/.readthedocs.yml |
| .style.yapf | https://github.com/LLAI-lab/DeepSpeed/blob/master/.style.yapf |
| .style.yapf | https://github.com/LLAI-lab/DeepSpeed/blob/master/.style.yapf |
| AGENTS.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/AGENTS.md |
| AGENTS.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/AGENTS.md |
| CLAUDE.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/CLAUDE.md |
| CLAUDE.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/CLAUDE.md |
| CODEOWNERS | https://github.com/LLAI-lab/DeepSpeed/blob/master/CODEOWNERS |
| CODEOWNERS | https://github.com/LLAI-lab/DeepSpeed/blob/master/CODEOWNERS |
| CODE_OF_CONDUCT.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/CODE_OF_CONDUCT.md |
| CODE_OF_CONDUCT.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/CODE_OF_CONDUCT.md |
| COMMITTERS.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/COMMITTERS.md |
| COMMITTERS.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/COMMITTERS.md |
| CONTRIBUTING.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/CONTRIBUTING.md |
| CONTRIBUTING.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/CONTRIBUTING.md |
| GOVERNANCE.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/GOVERNANCE.md |
| GOVERNANCE.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/GOVERNANCE.md |
| LICENSE | https://github.com/LLAI-lab/DeepSpeed/blob/master/LICENSE |
| LICENSE | https://github.com/LLAI-lab/DeepSpeed/blob/master/LICENSE |
| MANIFEST.in | https://github.com/LLAI-lab/DeepSpeed/blob/master/MANIFEST.in |
| MANIFEST.in | https://github.com/LLAI-lab/DeepSpeed/blob/master/MANIFEST.in |
| MANIFEST_win.in | https://github.com/LLAI-lab/DeepSpeed/blob/master/MANIFEST_win.in |
| MANIFEST_win.in | https://github.com/LLAI-lab/DeepSpeed/blob/master/MANIFEST_win.in |
| Makefile | https://github.com/LLAI-lab/DeepSpeed/blob/master/Makefile |
| Makefile | https://github.com/LLAI-lab/DeepSpeed/blob/master/Makefile |
| README.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/README.md |
| README.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/README.md |
| SECURITY.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/SECURITY.md |
| SECURITY.md | https://github.com/LLAI-lab/DeepSpeed/blob/master/SECURITY.md |
| build_win.bat | https://github.com/LLAI-lab/DeepSpeed/blob/master/build_win.bat |
| build_win.bat | https://github.com/LLAI-lab/DeepSpeed/blob/master/build_win.bat |
| environment.yml | https://github.com/LLAI-lab/DeepSpeed/blob/master/environment.yml |
| environment.yml | https://github.com/LLAI-lab/DeepSpeed/blob/master/environment.yml |
| install.sh | https://github.com/LLAI-lab/DeepSpeed/blob/master/install.sh |
| install.sh | https://github.com/LLAI-lab/DeepSpeed/blob/master/install.sh |
| setup.cfg | https://github.com/LLAI-lab/DeepSpeed/blob/master/setup.cfg |
| setup.cfg | https://github.com/LLAI-lab/DeepSpeed/blob/master/setup.cfg |
| setup.py | https://github.com/LLAI-lab/DeepSpeed/blob/master/setup.py |
| setup.py | https://github.com/LLAI-lab/DeepSpeed/blob/master/setup.py |
| version.txt | https://github.com/LLAI-lab/DeepSpeed/blob/master/version.txt |
| version.txt | https://github.com/LLAI-lab/DeepSpeed/blob/master/version.txt |
| README | https://github.com/LLAI-lab/DeepSpeed |
| Code of conduct | https://github.com/LLAI-lab/DeepSpeed |
| Contributing | https://github.com/LLAI-lab/DeepSpeed |
| License | https://github.com/LLAI-lab/DeepSpeed |
| Security | https://github.com/LLAI-lab/DeepSpeed |
| https://github.com/deepspeedai/DeepSpeed/blob/master/LICENSE |
| https://pypi.org/project/deepspeed/ |
| https://pepy.tech/project/deepspeed |
| https://github.com/LLAI-lab/DeepSpeed#build-pipeline-status |
| https://www.bestpractices.dev/projects/9530 |
| https://twitter.com/intent/follow?screen_name=DeepSpeedAI |
| https://twitter.com/DeepSpeedAI_JP |
| https://www.zhihu.com/people/deepspeed |
| https://join.slack.com/t/deepspeedworkspace/shared_invite/zt-3a8pjd8dd-PCj2hMvR4Y2syPwVnjEoww |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_light.svg#gh-light-mode-only |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/DeepSpeed_dark_transparent.svg#gh-dark-mode-only |
| https://github.com/LLAI-lab/DeepSpeed#latest-news |
| "Building Efficient Large-Scale Model Systems with DeepSpeed: From Open-Source Foundations to Emerging Research" | https://supercomputing-system-ai-lab.github.io/events/asplos2026-llm-tutorial/index.html |
| Our SuperOffload work received an Honorable Mention for the ASPLOS 2026 Best Paper Award | https://dl.acm.org/doi/10.1145/3760250.3762217 |
| DeepSpeed Core API updates: PyTorch-style backward and low-precision master states | https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/core_api_update/README.md |
| DeepSpeed ZeRO++ powers large-scale distillation training of LLMs for Recommendation Systems at LinkedIn | https://aclanthology.org/2025.emnlp-industry.119/ |
| Ray x DeepSpeed Meetup | https://luma.com/3wctqteh |
| here | https://docs.google.com/presentation/d/1eM3mY6oW9GYkRy1Xz0iOnbbEr5T1t0JJXOM5BKtR-Ks/edit?slide=id.g38615d6b4c2_0_87#slide=id.g38615d6b4c2_0_87 |
| SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips | https://pytorch.org/blog/superoffload-unleashing-the-power-of-large-scale-llm-training-on-superchips/ |
| Study of ZenFlow and ZeRO offload performance with DeepSpeed CPU core binding | https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/zenflow-corebinding/README.md |
| ZenFlow: Stall-Free Offloading Engine for LLM Training | https://pytorch.org/blog/zenflow-stall-free-offloading-engine-for-llm-training/ |
| Arctic Long Sequence Training (ALST) with DeepSpeed: Scalable And Efficient Training For Multi-Million Token Sequences | https://www.snowflake.com/en/engineering-blog/arctic-long-sequence-training-multi-million-token-ai/ |
| DeepNVMe: Affordable I/O scaling for Deep Learning Applications | https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepnvme/06-2025/README.md |
| DeepCompile: Unlocking Compiler Optimization for Distributed Training | https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepcompile/README.md |
| DeepSpeed AutoTP: Automatic Tensor Parallel Training of Hugging Face models | https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/huggingface-tp/README.md |
| Ulysses-Offload: Democratizing Long Context LLM Training | https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/ulysses-offload/README.md |
| https://github.com/LLAI-lab/DeepSpeed#extreme-speed-and-scale-for-dl-training |
| DeepSpeed | https://www.deepspeed.ai/ |
| MT-530B | https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/ |
| BLOOM | https://huggingface.co/blog/bloom-megatron-deepspeed |
| system innovations | https://www.deepspeed.ai/training/ |
| https://github.com/LLAI-lab/DeepSpeed#deepspeed-adoption |
| AI at Scale | https://www.microsoft.com/en-us/research/project/ai-at-scale/ |
| here | https://innovation.microsoft.com/en-us/exploring-ai-at-scale |
| Megatron-Turing NLG (530B) | https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/ |
| Jurassic-1 (178B) | https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf |
| BLOOM (176B) | https://huggingface.co/blog/bloom-megatron-deepspeed |
| GLM (130B) | https://github.com/THUDM/GLM-130B |
| xTrimoPGLM (100B) | https://www.biorxiv.org/content/10.1101/2023.07.05.547496v2 |
| YaLM (100B) | https://github.com/yandex/YaLM-100B |
| GPT-NeoX (20B) | https://github.com/EleutherAI/gpt-neox |
| AlexaTM (20B) | https://www.amazon.science/blog/20b-parameter-alexa-model-sets-new-marks-in-few-shot-learning |
| Turing NLG (17B) | https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ |
| METRO-LM (5.4B) | https://arxiv.org/pdf/2204.06644.pdf |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/transformers-light.png#gh-light-mode-only |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/transformers-dark.png#gh-dark-mode-only |
| Transformers with DeepSpeed | https://huggingface.co/docs/transformers/deepspeed |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/accelerate-light.png#gh-light-mode-only |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/accelerate-dark.png#gh-dark-mode-only |
| Accelerate with DeepSpeed | https://huggingface.co/docs/accelerate/usage_guides/deepspeed |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/lightning-light.svg#gh-light-mode-only |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/lightning-dark.svg#gh-dark-mode-only |
| Lightning with DeepSpeed | https://lightning.ai/docs/pytorch/stable/advanced/model_parallel.html#deepspeed |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/mosaicml.svg |
| MosaicML with DeepSpeed | https://docs.mosaicml.com/projects/composer/en/latest/trainer/using_the_trainer.html?highlight=deepspeed#deepspeed-integration |
| https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/images/determined.svg |
| Determined with DeepSpeed | https://docs.determined.ai/latest/training/apis-howto/deepspeed/overview.html |
| https://user-images.githubusercontent.com/58739961/187154444-fce76639-ac8d-429b-9354-c6fac64b7ef8.jpg |
| MMEngine with DeepSpeed | https://mmengine.readthedocs.io/en/latest/common_usage/large_model_training.html#deepspeed |
| https://github.com/LLAI-lab/DeepSpeed#build-pipeline-status |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/nv-pre-compile-ops.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/aws-torch-latest.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/amd-mi200.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/cpu-torch-latest.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/hpu-gaudi2.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/xpu-max1100.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/aws-accelerate.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/formatting.yml |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/pages/pages-build-deployment |
| https://deepspeed.readthedocs.io/en/latest/?badge=latest |
| https://github.com/deepspeedai/DeepSpeed/actions/workflows/python.yml |
| https://github.com/Ascend/Ascend-CI/actions/workflows/deepspeed.yaml |
| https://github.com/LLAI-lab/DeepSpeed#installation |
| torch's JIT C++ extension loader that relies on
ninja | https://pytorch.org/docs/stable/cpp_extension.html |
| https://github.com/LLAI-lab/DeepSpeed#requirements |
| PyTorch | https://pytorch.org/ |
| nvcc | https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/#introduction |
| hipcc | https://github.com/ROCm-Developer-Tools/HIPCC |
| https://github.com/LLAI-lab/DeepSpeed#contributed-hw-support |
| https://github.com/LLAI-lab/DeepSpeed#pypi |
| PyPI | https://pypi.org/project/deepspeed/ |
| advanced
installation instructions | https://www.deepspeed.ai/tutorials/advanced-install/ |
| https://github.com/LLAI-lab/DeepSpeed#windows |
| here | https://github.com/deepspeedai/DeepSpeed/tree/master/blogs/windows/08-2024/README.md |
| https://github.com/LLAI-lab/DeepSpeed#further-reading |
| deepspeed.ai | https://www.deepspeed.ai/ |
| Getting Started | https://www.deepspeed.ai/getting-started/ |
| DeepSpeed JSON Configuration | https://www.deepspeed.ai/docs/config-json/ |
| API Documentation | https://deepspeed.readthedocs.io/en/latest/ |
| Tutorials | https://www.deepspeed.ai/tutorials/ |
| Blogs | https://www.deepspeed.ai/posts/ |
| https://github.com/LLAI-lab/DeepSpeed#ci-funding |
| https://modal.com | https://modal.com |
| https://github.com/LLAI-lab/DeepSpeed#contributing |
| contributing | https://github.com/LLAI-lab/DeepSpeed/blob/master/CONTRIBUTING.md |
|
| https://github.com/deepspeedai/DeepSpeed/graphs/contributors |
| https://github.com/LLAI-lab/DeepSpeed#developer-certificate-of-origin |
| DCO | https://wiki.linuxfoundation.org/dco |
| https://developercertificate.org | https://developercertificate.org |
| https://github.com/LLAI-lab/DeepSpeed#code-of-conduct |
| Microsoft Open Source Code of
Conduct | https://opensource.microsoft.com/codeofconduct/ |
| Code of Conduct FAQ | https://opensource.microsoft.com/codeofconduct/faq/ |
| https://github.com/LLAI-lab/DeepSpeed#publications |
| arXiv:1910.02054 | https://arxiv.org/abs/1910.02054 |
| In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20) | https://dl.acm.org/doi/10.5555/3433701.3433727 |
| In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial) | https://dl.acm.org/doi/10.1145/3394486.3406703 |
| arXiv:2010.13369 | https://arxiv.org/abs/2010.13369 |
| NeurIPS 2020 | https://proceedings.neurips.cc/paper/2020/hash/a1140a3d0df1c81e24ae954d935e8926-Abstract.html |
| arXiv:2101.06840 | https://arxiv.org/abs/2101.06840 |
| USENIX ATC 2021 | https://www.usenix.org/conference/atc21/presentation/ren-jie |
| [paper] | https://arxiv.org/abs/2101.06840 |
| [slides] | https://www.usenix.org/system/files/atc21_slides_ren-jie.pdf |
| [blog] | https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/ |
| arXiv:2102.02888 | https://arxiv.org/abs/2102.02888 |
| ICML 2021 | http://proceedings.mlr.press/v139/tang21a.html |
| arXiv:2104.07857 | https://arxiv.org/abs/2104.07857 |
| SC 2021 | https://dl.acm.org/doi/abs/10.1145/3458817.3476205 |
| [paper] | https://arxiv.org/abs/2104.07857 |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/SC21-ZeRO-Infinity.pdf |
| [blog] | https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/ |
| arXiv:2104.06069 | https://arxiv.org/abs/2104.06069 |
| HiPC 2022 | https://hipc.org/advance-program/ |
| arXiv:2108.06084 | https://arxiv.org/abs/2108.06084 |
| NeurIPS 2022 | https://openreview.net/forum?id=JpZ5du_Kdh |
| arXiv:2202.06009 | https://arxiv.org/abs/2202.06009 |
| arXiv:2201.05596 | https://arxiv.org/abs/2201.05596 |
| ICML 2022 | https://proceedings.mlr.press/v162/rajbhandari22a.html |
| [pdf] | https://arxiv.org/abs/2201.05596 |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/ICML-5mins.pdf |
| [blog] | https://www.microsoft.com/en-us/research/blog/deepspeed-advancing-moe-inference-and-training-to-power-next-generation-ai-scale/ |
| arXiv:2201.11990 | https://arxiv.org/abs/2201.11990 |
| arXiv:2206.01859 | https://arxiv.org/abs/2206.01859 |
| NeurIPS 2022 | https://openreview.net/forum?id=xNeAhc2CNAl |
| arXiv:2206.01861 | https://arxiv.org/abs/2206.01861 |
| NeurIPS 2022 | https://openreview.net/forum?id=f-fVCElZ-G1 |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf |
| [blog] | https://www.microsoft.com/en-us/research/blog/deepspeed-compression-a-composable-library-for-extreme-compression-and-zero-cost-quantization/ |
| arXiv:2207.00032 | https://arxiv.org/abs/2207.00032 |
| SC 2022 | https://dl.acm.org/doi/abs/10.5555/3571885.3571946 |
| [paper] | https://arxiv.org/abs/2207.00032 |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/sc22-ds-inference.pdf |
| [blog] | https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/ |
| arXiv:2211.11586 | https://arxiv.org/abs/2211.11586 |
| arXiv:2212.03597 | https://arxiv.org/abs/2212.03597 |
| ENLSP2023 Workshop at NeurIPS2023 | https://neurips2023-enlsp.github.io/ |
| arXiv:2301.12017 | https://arxiv.org/abs/2301.12017 |
| ICML2023 | https://icml.cc/Conferences/2023 |
| ICLR:2023 | https://openreview.net/forum?id=Pgtn4l6eKjv |
| arXiv:2303.07226 | https://arxiv.org/abs/2303.07226 |
| Finding at EMNLP2023 | https://2023.emnlp.org/ |
| arXiv:2303.08374 | https://arxiv.org/abs/2303.08374 |
| arXiv:2303.06318 | https://arxiv.org/abs/2303.06318 |
| ICS 2023 | https://dl.acm.org/doi/10.1145/3577193.3593704 |
| arXiv:2306.10209 | https://arxiv.org/abs/2306.10209 |
| ML for Sys Workshop at NeurIPS2023 | http://mlforsystems.org/ |
| [blog] | https://www.microsoft.com/en-us/research/blog/deepspeed-zero-a-leap-in-speed-for-llm-and-chat-model-training-with-4x-less-communication/ |
| arXiv:2303.08302 | https://arxiv.org/abs/2303.08302 |
| ENLSP2023 Workshop at NeurIPS2023 | https://neurips2023-enlsp.github.io/ |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf |
| arXiv:2305.09847 | https://arxiv.org/abs/2305.09847 |
| arXiv:2308.01320 | https://arxiv.org/abs/2308.01320 |
| arXiv:2307.09782 | https://arxiv.org/abs/2307.09782 |
| ENLSP2023 Workshop at NeurIPS2023 | https://neurips2023-enlsp.github.io/ |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/zeroquant_series.pdf |
| arXiv:2309.14327 | https://arxiv.org/pdf/2309.14327.pdf |
| arXiv:2310.04610 | https://arxiv.org/abs/2310.04610 |
| [blog] | https://www.microsoft.com/en-us/research/blog/announcing-the-deepspeed4science-initiative-enabling-large-scale-scientific-discovery-through-sophisticated-ai-system-technologies/ |
| arXiv:2310.17723 | https://arxiv.org/abs/2310.17723 |
| arXiv:2312.08583 | https://arxiv.org/abs/2312.08583 |
| arXiv:2401.14112 | https://arxiv.org/abs/2401.14112 |
| System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models | https://dl.acm.org/doi/10.1145/3662158.3662806 |
| arXiv:2406.18820 | https://arxiv.org/abs/2406.18820 |
| arXiv:2506.13996 | https://arxiv.org/abs/2506.13996 |
| arXiv:2505.12242 | https://arxiv.org/abs/2505.12242 |
| arxiv | https://arxiv.org/abs/2509.21271 |
| ASPLOS 2026 | https://www.asplos-conference.org/asplos2026 |
| https://github.com/LLAI-lab/DeepSpeed#videos |
| Overview | https://www.youtube.com/watch?v=CaseqC45DNc&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=29 |
| ZeRO + large model training | https://www.youtube.com/watch?v=y4_bCiAsIAk&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=28 |
| 17B T-NLG demo | https://www.youtube.com/watch?v=9V-ZbP92drg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=27 |
| Fastest BERT training + RScan tuning | https://www.youtube.com/watch?v=o1K-ZG9F6u0&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=26 |
| part 1 | https://www.youtube.com/watch?v=_NOk-mBwDYg&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=92 |
| part 2 | https://www.youtube.com/watch?v=sG6_c4VXLww&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=94 |
| part 3 | https://www.youtube.com/watch?v=k9yPkBTayos&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=93 |
| FAQ | https://www.youtube.com/watch?v=nsHu6vEgPew&list=PLa85ZdUjfWS21mgibJ2vCvLziprjpKoW0&index=24 |
| ZeRO & Fastest BERT: Increasing the scale and speed of deep learning training in DeepSpeed | https://note.microsoft.com/MSR-Webinar-DeepSpeed-Registration-On-Demand.html |
| DeepSpeed on AzureML | https://youtu.be/yBVXR8G8Bg8 |
| Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference | https://www.youtube.com/watch?v=cntxC3g22oU |
| [slides] | https://github.com/LLAI-lab/DeepSpeed/blob/master/docs/assets/files/presentation-mlops.pdf |
| DeepSpeed: All the tricks to scale to gigantic models (Mark Saroufim) | https://www.youtube.com/watch?v=pDGI668pNg0 |
| Turing-NLG, DeepSpeed and the ZeRO optimizer (Yannic Kilcher) | https://www.youtube.com/watch?v=tC01FRB0M7w |
| Ultimate Guide To Scaling ML Models (The AI Epiphany) | https://www.youtube.com/watch?v=hc0u4avAkuM |
| www.deepspeed.ai/ | https://www.deepspeed.ai/ |
| Readme | https://github.com/LLAI-lab/DeepSpeed#readme-ov-file |
| License | https://github.com/LLAI-lab/DeepSpeed#License-1-ov-file |
| Code of conduct | https://github.com/LLAI-lab/DeepSpeed#coc-ov-file |
| Contributing | https://github.com/LLAI-lab/DeepSpeed#contributing-ov-file |
| Security policy | https://github.com/LLAI-lab/DeepSpeed#security-ov-file |
| Activity | https://github.com/LLAI-lab/DeepSpeed/activity |
| 0 forks | https://github.com/LLAI-lab/DeepSpeed/forks |
| Report repository | https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FLLAI-lab%2FDeepSpeed&report=LLAI-lab+%28user%29 |
|
| https://github.com |
| Terms | https://docs.github.com/site-policy/github-terms/github-terms-of-service |
| Privacy | https://docs.github.com/site-policy/privacy-policies/github-privacy-statement |
| Security | https://github.com/security |
| Status | https://www.githubstatus.com/ |
| Community | https://github.community/ |
| Docs | https://docs.github.com/ |
| Contact | https://support.github.com?tags=dotcom-footer |