René's URL Explorer Experiment


Title: DeepSpeedExamples/Intro101 at master · codedecde/DeepSpeedExamples · GitHub

Open Graph Title: DeepSpeedExamples/Intro101 at master · codedecde/DeepSpeedExamples

X Title: DeepSpeedExamples/Intro101 at master · codedecde/DeepSpeedExamples

Description: Example models using DeepSpeed. Contribute to codedecde/DeepSpeedExamples development by creating an account on GitHub.

Open Graph Description: Example models using DeepSpeed. Contribute to codedecde/DeepSpeedExamples development by creating an account on GitHub.

X Description: Example models using DeepSpeed. Contribute to codedecde/DeepSpeedExamples development by creating an account on GitHub.

Opengraph URL: https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/tree/*name(/*path)
route-controllerfiles
route-actiondisambiguate
fetch-noncev2:991f0725-1f2d-6287-5147-3781f259e3ac
current-catalog-service-hashf3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-idBAC6:3DF338:1231558:18BFD29:6A4D6E4F
html-safe-noncec96d664e5b509c1393ac031e604ea96ac092c466e1c66e65e806a90d2f3b06bf
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCQUM2OjNERjMzODoxMjMxNTU4OjE4QkZEMjk6NkE0RDZFNEYiLCJ2aXNpdG9yX2lkIjoiNDM4ODU2ODAxODI4MTM5Mzc0MyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacaac9133876ff101f72e195ae030086239b7af15320fd8625007a2ccccc032a89
hovercard-subject-tagrepository:418398950
github-keyboard-shortcutsrepository,source-code,file-tree,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///files/disambiguate
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101
twitter:imagehttps://opengraph.githubassets.com/7508611c4064edd815de583969405a0607a53feba9af12b2b349a57a50427f0b/codedecde/DeepSpeedExamples
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/7508611c4064edd815de583969405a0607a53feba9af12b2b349a57a50427f0b/codedecde/DeepSpeedExamples
og:image:altExample models using DeepSpeed. Contribute to codedecde/DeepSpeedExamples development by creating an account on GitHub.
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Nonec78260c7b3f4ac4b0356c8208537f05406a3a2d46a9ec3f735b372dcc519c39e
turbo-cache-controlno-cache
go-importgithub.com/codedecde/DeepSpeedExamples git https://github.com/codedecde/DeepSpeedExamples.git
octolytics-dimension-user_id10473264
octolytics-dimension-user_logincodedecde
octolytics-dimension-repository_id418398950
octolytics-dimension-repository_nwocodedecde/DeepSpeedExamples
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forktrue
octolytics-dimension-repository_parent_id237085793
octolytics-dimension-repository_parent_nwodeepspeedai/DeepSpeedExamples
octolytics-dimension-repository_network_root_id237085793
octolytics-dimension-repository_network_root_nwodeepspeedai/DeepSpeedExamples
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release399fff412f48461d435721c009efc17ea44be72f
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fcodedecde%2FDeepSpeedExamples%2Ftree%2Fmaster%2FIntro101
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub Copilot appDirect agents from issue to mergehttps://github.com/features/ai/github-app
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
View all resourceshttps://github.com/resources
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
GitHub Starshttps://stars.github.com
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fcodedecde%2FDeepSpeedExamples%2Ftree%2Fmaster%2FIntro101
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Ffiles%2Fdisambiguate&source=header-repo&source_repo=codedecde%2FDeepSpeedExamples
Reloadhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101
Reloadhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101
Reloadhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101
codedecde https://github.com/codedecde
DeepSpeedExampleshttps://github.com/codedecde/DeepSpeedExamples
deepspeedai/DeepSpeedExampleshttps://github.com/deepspeedai/DeepSpeedExamples
Notifications https://github.com/login?return_to=%2Fcodedecde%2FDeepSpeedExamples
Fork 0 https://github.com/login?return_to=%2Fcodedecde%2FDeepSpeedExamples
Star 0 https://github.com/login?return_to=%2Fcodedecde%2FDeepSpeedExamples
Code https://github.com/codedecde/DeepSpeedExamples
Pull requests 1 https://github.com/codedecde/DeepSpeedExamples/pulls
Actions https://github.com/codedecde/DeepSpeedExamples/actions
Projects https://github.com/codedecde/DeepSpeedExamples/projects
Security and quality 0 https://github.com/codedecde/DeepSpeedExamples/security
Insights https://github.com/codedecde/DeepSpeedExamples/pulse
Code https://github.com/codedecde/DeepSpeedExamples
Pull requests https://github.com/codedecde/DeepSpeedExamples/pulls
Actions https://github.com/codedecde/DeepSpeedExamples/actions
Projects https://github.com/codedecde/DeepSpeedExamples/projects
Security and quality https://github.com/codedecde/DeepSpeedExamples/security
Insights https://github.com/codedecde/DeepSpeedExamples/pulse
https://github.com/codedecde/DeepSpeedExamples
DeepSpeedExampleshttps://github.com/codedecde/DeepSpeedExamples/tree/master
Historyhttps://github.com/codedecde/DeepSpeedExamples/commits/master/Intro101
https://github.com/codedecde/DeepSpeedExamples/commits/master/Intro101
DeepSpeedExampleshttps://github.com/codedecde/DeepSpeedExamples/tree/master
..https://github.com/codedecde/DeepSpeedExamples/tree/master
testshttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101/tests
testshttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101/tests
README.mdhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/README.md
README.mdhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/README.md
requirements.txthttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/requirements.txt
requirements.txthttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/requirements.txt
train_bert.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert.py
train_bert.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert.py
train_bert_ds.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert_ds.py
train_bert_ds.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert_ds.py
README.mdhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#readme
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#training-a-masked-language-model-with-pytorch-and-deepspeed
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#table-of-contents
(1) Training a Transformer Encoder (BERT / Roberta) model for MLMhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#1-training-a-transformer-encoder-bert--roberta-model-for-mlm
1.0 Some Good Practiceshttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#10-some-good-practices
1.1 The Masked Language Modeling Taskhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#11-the-masked-language-modeling-task
1.2 Creating a Transformer modelhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#12-creating-a-transformer-model
1.3 Training the Modelhttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#13-training-the-model
(2) Integrating DeepSpeed For More Efficient Traininghttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#2-integrating-deepspeed-for-more-efficient-training
2.0 Core DeepSpeed Code Changeshttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#20-core-deepspeed-code-changes
2.1 Launching Traininghttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#21-launching-training
2.2 Mixed Precision Training (fp16)https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#22-mixed-precision-training-fp16
2.3 Zero Redundancy Optimizer (ZeRO)https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#23-zero-redundancy-optimizer-zero
Referenceshttps://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#references
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#1-training-a-transformer-encoder-bert--roberta-model-for-mlm
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#10-some-good-practices
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#version-control-and-reproducibility
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#writing-unit-tests
herehttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/tests/test_train_bert.py
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#11-the-masked-language-modeling-task
wikitext-2-v1https://huggingface.co/datasets/wikitext
HuggingFace datasetshttps://github.com/huggingface/datasets
train_bert.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert.py
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#12-creating-a-transformer-model
train_bert.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert.py
3https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#3
1https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#1
2https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#2
thishttps://nlp.seas.harvard.edu/2018/04/03/attention.html
thishttps://jalammar.github.io/illustrated-bert/
thishttps://jalammar.github.io/illustrated-transformer/
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#13-training-the-model
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#2-integrating-deepspeed-for-more-efficient-training
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#20-core-deepspeed-code-changes
Writing DeepSpeed Modelshttps://www.deepspeed.ai/getting-started/#writing-deepspeed-models
DeepSpeed API documentationhttps://deepspeed.readthedocs.io/en/latest/
config JSON documentationhttps://www.deepspeed.ai/docs/config-json/
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#add-deepspeedinitialize--config
deepspeed.initialize API documentationhttps://deepspeed.readthedocs.io/en/latest/initialize.html#training-initialization
train_bert.pyhttps://github.com/codedecde/DeepSpeedExamples/blob/master/Intro101/train_bert.py
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#update-the-training-loop
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#update-checkpoint-save-and-load
model checkpoint API documentationhttps://deepspeed.readthedocs.io/en/latest/model-checkpointing.html
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#21-launching-training
more details about the DeepSpeed launcherhttps://www.deepspeed.ai/getting-started/#launching-deepspeed-training
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#22-mixed-precision-training-fp16
[3]https://arxiv.org/pdf/1710.03740v3.pdf
fp16 training optionshttps://www.deepspeed.ai/docs/config-json/#fp16-training-options
[3]https://arxiv.org/pdf/1710.03740v3.pdf
Tips for Optimizing GPU Performance Using Tensor Coreshttps://developer.nvidia.com/blog/optimizing-gpu-performance-tensor-cores/
[3]https://arxiv.org/pdf/1710.03740v3.pdf
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#23-zero-redundancy-optimizer-zero
4https://arxiv.org/pdf/1910.02054.pdf
5https://www.usenix.org/system/files/atc21-ren-jie.pdf
6https://arxiv.org/abs/2104.07857
tutorial deep diving ZeROhttps://www.deepspeed.ai/tutorials/zero/
tutorial deep diving ZeRO Offloadhttps://www.deepspeed.ai/tutorials/zero-offload/
ZeRO optimizationshttps://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training
https://github.com/codedecde/DeepSpeedExamples/tree/master/Intro101#references
Vaswani et al. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17)https://arxiv.org/pdf/1706.03762.pdf
J. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT'19)https://aclanthology.org/N19-1423.pdf
P. Micikevicius et al. Mixed Precision Training (ICLR'18)https://arxiv.org/pdf/1710.03740v3.pdf
S. Rajbhandari, J. Rasley, O. Ruwase, Y. He. ZeRO: memory optimizations toward training trillion parameter models. (SC‘20)https://arxiv.org/pdf/1910.02054.pdf
J. Ren, S. Rajbhandari, R. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, Y. He. ZeRO-Offload: Democratizing Billion-Scale Model Training. (ATC'21)https://www.usenix.org/system/files/atc21-ren-jie.pdf
S. Rajbhandari, O. Ruwase, J. Rasley, S. Smith, Y. He. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (SC'21)https://arxiv.org/abs/2104.07857
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.