René's URL Explorer Experiment


Title: fix: the source part should not participate in loss calculation in SFT stage by xffxff · Pull Request #762 · deepspeedai/DeepSpeedExamples · GitHub

Open Graph Title: fix: the source part should not participate in loss calculation in SFT stage by xffxff · Pull Request #762 · deepspeedai/DeepSpeedExamples

X Title: fix: the source part should not participate in loss calculation in SFT stage by xffxff · Pull Request #762 · deepspeedai/DeepSpeedExamples

Description: Example models using DeepSpeed. Contribute to deepspeedai/DeepSpeedExamples development by creating an account on GitHub.

Open Graph Description: fix #660 In the SFT stage, it's essential that the source part doesn't contribute to the loss calculation, only the completion part should be considered. To address this issue, I've adj...

X Description: fix #660 In the SFT stage, it's essential that the source part doesn't contribute to the loss calculation, only the completion part should be considered. To address this issue, I&am...

Opengraph URL: https://github.com/deepspeedai/DeepSpeedExamples/pull/762

X: @github

direct link

Domain: github.com

route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:8eb44099-1cda-9ddc-c7bd-dcdcc0625476
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idA61E:19F979:17FA091:21A78FF:6A4CBBEE
html-safe-nonce5af266f4d4c5a7562ed85c948742abc2cd48485792a0528853dfe3f445d032b8
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNjFFOjE5Rjk3OToxN0ZBMDkxOjIxQTc4RkY6NkE0Q0JCRUUiLCJ2aXNpdG9yX2lkIjoiMTU3NTA4OTQ0MTIzNTMxOTc5MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacd10d8bd40a55ad8bbb1abbbdd359923aa1caba1b3b47f2c25ed87cdc87672bd7
hovercard-subject-tagpull_request:1549723146
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/deepspeedai/DeepSpeedExamples/762/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/39cb9efa9409091b2757e404dfec24c5cba50f8d33856f847ef4ab43c0e38def/deepspeedai/DeepSpeedExamples/pull/762
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/39cb9efa9409091b2757e404dfec24c5cba50f8d33856f847ef4ab43c0e38def/deepspeedai/DeepSpeedExamples/pull/762
og:image:altfix #660 In the SFT stage, it's essential that the source part doesn't contribute to the loss calculation, only the completion part should be considered. To address this issue, I've adj...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamexffxff
hostnamegithub.com
expected-hostnamegithub.com
None3d11bb817438277de2a940854450e83a7d32b6aeb5014e9e6b00a6423900251c
turbo-cache-controlno-cache
go-importgithub.com/deepspeedai/DeepSpeedExamples git https://github.com/deepspeedai/DeepSpeedExamples.git
octolytics-dimension-user_id74068820
octolytics-dimension-user_logindeepspeedai
octolytics-dimension-repository_id237085793
octolytics-dimension-repository_nwodeepspeedai/DeepSpeedExamples
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id237085793
octolytics-dimension-repository_network_root_nwodeepspeedai/DeepSpeedExamples
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releaseae90d426644ca15e89bacceb72e51f4e9dbf85f7
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/deepspeedai/DeepSpeedExamples/pull/762#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdeepspeedai%2FDeepSpeedExamples%2Fpull%2F762
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub Copilot appDirect agents from issue to mergehttps://github.com/features/ai/github-app
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
View all resourceshttps://github.com/resources
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
GitHub Starshttps://stars.github.com
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdeepspeedai%2FDeepSpeedExamples%2Fpull%2F762
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=deepspeedai%2FDeepSpeedExamples
Reloadhttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
Reloadhttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
Reloadhttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
Please reload this pagehttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
deepspeedai https://github.com/deepspeedai
DeepSpeedExampleshttps://github.com/deepspeedai/DeepSpeedExamples
Notifications https://github.com/login?return_to=%2Fdeepspeedai%2FDeepSpeedExamples
Fork 1.1k https://github.com/login?return_to=%2Fdeepspeedai%2FDeepSpeedExamples
Star 6.8k https://github.com/login?return_to=%2Fdeepspeedai%2FDeepSpeedExamples
Code https://github.com/deepspeedai/DeepSpeedExamples
Issues 300 https://github.com/deepspeedai/DeepSpeedExamples/issues
Pull requests 26 https://github.com/deepspeedai/DeepSpeedExamples/pulls
Actions https://github.com/deepspeedai/DeepSpeedExamples/actions
Projects https://github.com/deepspeedai/DeepSpeedExamples/projects
Security and quality 0 https://github.com/deepspeedai/DeepSpeedExamples/security
Insights https://github.com/deepspeedai/DeepSpeedExamples/pulse
Code https://github.com/deepspeedai/DeepSpeedExamples
Issues https://github.com/deepspeedai/DeepSpeedExamples/issues
Pull requests https://github.com/deepspeedai/DeepSpeedExamples/pulls
Actions https://github.com/deepspeedai/DeepSpeedExamples/actions
Projects https://github.com/deepspeedai/DeepSpeedExamples/projects
Security and quality https://github.com/deepspeedai/DeepSpeedExamples/security
Insights https://github.com/deepspeedai/DeepSpeedExamples/pulse
xffxffhttps://github.com/xffxff
deepspeedai:masterhttps://github.com/deepspeedai/DeepSpeedExamples/tree/master
xffxff:sft_losshttps://github.com/xffxff/DeepSpeedExamples/tree/sft_loss
Conversationhttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
Commits1 (1)https://github.com/deepspeedai/DeepSpeedExamples/pull/762/commits
Checkshttps://github.com/deepspeedai/DeepSpeedExamples/pull/762/checks
Files changedhttps://github.com/deepspeedai/DeepSpeedExamples/pull/762/files
fix: the source part should not participate in loss calculation in SFT stagehttps://github.com/deepspeedai/DeepSpeedExamples/pull/762#top
xffxffhttps://github.com/xffxff
deepspeedai:masterhttps://github.com/deepspeedai/DeepSpeedExamples/tree/master
xffxff:sft_losshttps://github.com/xffxff/DeepSpeedExamples/tree/sft_loss
https://github.com/xffxff
xffxffhttps://github.com/xffxff
Oct 10, 2023https://github.com/deepspeedai/DeepSpeedExamples/pull/762#issue-1935106561
Please reload this pagehttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
#660https://github.com/deepspeedai/DeepSpeedExamples/issues/660
torch.nn.CrossEntropyLosshttps://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss
OPTForCausalLMhttps://github.com/huggingface/transformers/blob/a9862a0f495bac3a6ecd5165686610fd5d91c848/src/transformers/models/opt/modeling_opt.py#L966
LLamaForCausalLMhttps://github.com/huggingface/transformers/blob/a9862a0f495bac3a6ecd5165686610fd5d91c848/src/transformers/models/llama/modeling_llama.py#L1061
https://private-user-images.githubusercontent.com/30254428/273876738-85d064e8-acf2-4118-b734-a9bd6971d181.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3ODM0MTQwNDIsIm5iZiI6MTc4MzQxMzc0MiwicGF0aCI6Ii8zMDI1NDQyOC8yNzM4NzY3MzgtODVkMDY0ZTgtYWNmMi00MTE4LWI3MzQtYTliZDY5NzFkMTgxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjA3MDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwNzA3VDA4NDIyMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZjMzJkOGVmMjJhYThmMmE0NWM1N2NmMzBjNWRkMzJmZDNlZGJkYTkyMDM3OWQ1NDRiNmEyMDAyOTg4NzVhMjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JnJlc3BvbnNlLWNvbnRlbnQtdHlwZT1pbWFnZSUyRnBuZyJ9.FqlBqDV1QUJgY_GfJ2_YkQzkbKDlNwEXkfHL0g7MXds
Please reload this pagehttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
https://github.com/xffxff
fix: the prompt part should not participate in loss calculation in SF…https://github.com/deepspeedai/DeepSpeedExamples/pull/762/commits/8e58c749951163f0dc48f0c78b80c96ddadf1c9d
8e58c74https://github.com/deepspeedai/DeepSpeedExamples/pull/762/commits/8e58c749951163f0dc48f0c78b80c96ddadf1c9d
https://github.com/xffxff
xffxffhttps://github.com/xffxff
RezaYazdaniAminabadihttps://github.com/RezaYazdaniAminabadi
ShadenSmithhttps://github.com/ShadenSmith
arashbhttps://github.com/arashb
awan-10https://github.com/awan-10
conglonglihttps://github.com/conglongli
duli2012https://github.com/duli2012
eltonzhenghttps://github.com/eltonzheng
jeffrahttps://github.com/jeffra
minjiazhttps://github.com/minjiaz
mrwyattiihttps://github.com/mrwyattii
samyamhttps://github.com/samyam
tjruwasehttps://github.com/tjruwase
xiaoxiawu-microsofthttps://github.com/xiaoxiawu-microsoft
yaozheweihttps://github.com/yaozhewei
code ownershttps://github.com/deepspeedai/DeepSpeedExamples/blob/0855679ead5ff4eed550dba3c8bd1737f4a3ec24/CODEOWNERS#L1
October 10, 2023 11:14https://github.com/deepspeedai/DeepSpeedExamples/pull/762#event-10602953502
https://github.com/AndyW-llm
AndyW-llmhttps://github.com/AndyW-llm
Nov 28, 2023https://github.com/deepspeedai/DeepSpeedExamples/pull/762#issuecomment-1830668298
Please reload this pagehttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
https://github.co/hiddenchars
https://github.com/deepspeedai/DeepSpeedExamples/pull/{{ revealButtonHref }}
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdeepspeedai%2FDeepSpeedExamples%2Fpull%2F762
https://github.com/jeffra
jeffra https://github.com/jeffra
https://github.com/samyam
samyam https://github.com/samyam
https://github.com/tjruwase
tjruwase https://github.com/tjruwase
https://github.com/ShadenSmith
ShadenSmith https://github.com/ShadenSmith
https://github.com/conglongli
conglongli https://github.com/conglongli
https://github.com/awan-10
awan-10 https://github.com/awan-10
https://github.com/eltonzheng
eltonzheng https://github.com/eltonzheng
https://github.com/minjiaz
minjiaz https://github.com/minjiaz
https://github.com/RezaYazdaniAminabadi
RezaYazdaniAminabadi https://github.com/RezaYazdaniAminabadi
https://github.com/duli2012
duli2012 https://github.com/duli2012
https://github.com/mrwyattii
mrwyattii https://github.com/mrwyattii
https://github.com/yaozhewei
yaozhewei https://github.com/yaozhewei
https://github.com/arashb
arashb https://github.com/arashb
https://github.com/xiaoxiawu-microsoft
xiaoxiawu-microsoft https://github.com/xiaoxiawu-microsoft
[Bug] Step1: Not mask source part https://github.com/deepspeedai/DeepSpeedExamples/issues/660
Please reload this pagehttps://github.com/deepspeedai/DeepSpeedExamples/pull/762
https://github.com/xffxff
https://github.com/AndyW-llm
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.