René's URL Explorer Experiment


Title: bpo-40334: Refactor peg_generator to receive a Tokens file when building c code by pablogsal · Pull Request #19745 · python/cpython · GitHub

Open Graph Title: bpo-40334: Refactor peg_generator to receive a Tokens file when building c code by pablogsal · Pull Request #19745 · python/cpython

X Title: bpo-40334: Refactor peg_generator to receive a Tokens file when building c code by pablogsal · Pull Request #19745 · python/cpython

Description: https://bugs.python.org/issue40334 This PR does the following: Fix a bunch of (very minor) mypy stuff that was missing. Separate the C parser and the Python parser in pegen main (because both receive different arguments). Thread down all these changes to the generator build module. Add a new option to the C parser command line to receive the Tokens file. Thread down the Tokens file and add code to parse it and calculate the required token information. Use the new tokens in the c_generator (and simplify some code that was hardcoding some token names). Update the build files (Makefile and the Windows one) to use the new option. Run black over the source.

Open Graph Description: https://bugs.python.org/issue40334 This PR does the following: Fix a bunch of (very minor) mypy stuff that was missing. Separate the C parser and the Python parser in pegen main (because both rec...

X Description: https://bugs.python.org/issue40334 This PR does the following: Fix a bunch of (very minor) mypy stuff that was missing. Separate the C parser and the Python parser in pegen main (because both rec...

Opengraph URL: https://github.com/python/cpython/pull/19745

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:9f857d2e-638d-0675-4a0e-3b2ea59f6a47
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id965A:2AA429:BB84F9:103D23B:696A3458
html-safe-nonceeee5a3b87b84c61eea142330fc7060adadaac30b6ca919685e86586a6ac94039
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5NjVBOjJBQTQyOTpCQjg0Rjk6MTAzRDIzQjo2OTZBMzQ1OCIsInZpc2l0b3JfaWQiOiI0NzY4MzUzNjE1MjY2NTkxODMyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacf5164911a5226acad89a1a6339ef3cc9ac98f392951040ed5fde5660635dfe42
hovercard-subject-tagpull_request:409812545
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/19745/files
twitter:imagehttps://avatars.githubusercontent.com/u/11718525?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/11718525?s=400&v=4
og:image:althttps://bugs.python.org/issue40334 This PR does the following: Fix a bunch of (very minor) mypy stuff that was missing. Separate the C parser and the Python parser in pegen main (because both rec...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None321736bfdb3f591415ae895a0459bec204b26a76caf47ba5c980634cfacc4538
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release7a9163cefd1ea4bd06f8eb7c082f43e4e53f626f
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/19745/files#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F19745%2Ffiles
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F19745%2Ffiles
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/19745/files
Reloadhttps://github.com/python/cpython/pull/19745/files
Reloadhttps://github.com/python/cpython/pull/19745/files
python https://github.com/python
cpythonhttps://github.com/python/cpython
Please reload this pagehttps://github.com/python/cpython/pull/19745/files
Notifications https://github.com/login?return_to=%2Fpython%2Fcpython
Fork 33.9k https://github.com/login?return_to=%2Fpython%2Fcpython
Star 71.1k https://github.com/login?return_to=%2Fpython%2Fcpython
Code https://github.com/python/cpython
Issues 5k+ https://github.com/python/cpython/issues
Pull requests 2.1k https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects 31 https://github.com/python/cpython/projects
Security Uh oh! There was an error while loading. Please reload this page. https://github.com/python/cpython/security
Please reload this pagehttps://github.com/python/cpython/pull/19745/files
Insights https://github.com/python/cpython/pulse
Code https://github.com/python/cpython
Issues https://github.com/python/cpython/issues
Pull requests https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects https://github.com/python/cpython/projects
Security https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
pablogsalhttps://github.com/pablogsal
python:masterhttps://github.com/python/cpython/tree/master
pablogsal:bpo-40334-use-tokenshttps://github.com/pablogsal/cpython/tree/bpo-40334-use-tokens
Conversation 19 https://github.com/python/cpython/pull/19745
Commits 4 https://github.com/python/cpython/pull/19745/commits
Checks 0 https://github.com/python/cpython/pull/19745/checks
Files changed https://github.com/python/cpython/pull/19745/files
Please reload this pagehttps://github.com/python/cpython/pull/19745/files
bpo-40334: Refactor peg_generator to receive a Tokens file when building c code https://github.com/python/cpython/pull/19745/files#top
Show all changes 4 commits https://github.com/python/cpython/pull/19745/files
a371b79 bpo-40334: Refactor peg_generator to receive a Tokens file when build… pablogsal Apr 28, 2020 https://github.com/python/cpython/pull/19745/commits/a371b79b3998e2f1d5afd1570c682c90608f1c33
417d503 Apply suggestions from code review pablogsal Apr 28, 2020 https://github.com/python/cpython/pull/19745/commits/417d503f8b25b1e23930081846424fb0ab4054ac
c268ef1 Update Tools/peg_generator/pegen/c_generator.py pablogsal Apr 28, 2020 https://github.com/python/cpython/pull/19745/commits/c268ef13a77768edac77a84cbfc3c370ea0c4183
9b8dfd8 Address feedback pablogsal Apr 28, 2020 https://github.com/python/cpython/pull/19745/commits/9b8dfd86e42f04ccc56c155ffd762ba1ca67a5ff
Clear filters https://github.com/python/cpython/pull/19745/files
Please reload this pagehttps://github.com/python/cpython/pull/19745/files
Please reload this pagehttps://github.com/python/cpython/pull/19745/files
Makefile.pre.in https://github.com/python/cpython/pull/19745/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
regen.vcxproj https://github.com/python/cpython/pull/19745/files#diff-43102923295bf7b461a6e7d7d384865a0c45b0146dcc3d0dc6731f6ad0a8e0e4
Makefile https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
__main__.py https://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
build.py https://github.com/python/cpython/pull/19745/files#diff-476cab90bd9f052e2ca52fc323d61913145493cfabeda68fff8feeb086728a9a
c_generator.py https://github.com/python/cpython/pull/19745/files#diff-72bca3f9d3b81d7845baeb20190030d93e9630cb86ebba88854010f7142828a0
testutil.py https://github.com/python/cpython/pull/19745/files#diff-cee475b07bf16bd27f86cf3e38714cb7b54f11db803435d0a0cc070ed4202f0c
test_parse_directory.py https://github.com/python/cpython/pull/19745/files#diff-fd8394da057a60eefbfffe0e612ca4ba33d7f9f1c279cff38953cf81f402791d
Makefile.pre.inhttps://github.com/python/cpython/pull/19745/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
View file https://github.com/pablogsal/cpython/blob/9b8dfd86e42f04ccc56c155ffd762ba1ca67a5ff/Makefile.pre.in
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/19745/{{ revealButtonHref }}
https://github.com/python/cpython/pull/19745/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
https://github.com/python/cpython/pull/19745/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
PCbuild/regen.vcxprojhttps://github.com/python/cpython/pull/19745/files#diff-43102923295bf7b461a6e7d7d384865a0c45b0146dcc3d0dc6731f6ad0a8e0e4
View file https://github.com/pablogsal/cpython/blob/9b8dfd86e42f04ccc56c155ffd762ba1ca67a5ff/PCbuild/regen.vcxproj
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/19745/{{ revealButtonHref }}
https://github.com/python/cpython/pull/19745/files#diff-43102923295bf7b461a6e7d7d384865a0c45b0146dcc3d0dc6731f6ad0a8e0e4
https://github.com/python/cpython/pull/19745/files#diff-43102923295bf7b461a6e7d7d384865a0c45b0146dcc3d0dc6731f6ad0a8e0e4
Tools/peg_generator/Makefilehttps://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
View file https://github.com/pablogsal/cpython/blob/9b8dfd86e42f04ccc56c155ffd762ba1ca67a5ff/Tools/peg_generator/Makefile
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/19745/{{ revealButtonHref }}
https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
https://github.com/python/cpython/pull/19745/files#diff-2a191e7a9d86182d747a87bff2788935f3cfa5cc4e5e15fd605aeb58e274f902
Tools/peg_generator/pegen/__main__.pyhttps://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
View file https://github.com/pablogsal/cpython/blob/9b8dfd86e42f04ccc56c155ffd762ba1ca67a5ff/Tools/peg_generator/pegen/__main__.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/19745/{{ revealButtonHref }}
https://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
https://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
https://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
https://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
https://github.com/python/cpython/pull/19745/files#diff-a033bd44751bf81c93206c71164b2520cba579c4b4d5c1cc63a23b6451d88b38
Please reload this pagehttps://github.com/python/cpython/pull/19745/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.