René's URL Explorer Experiment


Title: gh-97973: Return all necessary information from the tokenizer by lysnikolaou · Pull Request #97984 · python/cpython · GitHub

Open Graph Title: gh-97973: Return all necessary information from the tokenizer by lysnikolaou · Pull Request #97984 · python/cpython

X Title: gh-97973: Return all necessary information from the tokenizer by lysnikolaou · Pull Request #97984 · python/cpython

Description: Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that th...

Open Graph Description: Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that th...

X Description: Right now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that th...

Opengraph URL: https://github.com/python/cpython/pull/97984

X: @github

direct link

Domain: github.com

route-pattern/_view_fragments/voltron/pull_requests/show/:user_id/:repository/:id/pull_request_layout(.:format)
route-controllervoltron_pull_requests_fragments
route-actionpull_request_layout
fetch-noncev2:513a9812-c37c-ffb3-31f4-e3ea7c65723a
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idA21C:33C049:4E799E:695DA3:6969D537
html-safe-nonce3b22919fd7b76e29cc5020aece73499d7e89bf9d6ca928340c4241c3e87bfbc6
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBMjFDOjMzQzA0OTo0RTc5OUU6Njk1REEzOjY5NjlENTM3IiwidmlzaXRvcl9pZCI6IjMxOTU1NTAyMjc5OTI1MzIyODAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac63fa5b1cf11a25542c9888bb7610bd53a8ae4cb85cb2d8c82d814d3b4bb6539d
hovercard-subject-tagpull_request:1079460276
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/pull_requests_fragments/pull_request_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/voltron/pull_requests/show/python/cpython/97984/pull_request_layout
twitter:imagehttps://opengraph.githubassets.com/f90ec7e226a2056c5e241bd770ebac6d1ea04b907a1dfc9fe2c865aac2087a9f/python/cpython/pull/97984
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/f90ec7e226a2056c5e241bd770ebac6d1ea04b907a1dfc9fe2c865aac2087a9f/python/cpython/pull/97984
og:image:altRight now, the tokenizer only returns type and two pointers to the start and end of the token. This PR modifies the tokenizer to return the type and set all of the necessary information, so that th...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamelysnikolaou
hostnamegithub.com
expected-hostnamegithub.com
Noneacedec8b5f975d9e3d494ddd8f949b0b8a0de59d393901e26f73df9dcba80056
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release83c08c21cdda978090dc44364b71aa5bc6dcea79
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/97984#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F97984
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F97984
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fpull_requests_fragments%2Fpull_request_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/97984
Reloadhttps://github.com/python/cpython/pull/97984
Reloadhttps://github.com/python/cpython/pull/97984
python https://github.com/python
cpythonhttps://github.com/python/cpython
Please reload this pagehttps://github.com/python/cpython/pull/97984
Notifications https://github.com/login?return_to=%2Fpython%2Fcpython
Fork 33.9k https://github.com/login?return_to=%2Fpython%2Fcpython
Star 71.1k https://github.com/login?return_to=%2Fpython%2Fcpython
Code https://github.com/python/cpython
Issues 5k+ https://github.com/python/cpython/issues
Pull requests 2.1k https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects 31 https://github.com/python/cpython/projects
Security Uh oh! There was an error while loading. Please reload this page. https://github.com/python/cpython/security
Please reload this pagehttps://github.com/python/cpython/pull/97984
Insights https://github.com/python/cpython/pulse
Code https://github.com/python/cpython
Issues https://github.com/python/cpython/issues
Pull requests https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects https://github.com/python/cpython/projects
Security https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
Jump to bottomhttps://github.com/python/cpython/pull/97984#issue-comment-box
lysnikolaouhttps://github.com/lysnikolaou
python:mainhttps://github.com/python/cpython/tree/main
lysnikolaou:tokenizer-return-token-2https://github.com/lysnikolaou/cpython/tree/tokenizer-return-token-2
gh-97973: Return all necessary information from the tokenizer https://github.com/python/cpython/pull/97984#top
lysnikolaouhttps://github.com/lysnikolaou
python:mainhttps://github.com/python/cpython/tree/main
lysnikolaou:tokenizer-return-token-2https://github.com/lysnikolaou/cpython/tree/tokenizer-return-token-2
Conversation 1 https://github.com/python/cpython/pull/97984
Commits 7 https://github.com/python/cpython/pull/97984/commits
Checks 0 https://github.com/python/cpython/pull/97984/checks
Files changed https://github.com/python/cpython/pull/97984/files
Please reload this pagehttps://github.com/python/cpython/pull/97984
https://github.co/hiddenchars
https://github.com/python/cpython/pull/{{ revealButtonHref }}
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
Oct 6, 2022https://github.com/python/cpython/pull/97984#issue-1400146951
Please reload this pagehttps://github.com/python/cpython/pull/97984
Tokenizer returnings pointers make calculation of position info more complicated #97973https://github.com/python/cpython/issues/97973
Please reload this pagehttps://github.com/python/cpython/pull/97984
https://github.com/lysnikolaou
pythongh-97973https://github.com/python/cpython/issues/97973
: Return all necessary information from the tokenizerhttps://github.com/python/cpython/pull/97984/commits/0a7d0f639cfc3681a96da74dc2ec4831533471d5
0a7d0f6https://github.com/python/cpython/pull/97984/commits/0a7d0f639cfc3681a96da74dc2ec4831533471d5
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
pablogsalhttps://github.com/pablogsal
code ownerhttps://github.com/python/cpython/blob/23e83a84651bbcf1f3778baf3ab0b4cbfead75e3/.github/CODEOWNERS#L87
October 6, 2022 18:44https://github.com/python/cpython/pull/97984#event-7537799490
https://github.com/bedevere-bot
bedevere-bothttps://github.com/bedevere-bot
awaiting core review https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22awaiting%20core%20review%22
Oct 6, 2022https://github.com/python/cpython/pull/97984#event-7537799714
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
Oct 6, 2022https://github.com/python/cpython/pull/97984#event-7537802775
Tokenizer returnings pointers make calculation of position info more complicated #97973 https://github.com/python/cpython/issues/97973
lysnikolaouhttps://github.com/lysnikolaou
October 6, 2022 12:35https://github.com/python/cpython/pull/97984#commits-pushed-c2320ba
https://github.com/lysnikolaou
Pass pointers so that struct does not get copiedhttps://github.com/python/cpython/pull/97984/commits/c2320ba5672eaaad1463c8138ed94a07380a7aec
c2320bahttps://github.com/python/cpython/pull/97984/commits/c2320ba5672eaaad1463c8138ed94a07380a7aec
https://github.com/lysnikolaou
Merge branch 'main' into tokenizer-return-token-2https://github.com/python/cpython/pull/97984/commits/dee33594de0c00a95e472edbd7abf96bae7f9de7
dee3359https://github.com/python/cpython/pull/97984/commits/dee33594de0c00a95e472edbd7abf96bae7f9de7
https://github.com/lysnikolaou
Remove news blurb for nowhttps://github.com/python/cpython/pull/97984/commits/b28afeda709b8f0eb3edca7c3e1daceac0798e40
b28afedhttps://github.com/python/cpython/pull/97984/commits/b28afeda709b8f0eb3edca7c3e1daceac0798e40
https://github.com/apps/blurb-it
📜🤖 Added by blurb_it.https://github.com/python/cpython/pull/97984/commits/1e52242ba712aeaa695b4f0c112c77d631392443
1e52242https://github.com/python/cpython/pull/97984/commits/1e52242ba712aeaa695b4f0c112c77d631392443
https://github.com/lysnikolaou
Merge branch 'main' into tokenizer-return-token-2https://github.com/python/cpython/pull/97984/commits/0aea48d5bb31e374d1dbc9799a22a82db86d9d46
0aea48dhttps://github.com/python/cpython/pull/97984/commits/0aea48d5bb31e374d1dbc9799a22a82db86d9d46
https://github.com/lysnikolaou
Add assertionhttps://github.com/python/cpython/pull/97984/commits/689c670ae876baba229ee4bdd85ab02872600e8c
689c670https://github.com/python/cpython/pull/97984/commits/689c670ae876baba229ee4bdd85ab02872600e8c
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
Oct 6, 2022 https://github.com/python/cpython/pull/97984#pullrequestreview-1133690255
View reviewed changes https://github.com/python/cpython/pull/97984/files/689c670ae876baba229ee4bdd85ab02872600e8c
pablogsalhttps://github.com/pablogsal
https://github.com/python/cpython/pull/97984#pullrequestreview-1133690255
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/97984
https://github.com/bedevere-bot
bedevere-bothttps://github.com/bedevere-bot
awaiting merge https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22awaiting%20merge%22
awaiting core review https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22awaiting%20core%20review%22
Oct 6, 2022https://github.com/python/cpython/pull/97984#event-7538993524
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
cbf0afdhttps://github.com/python/cpython/commit/cbf0afd8a1474d68310331af9218606959d4cc22
Oct 6, 2022https://github.com/python/cpython/pull/97984#event-7539246683
https://github.com/bedevere-bot
bedevere-bothttps://github.com/bedevere-bot
awaiting merge https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22awaiting%20merge%22
Oct 6, 2022https://github.com/python/cpython/pull/97984#event-7539246852
https://github.com/lysnikolaou
lysnikolaouhttps://github.com/lysnikolaou
Oct 7, 2022 https://github.com/python/cpython/pull/97984#ref-pullrequest-1401750374
Use new tokenizer column offset to fix fstring ast locations we-like-parsers/cpython#195 https://github.com/we-like-parsers/cpython/pull/195
carljmhttps://github.com/carljm
Oct 8, 2022 https://github.com/python/cpython/pull/97984#ref-commit-415ed49
https://github.com/carljm
Merge branch 'main' into typewatchhttps://github.com/carljm/cpython/commit/415ed492de0dc27707962eb1756cf183dadd0885
415ed49https://github.com/carljm/cpython/commit/415ed492de0dc27707962eb1756cf183dadd0885
pythonGH-97002https://github.com/python/cpython/issues/97002
pythonGH-97996https://github.com/python/cpython/pull/97996
pythongh-97973https://github.com/python/cpython/issues/97973
pythonGH-97984https://github.com/python/cpython/pull/97984
pythongh-96078https://github.com/python/cpython/issues/96078
pythongh-97965https://github.com/python/cpython/pull/97965
pythongh-65961https://github.com/python/cpython/issues/65961
pythonGH-97990https://github.com/python/cpython/pull/97990
pythongh-97850https://github.com/python/cpython/issues/97850
python#97935https://github.com/python/cpython/pull/97935
pythonGH-97992https://github.com/python/cpython/pull/97992
pythongh-94590https://github.com/python/cpython/issues/94590
python#94591https://github.com/python/cpython/pull/94591
python#97974https://github.com/python/cpython/pull/97974
pythongh-97781https://github.com/python/cpython/issues/97781
pythonGH-97785https://github.com/python/cpython/pull/97785
pythongh-86482https://github.com/python/cpython/issues/86482
python#23291https://github.com/python/cpython/pull/23291
pythongh-97943https://github.com/python/cpython/issues/97943
python#97949https://github.com/python/cpython/pull/97949
mpagehttps://github.com/mpage
Oct 11, 2022 https://github.com/python/cpython/pull/97984#ref-commit-078e3ed
https://github.com/lysnikolaou
https://github.com/mpage
pythongh-97973https://github.com/python/cpython/issues/97973
: Return all necessary information from the tokenizer (https://github.com/mpage/cpython/commit/078e3eddac3f46469962f8205580cdc6348e631d
p…https://github.com/python/cpython/pull/97984
078e3edhttps://github.com/mpage/cpython/commit/078e3eddac3f46469962f8205580cdc6348e631d
…ythonGH-97984https://github.com/python/cpython/pull/97984
Sign up for freehttps://github.com/join?source=comment-repo
Sign in to commenthttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F97984
https://github.com/pablogsal
pablogsal https://github.com/pablogsal
https://github.com/python/cpython/pull/97984/files/689c670ae876baba229ee4bdd85ab02872600e8c
Tokenizer returnings pointers make calculation of position info more complicated https://github.com/python/cpython/issues/97973
Please reload this pagehttps://github.com/python/cpython/pull/97984
https://github.com/lysnikolaou
https://github.com/pablogsal
https://github.com/bedevere-bot
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.