René's URL Explorer Experiment


Title: Base the tokenizer API on source offsets · Issue #153569 · python/cpython · GitHub

Open Graph Title: Base the tokenizer API on source offsets · Issue #153569 · python/cpython

X Title: Base the tokenizer API on source offsets · Issue #153569 · python/cpython

Description: The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings. I think...

Open Graph Description: The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with ...

X Description: The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with ...

Opengraph URL: https://github.com/python/cpython/issues/153569

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Base the tokenizer API on source offsets","articleBody":"The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings.\n\nI think tokenizer positions should be offsets into the decoded source instead:\n\n```c\ntypedef Py_ssize_t TokenizerOffset;\n\ntypedef struct {\n    TokenizerOffset start;\n    TokenizerOffset end;\n} TokenizerSpan;\n```\n\nThe main ideas would be:\n\n- One source object owns the decoded text.\n- The cursor only stores its current offset and line boundaries.\n- Tokens, errors and f-string state use offset spans instead of pointers.\n- pegen and `_tokenize` ask the source for a view or copy of a span.\n- Sequential tokenization keeps the current line in the cursor, while uncommon line lookups can use a small sparse index.\n- Normal parsing can keep one contiguous buffer, while `tokenize(readline)` can eventually use reclaimable chunks.\n\nFor incremental tokenization, new decoded input would be appended only when the cursor needs more data. Existing offsets would remain valid even if the underlying storage moves. Once `tokenize` has returned copied token and line strings, old input could be discarded when no active token, cursor, f-string frame or error still refers to it.\n\nThis is very nice because it separates source storage from tokenizer state, removes pointer rebasing, and means consumers no longer need to access tokenizer internals.\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-153585\n* gh-153587\n* gh-156472\n* gh-156482\n* gh-156484\n* gh-156654\n* gh-157055\n* gh-158124\n* gh-158849\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/pablogsal","@type":"Person","name":"pablogsal"},"datePublished":"2026-07-11T13:57:27Z","url":"https://github.com/python/cpython/issues/153569"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:3a68825f-3ff7-e03e-d987-a100dc3311cd
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB3B0:3A4875:32804AB:434B295:6B01F253
html-safe-noncefbc8744bd84fbf4d975d3ec5214910d57a45ebb943388b1911a8411f7d24a298
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCM0IwOjNBNDg3NTozMjgwNEFCOjQzNEIyOTU6NkIwMUYyNTMiLCJ2aXNpdG9yX2lkIjoiODY2NjUwMzY1MTU0NTcwNzA5MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacca3b08d04c0a527d51fb9ad730ba26435449eff0e155b321dd949ff55067f46d
hovercard-subject-tagissue:4862561196
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/153569/issue_layout
twitter:imagehttps://opengraph.githubassets.com/73faa20b1bef0b1e8c975ae4e090fb990f0dfd15728a2b0dc83b3b9454a642ef/python/cpython/issues/153569
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/73faa20b1bef0b1e8c975ae4e090fb990f0dfd15728a2b0dc83b3b9454a642ef/python/cpython/issues/153569
og:image:altThe tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with ...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamepablogsal
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/153569#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F153569
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F153569
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/153569
Reloadhttps://github.com/python/cpython/issues/153569
Reloadhttps://github.com/python/cpython/issues/153569
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/153569
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/153569
Base the tokenizer API on source offsetshttps://github.com/python/cpython/issues/153569#top
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
on Jul 11, 2026https://github.com/python/cpython/issues/153569#issue-4862561196
gh-153569: Split tokenizer lexer into focused files #153585https://github.com/python/cpython/pull/153585
gh-153569: Add tokenizer source and cursor primitives #153587https://github.com/python/cpython/pull/153587
gh-153569: Replace tokenizer input readers #156472https://github.com/python/cpython/pull/156472
gh-153569: return token spans and unify decoded source storage #156482https://github.com/python/cpython/pull/156482
gh-153569: centralize formatted-string state and source spans #156484https://github.com/python/cpython/pull/156484
gh-153569: hide tokenizer state behind token and source operations #156654https://github.com/python/cpython/pull/156654
gh-153569: consolidate tokenizer input, layout, and diagnostics #157055https://github.com/python/cpython/pull/157055
gh-153569: Simplify tokenizer state #158124https://github.com/python/cpython/pull/158124
gh-153569: Simplify tokenizer source views and reader ownership #158849https://github.com/python/cpython/pull/158849
Nexthttps://github.com/python/cpython/issues/153569?timeline_page=1
https://github.com/Arbaaz123676
Arbaaz123676https://github.com/Arbaaz123676
on Jul 11, 2026https://github.com/python/cpython/issues/153569#issuecomment-4946567415
@pablogsalhttps://github.com/pablogsal
https://github.com/pablogsal
pablogsalhttps://github.com/pablogsal
on Jul 11, 2026https://github.com/python/cpython/issues/153569#issuecomment-4946668597
@pablogsalhttps://github.com/pablogsal
@Arbaaz123676https://github.com/Arbaaz123676
pablogsalhttps://github.com/pablogsal
on Jul 11, 2026https://github.com/python/cpython/issues/153569#event-27859157951
gh-153569: Split tokenizer lexer into focused files (#153585)https://github.com/python/cpython/commit/a2d3787105d1f1a5efd0bcf5ccc6b17f5a17d8e8
gh-153569https://github.com/python/cpython/issues/153569
#153585https://github.com/python/cpython/pull/153585
a2d3787https://github.com/python/cpython/commit/a2d3787105d1f1a5efd0bcf5ccc6b17f5a17d8e8
pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/2682fe5435099b36550a614a76e45ddcceee6e30
pythongh-153569https://github.com/python/cpython/issues/153569
2682fe5https://github.com/pablogsal/cpython/commit/2682fe5435099b36550a614a76e45ddcceee6e30
pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/1bf43aaa011ce26b4e5615a8016b74a2f83a57a6
pythongh-153569https://github.com/python/cpython/issues/153569
1bf43aahttps://github.com/pablogsal/cpython/commit/1bf43aaa011ce26b4e5615a8016b74a2f83a57a6
picnixzhttps://github.com/picnixz
type-featureA feature request or enhancementhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-feature%22
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
on Jul 12, 2026https://github.com/python/cpython/issues/153569#event-27875438869
pablogsalhttps://github.com/pablogsal
on Jul 18, 2026https://github.com/python/cpython/issues/153569#event-28158666849
fixup! pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/edf6d0afa64b5265f9ef941f8c40135a2ad25954
pythongh-153569https://github.com/python/cpython/issues/153569
edf6d0ahttps://github.com/pablogsal/cpython/commit/edf6d0afa64b5265f9ef941f8c40135a2ad25954
fixup! pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/b19b995ca460e8d8799b7688451a365330062c99
pythongh-153569https://github.com/python/cpython/issues/153569
b19b995https://github.com/pablogsal/cpython/commit/b19b995ca460e8d8799b7688451a365330062c99
fixup! pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/0dea29d641fbd74d98ae67cab23e5efc7011d5ad
pythongh-153569https://github.com/python/cpython/issues/153569
0dea29dhttps://github.com/pablogsal/cpython/commit/0dea29d641fbd74d98ae67cab23e5efc7011d5ad
pablogsalhttps://github.com/pablogsal
on Aug 25, 2026https://github.com/python/cpython/issues/153569#event-30012872415
pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/697e7dd9882062ea0778eea04a1cc5cd8588f02b
pythongh-153569https://github.com/python/cpython/issues/153569
697e7ddhttps://github.com/pablogsal/cpython/commit/697e7dd9882062ea0778eea04a1cc5cd8588f02b
fixup! pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/100072cdf64821afd159058d88e7ddce60268518
pythongh-153569https://github.com/python/cpython/issues/153569
100072chttps://github.com/pablogsal/cpython/commit/100072cdf64821afd159058d88e7ddce60268518
fixup! pythongh-153569: Add tokenizer source and cursor primitiveshttps://github.com/pablogsal/cpython/commit/4b0454f54481c76371da6057600a95714f17fe1f
pythongh-153569https://github.com/python/cpython/issues/153569
4b0454fhttps://github.com/pablogsal/cpython/commit/4b0454f54481c76371da6057600a95714f17fe1f
gh-153569: Add tokenizer source and cursor primitives (#153587)https://github.com/python/cpython/commit/f54fd2ab6e1f3e3e81a88be3d94ff9d5867b3b3a
gh-153569https://github.com/python/cpython/issues/153569
#153587https://github.com/python/cpython/pull/153587
f54fd2ahttps://github.com/python/cpython/commit/f54fd2ab6e1f3e3e81a88be3d94ff9d5867b3b3a
pythongh-153569: preserve unterminated interactive inputhttps://github.com/pablogsal/cpython/commit/adf62b2607fc34a0cb8eb9edd8d9638c98639ea3
pythongh-153569https://github.com/python/cpython/issues/153569
adf62b2https://github.com/pablogsal/cpython/commit/adf62b2607fc34a0cb8eb9edd8d9638c98639ea3
pablogsalhttps://github.com/pablogsal
on Oct 5, 2026https://github.com/python/cpython/issues/153569#event-32516710027
pythongh-153569: Use logical spans in tokenizer views and column calc…https://github.com/pablogsal/cpython/commit/3f78561b80874943362e5a03c8377516fc12d0f1
pythongh-153569https://github.com/python/cpython/issues/153569
3f78561https://github.com/pablogsal/cpython/commit/3f78561b80874943362e5a03c8377516fc12d0f1
pythongh-153569: Move tokenizer input state and encoding ownership in…https://github.com/pablogsal/cpython/commit/723a6786a83e4390b81c97ded4f77832d49f9805
pythongh-153569https://github.com/python/cpython/issues/153569
723a678https://github.com/pablogsal/cpython/commit/723a6786a83e4390b81c97ded4f77832d49f9805
pythongh-153569: Keep tokenizer source scans and copies on offsets an…https://github.com/pablogsal/cpython/commit/f36c44b4bc7a88851aa25ed628ec7f6d8a0add16
pythongh-153569https://github.com/python/cpython/issues/153569
f36c44bhttps://github.com/pablogsal/cpython/commit/f36c44b4bc7a88851aa25ed628ec7f6d8a0add16
pythongh-153569: Move tokenizer source access into the source APIhttps://github.com/pablogsal/cpython/commit/8f7e822f6b2e1bb9ecff82253776988d63f5e6c4
pythongh-153569https://github.com/python/cpython/issues/153569
8f7e822https://github.com/pablogsal/cpython/commit/8f7e822f6b2e1bb9ecff82253776988d63f5e6c4
pythongh-153569: Use logical spans in tokenizer views and column calc…https://github.com/pablogsal/cpython/commit/2b46f42c865d1e812022672d25310307b72e54e3
pythongh-153569https://github.com/python/cpython/issues/153569
2b46f42https://github.com/pablogsal/cpython/commit/2b46f42c865d1e812022672d25310307b72e54e3
pythongh-153569: Move tokenizer input state and encoding ownership in…https://github.com/pablogsal/cpython/commit/1cfbc932fbf4364d364e5300f0d194ed767188bc
pythongh-153569https://github.com/python/cpython/issues/153569
1cfbc93https://github.com/pablogsal/cpython/commit/1cfbc932fbf4364d364e5300f0d194ed767188bc
pythongh-153569: Keep tokenizer source scans and copies on offsets an…https://github.com/pablogsal/cpython/commit/e60d602e6beb7fcd379b1da6548189beab487897
pythongh-153569https://github.com/python/cpython/issues/153569
e60d602https://github.com/pablogsal/cpython/commit/e60d602e6beb7fcd379b1da6548189beab487897
pythongh-153569: Remove redundant tokenizer views and simplify reader…https://github.com/pablogsal/cpython/commit/7da4c7f46072e28e637481ca6bdeae042de4ad2e
pythongh-153569https://github.com/python/cpython/issues/153569
7da4c7fhttps://github.com/pablogsal/cpython/commit/7da4c7f46072e28e637481ca6bdeae042de4ad2e
pythongh-153569: Remove redundant tokenizer views and simplify reader…https://github.com/pablogsal/cpython/commit/ffb781d20a91aea10b7542fddff0475ddf0c8eef
pythongh-153569https://github.com/python/cpython/issues/153569
ffb781dhttps://github.com/pablogsal/cpython/commit/ffb781d20a91aea10b7542fddff0475ddf0c8eef
pythongh-153569: Reuse the incremental decoder factory helperhttps://github.com/pablogsal/cpython/commit/33d219cabc8ad50f1a78d396b4f69d924caba304
pythongh-153569https://github.com/python/cpython/issues/153569
33d219chttps://github.com/pablogsal/cpython/commit/33d219cabc8ad50f1a78d396b4f69d924caba304
pablogsalhttps://github.com/pablogsal
on Oct 13, 2026https://github.com/python/cpython/issues/153569#event-33132235242
pythongh-153569: Export incremental decoder helper for parser generat…https://github.com/pablogsal/cpython/commit/d386e50c1adf53ff0a5dec11cfd9448ce23f5cdb
pythongh-153569https://github.com/python/cpython/issues/153569
d386e50https://github.com/pablogsal/cpython/commit/d386e50c1adf53ff0a5dec11cfd9448ce23f5cdb
gh-153569: Simplify tokenizer source views and reader ownership (#158849https://github.com/python/cpython/commit/f892b11f22f513c7297e403535fa8511fd0bb0fe
gh-153569https://github.com/python/cpython/issues/153569
#158849https://github.com/python/cpython/pull/158849
f892b11https://github.com/python/cpython/commit/f892b11f22f513c7297e403535fa8511fd0bb0fe
kevinprimeraquecreo-stackhttps://github.com/kevinprimeraquecreo-stack
Project_VKhttps://github.com/users/kevinprimeraquecreo-stack/projects/1
on Oct 18, 2026https://github.com/python/cpython/issues/153569#event-33592117748
github-project-automationhttps://github.com/apps/github-project-automation
Project_VKhttps://github.com/users/kevinprimeraquecreo-stack/projects/1
on Oct 18, 2026https://github.com/python/cpython/issues/153569
kevinprimeraquecreo-stackhttps://github.com/kevinprimeraquecreo-stack
Project_VKhttps://github.com/users/kevinprimeraquecreo-stack/projects/1
on Oct 18, 2026https://github.com/python/cpython/issues/153569#event-33592173542
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/153569
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/153569
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.