René's URL Explorer Experiment


Title: gh-153569: return token spans and unify decoded source storage by pablogsal · Pull Request #156482 · python/cpython · GitHub

Open Graph Title: gh-153569: return token spans and unify decoded source storage by pablogsal · Pull Request #156482 · python/cpython

X Title: gh-153569: return token spans and unify decoded source storage by pablogsal · Pull Request #156482 · python/cpython

Description: Return tokenizer tokens as logical source spans with start/end locations, and give all decoded input one owner in SourceText. File and readline input discard consumed windows while advancing a logical base offset. Discard reuses the allocation and resets line metadata; prepared and interactive input retain their source. The separate streaming buffer allocation is removed. This prerequisite still rebases scanner pointers around append; #156654 replaces those pointers with offsets. Debug source growth forces relocation. Tests cover f-/t-string growth in both token modes, interactive multiline input, discard/reuse, implicit-line metadata reset, and offset limits. This follows #156472. The rest of the stack is #156484, then #156654. Issue: gh-153569

Open Graph Description: Return tokenizer tokens as logical source spans with start/end locations, and give all decoded input one owner in SourceText. File and readline input discard consumed windows while advancing a logi...

X Description: Return tokenizer tokens as logical source spans with start/end locations, and give all decoded input one owner in SourceText. File and readline input discard consumed windows while advancing a logi...

Opengraph URL: https://github.com/python/cpython/pull/156482

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:124660a0-db81-a7e4-fb0e-f90800073806
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id87A6:29AEE8:384C1E0:4AEEFB6:6B0214EF
html-safe-nonce159571d6baae54518ca0045f90d5262afc2fba27e7931b2e9fcdf8896b80e14d
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4N0E2OjI5QUVFODozODRDMUUwOjRBRUVGQjY6NkIwMjE0RUYiLCJ2aXNpdG9yX2lkIjoiMjg1MjA1NTQyMTA0MzIxNzY0NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac6a75dbddea7ab82e9e2a0a7a851a8fa657808534b3298a8e3bcc489d01515573
hovercard-subject-tagpull_request:4377727254
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/156482/files
twitter:imagehttps://avatars.githubusercontent.com/u/11718525?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/11718525?s=400&v=4
og:image:altReturn tokenizer tokens as logical source spans with start/end locations, and give all decoded input one owner in SourceText. File and readline input discard consumed windows while advancing a logi...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/156482/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F156482%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F156482%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/156482/files
Reloadhttps://github.com/python/cpython/pull/156482/files
Reloadhttps://github.com/python/cpython/pull/156482/files
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
pablogsalhttps://github.com/pablogsal
python:mainhttps://github.com/python/cpython/tree/main
pablogsal:gh-153569-tokenizer-offset-statehttps://github.com/pablogsal/cpython/tree/gh-153569-tokenizer-offset-state
Conversation 3 https://github.com/python/cpython/pull/156482
Commits 6 https://github.com/python/cpython/pull/156482/commits
Checks 68 https://github.com/python/cpython/pull/156482/checks
Files changed https://github.com/python/cpython/pull/156482/files
Please reload this pagehttps://github.com/python/cpython/pull/156482/files
gh-153569: return token spans and unify decoded source storage https://github.com/python/cpython/pull/156482/files#top
Show all changes 6 commits https://github.com/python/cpython/pull/156482/files
44ef2d1 gh-153569: bound tokenizer input storage pablogsal Aug 30, 2026 https://github.com/python/cpython/pull/156482/commits/44ef2d1caace7e77285079fb9601c6b5bad4ec4d
f381082 gh-153569: return tokenizer tokens as source spans pablogsal Aug 30, 2026 https://github.com/python/cpython/pull/156482/commits/f381082d3b34f3cc2747f5da998651c58079e116
b60cc04 gh-153569: enforce tokenizer buffer view lifetimes pablogsal Sep 5, 2026 https://github.com/python/cpython/pull/156482/commits/b60cc049a7098266e574667eba32aa94a0b5bcf7
2bdef78 gh-153569: Unify decoded tokenizer storage ownership pablogsal Sep 5, 2026 https://github.com/python/cpython/pull/156482/commits/2bdef783ebcec30fa5905b57d68d79b3a6172e28
27c3d73 gh-153569: Avoid overflow when sizing implicit line flags pablogsal Sep 5, 2026 https://github.com/python/cpython/pull/156482/commits/27c3d732c3d375d9d133782038984b2da7647da7
6117ae3 Remove stale tokenizer buffer ownership comment pablogsal Sep 5, 2026 https://github.com/python/cpython/pull/156482/commits/6117ae337c2242a5a7767a34a81809f6ff6fb4c0
Clear filters https://github.com/python/cpython/pull/156482/files
Please reload this pagehttps://github.com/python/cpython/pull/156482/files
Please reload this pagehttps://github.com/python/cpython/pull/156482/files
test_tokenizer.py https://github.com/python/cpython/pull/156482/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
test_repl.py https://github.com/python/cpython/pull/156482/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
test_tokenize.py https://github.com/python/cpython/pull/156482/files#diff-b07c4f10c75a22e102f114227a0a9b7b763ddda7a330109994e69b29540ea9bb
tokenizer.c https://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
buffer.c https://github.com/python/cpython/pull/156482/files#diff-b4dbec3be58c399219891041d458bb2d24e9fd7d283e03ed09027e8484a56c32
buffer.h https://github.com/python/cpython/pull/156482/files#diff-19981875a12620209e5d2bd5f421e7dd5bcd5d2f9d880255b2f3ed65f242f463
lexer.c https://github.com/python/cpython/pull/156482/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
lexer.h https://github.com/python/cpython/pull/156482/files#diff-4c8d59cfb542d6b3e63561140f56a78e991cd0b8019f7de332deba771863735b
state.c https://github.com/python/cpython/pull/156482/files#diff-d320456c1594700fd4ea3c33fbfc5f7fd74d4fd45fd225354ea8db7135c04045
state.h https://github.com/python/cpython/pull/156482/files#diff-9efe28cd84a24747de9e363846c7d9255a3d3a7073896547f570bf91b4fd3667
string.c https://github.com/python/cpython/pull/156482/files#diff-c8bf656bc09ae7acf6bc1c311f93e578b1df1efab51aa5dcfec00aacbb8425f7
pegen.c https://github.com/python/cpython/pull/156482/files#diff-ec9a9d08c5b1d518997db0f5b4795c528a41efc35507d6c2a384eee43f047cb7
cursor.c https://github.com/python/cpython/pull/156482/files#diff-612535281447d780b732c38843952f91678c5d75d8396e55445dc375319d3813
cursor.h https://github.com/python/cpython/pull/156482/files#diff-f63563ab928ae9883992ec0dddbd66eab2ffd2abf172a22c91d754f7ac649f8c
reader.c https://github.com/python/cpython/pull/156482/files#diff-ee1ccf9c461ad2982f27a89c79c7ea2e5868fef86dbd8d8d1d873e1624208590
source.c https://github.com/python/cpython/pull/156482/files#diff-d091331acd32f48897fc1503341091ee6066cbe2208017651deef042b32273da
source.h https://github.com/python/cpython/pull/156482/files#diff-033fac2d1c68ebec297cd0aaf7ee48c836b7ebab03362fcecc53cedcdac043d1
Python-tokenize.c https://github.com/python/cpython/pull/156482/files#diff-68d96c581022a408dbcf0f48df1126c3a43154ed2dd20b166aa5269ab27f3dcb
Lib/test/test_capi/test_tokenizer.pyhttps://github.com/python/cpython/pull/156482/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Lib/test/test_capi/test_tokenizer.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
https://github.com/python/cpython/pull/156482/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
Lib/test/test_repl.pyhttps://github.com/python/cpython/pull/156482/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Lib/test/test_repl.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
https://github.com/python/cpython/pull/156482/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L311
Lib/test/test_tokenize.pyhttps://github.com/python/cpython/pull/156482/files#diff-b07c4f10c75a22e102f114227a0a9b7b763ddda7a330109994e69b29540ea9bb
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Lib/test/test_tokenize.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-b07c4f10c75a22e102f114227a0a9b7b763ddda7a330109994e69b29540ea9bb
https://github.com/python/cpython/pull/156482/files#diff-b07c4f10c75a22e102f114227a0a9b7b763ddda7a330109994e69b29540ea9bb
Modules/_testinternalcapi/tokenizer.chttps://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Modules/_testinternalcapi/tokenizer.c
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
https://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
https://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
https://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
https://github.com/python/cpython/pull/156482/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L313
Parser/lexer/buffer.chttps://github.com/python/cpython/pull/156482/files#diff-b4dbec3be58c399219891041d458bb2d24e9fd7d283e03ed09027e8484a56c32
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Parser/lexer/buffer.c
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L313
Parser/lexer/buffer.hhttps://github.com/python/cpython/pull/156482/files#diff-19981875a12620209e5d2bd5f421e7dd5bcd5d2f9d880255b2f3ed65f242f463
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Parser/lexer/buffer.h
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-19981875a12620209e5d2bd5f421e7dd5bcd5d2f9d880255b2f3ed65f242f463
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L313
Parser/lexer/lexer.chttps://github.com/python/cpython/pull/156482/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Parser/lexer/lexer.c
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
https://github.com/python/cpython/pull/156482/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
https://github.com/python/cpython/pull/156482/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
https://github.com/python/cpython/pull/156482/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L313
Parser/lexer/lexer.hhttps://github.com/python/cpython/pull/156482/files#diff-4c8d59cfb542d6b3e63561140f56a78e991cd0b8019f7de332deba771863735b
View file https://github.com/pablogsal/cpython/blob/6117ae337c2242a5a7767a34a81809f6ff6fb4c0/Parser/lexer/lexer.h
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/156482/{{ revealButtonHref }}
https://github.com/python/cpython/pull/156482/files#diff-4c8d59cfb542d6b3e63561140f56a78e991cd0b8019f7de332deba771863735b
Please reload this pagehttps://github.com/python/cpython/pull/156482/files
Please reload this pagehttps://github.com/python/cpython/pull/156482/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.