René's URL Explorer Experiment


Title: gh-153569: consolidate tokenizer input, layout, and diagnostics by pablogsal · Pull Request #157055 · python/cpython · GitHub

Open Graph Title: gh-153569: consolidate tokenizer input, layout, and diagnostics by pablogsal · Pull Request #157055 · python/cpython

X Title: gh-153569: consolidate tokenizer input, layout, and diagnostics by pablogsal · Pull Request #157055 · python/cpython

Description: Put decoded input ownership and interactive-input control in the reader, and group indentation and logical-line handling in the layout state. Record diagnostic locations and source spans separately from the scanning cursor. Parser consumers use the tokenizer snapshot and view APIs, including the diagnostic context. Remove the unused cursor and source lookup APIs, duplicate accessors, obsolete state, and dead lexer header and debug helper. Part of gh-153569. Last PR in the stack; the preceding PRs have landed.

Open Graph Description: Put decoded input ownership and interactive-input control in the reader, and group indentation and logical-line handling in the layout state. Record diagnostic locations and source spans separately...

X Description: Put decoded input ownership and interactive-input control in the reader, and group indentation and logical-line handling in the layout state. Record diagnostic locations and source spans separately...

Opengraph URL: https://github.com/python/cpython/pull/157055

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:1468ea68-87ab-c5bd-15e1-bcd65e1a3942
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCE44:25F5AB:545059:6B043C:6B021677
html-safe-nonce5abd50889b3dcca911a6afc7604155aaede1e604d8c8df6da72b4815b25ea16e
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRTQ0OjI1RjVBQjo1NDUwNTk6NkIwNDNDOjZCMDIxNjc3IiwidmlzaXRvcl9pZCI6Ijc5MDQ1MzgxODUwNzY3MDg5ODMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacba9419614c49dfefa5d4c3cea6d0a09cbbfa257d7c4ffe70d11861a3e4fd9726
hovercard-subject-tagpull_request:4458931642
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/157055/files
twitter:imagehttps://avatars.githubusercontent.com/u/11718525?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/11718525?s=400&v=4
og:image:altPut decoded input ownership and interactive-input control in the reader, and group indentation and logical-line handling in the layout state. Record diagnostic locations and source spans separately...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/157055/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F157055%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F157055%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/157055/files
Reloadhttps://github.com/python/cpython/pull/157055/files
Reloadhttps://github.com/python/cpython/pull/157055/files
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
pablogsalhttps://github.com/pablogsal
python:mainhttps://github.com/python/cpython/tree/main
pablogsal:gh-153569-tokenizer-input-diagnosticshttps://github.com/pablogsal/cpython/tree/gh-153569-tokenizer-input-diagnostics
Conversation 0 https://github.com/python/cpython/pull/157055
Commits 6 https://github.com/python/cpython/pull/157055/commits
Checks 78 https://github.com/python/cpython/pull/157055/checks
Files changed https://github.com/python/cpython/pull/157055/files
Please reload this pagehttps://github.com/python/cpython/pull/157055/files
gh-153569: consolidate tokenizer input, layout, and diagnostics https://github.com/python/cpython/pull/157055/files#top
Show all changes 6 commits https://github.com/python/cpython/pull/157055/files
9ad9e67 gh-153569: move tokenizer input state and relocation into the reader pablogsal Sep 6, 2026 https://github.com/python/cpython/pull/157055/commits/9ad9e674333cbc2cebd6436e844a5de48337b1e3
11e1334 gh-153569: borrow diagnostic lines through the source API pablogsal Sep 6, 2026 https://github.com/python/cpython/pull/157055/commits/11e1334f6a982d4e9c6f94f19c3c3a23c30836c7
2311ddb gh-153569: group indentation and logical-line state pablogsal Sep 6, 2026 https://github.com/python/cpython/pull/157055/commits/2311ddb1a094582fb1fbc858b4c0c4e29cd6e127
68f376e gh-153569: remove unused tokenizer cursor and source lookup APIs pablogsal Sep 6, 2026 https://github.com/python/cpython/pull/157055/commits/68f376e05ce547da74d5b78d289ef2a33a997a9d
28e15b4 gh-153569: report tokenizer diagnostics without rewinding the scanner pablogsal Sep 6, 2026 https://github.com/python/cpython/pull/157055/commits/28e15b4242f4dd9613b7ad58b661e0f6f2685363
9016f1d gh-153569: use tokenizer views and remove obsolete API remnants pablogsal Sep 12, 2026 https://github.com/python/cpython/pull/157055/commits/9016f1dfc04204e37451ef1d728618bee98179ae
Clear filters https://github.com/python/cpython/pull/157055/files
Please reload this pagehttps://github.com/python/cpython/pull/157055/files
Please reload this pagehttps://github.com/python/cpython/pull/157055/files
test_tokenizer.py https://github.com/python/cpython/pull/157055/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
test_codeop.py https://github.com/python/cpython/pull/157055/files#diff-8e1ec51814ab6d8b9029ea7d1bee5c2914973c5a57e0d333145e8ed0cd984cfc
test_repl.py https://github.com/python/cpython/pull/157055/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
test_source_encoding.py https://github.com/python/cpython/pull/157055/files#diff-1964a8b9e17ccd9a653e3c7df9d2428af06a437a618160dd92d002c2b683e3d3
test_tstring.py https://github.com/python/cpython/pull/157055/files#diff-8ec218d660fe515f0fe91bca85302204ea791b710cbba8985aaa30f2ff5fafdb
Makefile.pre.in https://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
tokenizer.c https://github.com/python/cpython/pull/157055/files#diff-bf867c315e7bf23f02713dfa30288b5352e1b1cc050dfebb334287359d33d963
_freeze_module.vcxproj https://github.com/python/cpython/pull/157055/files#diff-d6e731738eec137782f1f1add8cb8ae28cbe912b6bedfe170eb91e90c388702e
_freeze_module.vcxproj.filters https://github.com/python/cpython/pull/157055/files#diff-5e3a79d068beb7e72d27ca9ef4847037a9ee40a43dc221fe9868d41b7a9041ff
pythoncore.vcxproj https://github.com/python/cpython/pull/157055/files#diff-4295039788386c2986b811f2a6b59cf088be59822b24b4f1243c9e9b184f65fb
pythoncore.vcxproj.filters https://github.com/python/cpython/pull/157055/files#diff-9e715828befc9c50c8b483b06d4923cda0db44f57d0c20e6a6d428ba60bfa0c5
layout.c https://github.com/python/cpython/pull/157055/files#diff-220cabe3c51ecb5667c1fe47a549e88a9fb933632a5a83b71c8a9fa3e75c1ac0
lexer.c https://github.com/python/cpython/pull/157055/files#diff-4903ddce642a378ef03f37f72332cf1f5acb5bb0c6247669145fc3d8ea7cd43e
lexer.h https://github.com/python/cpython/pull/157055/files#diff-4c8d59cfb542d6b3e63561140f56a78e991cd0b8019f7de332deba771863735b
lexer_internal.h https://github.com/python/cpython/pull/157055/files#diff-8b0f5e75070e787646998b2177e239a0048aa5adce6e2068ad6bbfca1dc46eda
state.c https://github.com/python/cpython/pull/157055/files#diff-d320456c1594700fd4ea3c33fbfc5f7fd74d4fd45fd225354ea8db7135c04045
state.h https://github.com/python/cpython/pull/157055/files#diff-9efe28cd84a24747de9e363846c7d9255a3d3a7073896547f570bf91b4fd3667
string.c https://github.com/python/cpython/pull/157055/files#diff-c8bf656bc09ae7acf6bc1c311f93e578b1df1efab51aa5dcfec00aacbb8425f7
pegen_errors.c https://github.com/python/cpython/pull/157055/files#diff-f15d7b3d0bc84baf512dd165bea4e2e1dddb1fe33f836ee08b811621df3260cb
api.c https://github.com/python/cpython/pull/157055/files#diff-4aed35731bef9e03b57991e87f1560266654a140343fb6cace980e611993aabc
cursor.c https://github.com/python/cpython/pull/157055/files#diff-612535281447d780b732c38843952f91678c5d75d8396e55445dc375319d3813
cursor.h https://github.com/python/cpython/pull/157055/files#diff-f63563ab928ae9883992ec0dddbd66eab2ffd2abf172a22c91d754f7ac649f8c
decoder.c https://github.com/python/cpython/pull/157055/files#diff-20b697a19ba8581bf383b48255bb0cc3bc160c264aeaeeea96be6e2daf471741
helpers.c https://github.com/python/cpython/pull/157055/files#diff-2f9c676ca7974a32907e7c98153e5942c869a80c67d2fd075acb5b589b45eddb
helpers.h https://github.com/python/cpython/pull/157055/files#diff-7f7d9ba2dceb9ccf35b4d482f2d2ffc257e2d04abc4b1062e162d86287595b44
reader.c https://github.com/python/cpython/pull/157055/files#diff-ee1ccf9c461ad2982f27a89c79c7ea2e5868fef86dbd8d8d1d873e1624208590
reader.h https://github.com/python/cpython/pull/157055/files#diff-dabb9c3ef309f6b5363b28c7bcded861fe806c0bedf1368c52387a257b4ec9ed
reader_internal.h https://github.com/python/cpython/pull/157055/files#diff-b0bf3853f60d9c70ce122af7b63f26d886e22d2bf72e06c5a04402dc26b5a494
source.c https://github.com/python/cpython/pull/157055/files#diff-d091331acd32f48897fc1503341091ee6066cbe2208017651deef042b32273da
source.h https://github.com/python/cpython/pull/157055/files#diff-033fac2d1c68ebec297cd0aaf7ee48c836b7ebab03362fcecc53cedcdac043d1
tokenizer.h https://github.com/python/cpython/pull/157055/files#diff-be88e98f60aa508893fa57029c0cafd698c2b1ad66dc1d532919fc8fd0373dd8
build.py https://github.com/python/cpython/pull/157055/files#diff-476cab90bd9f052e2ca52fc323d61913145493cfabeda68fff8feeb086728a9a
Lib/test/test_capi/test_tokenizer.pyhttps://github.com/python/cpython/pull/157055/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
View file https://github.com/pablogsal/cpython/blob/9016f1dfc04204e37451ef1d728618bee98179ae/Lib/test/test_capi/test_tokenizer.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/157055/{{ revealButtonHref }}
https://github.com/python/cpython/pull/157055/files#diff-16b4a247c227d32e31db7b6b107e9236edad213c710cb5b31584c846e93a533d
Lib/test/test_codeop.pyhttps://github.com/python/cpython/pull/157055/files#diff-8e1ec51814ab6d8b9029ea7d1bee5c2914973c5a57e0d333145e8ed0cd984cfc
View file https://github.com/pablogsal/cpython/blob/9016f1dfc04204e37451ef1d728618bee98179ae/Lib/test/test_codeop.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/157055/{{ revealButtonHref }}
https://github.com/python/cpython/pull/157055/files#diff-8e1ec51814ab6d8b9029ea7d1bee5c2914973c5a57e0d333145e8ed0cd984cfc
https://github.com/python/cpython/pull/157055/files#diff-8e1ec51814ab6d8b9029ea7d1bee5c2914973c5a57e0d333145e8ed0cd984cfc
Lib/test/test_repl.pyhttps://github.com/python/cpython/pull/157055/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
View file https://github.com/pablogsal/cpython/blob/9016f1dfc04204e37451ef1d728618bee98179ae/Lib/test/test_repl.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/157055/{{ revealButtonHref }}
https://github.com/python/cpython/pull/157055/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
https://github.com/python/cpython/pull/157055/files#diff-5401943d2b776b0ff00529833651ea8a9ad5ecbbf064efc86a5caa7d746ecd91
Lib/test/test_source_encoding.pyhttps://github.com/python/cpython/pull/157055/files#diff-1964a8b9e17ccd9a653e3c7df9d2428af06a437a618160dd92d002c2b683e3d3
View file https://github.com/pablogsal/cpython/blob/9016f1dfc04204e37451ef1d728618bee98179ae/Lib/test/test_source_encoding.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/157055/{{ revealButtonHref }}
https://github.com/python/cpython/pull/157055/files#diff-1964a8b9e17ccd9a653e3c7df9d2428af06a437a618160dd92d002c2b683e3d3
https://github.com/python/cpython/pull/157055/files#diff-1964a8b9e17ccd9a653e3c7df9d2428af06a437a618160dd92d002c2b683e3d3
https://github.com/python/cpython/pull/157055/files#diff-1964a8b9e17ccd9a653e3c7df9d2428af06a437a618160dd92d002c2b683e3d3
https://github.com/python/cpython/pull/157055/files#diff-1964a8b9e17ccd9a653e3c7df9d2428af06a437a618160dd92d002c2b683e3d3
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L349
Lib/test/test_tstring.pyhttps://github.com/python/cpython/pull/157055/files#diff-8ec218d660fe515f0fe91bca85302204ea791b710cbba8985aaa30f2ff5fafdb
View file https://github.com/pablogsal/cpython/blob/9016f1dfc04204e37451ef1d728618bee98179ae/Lib/test/test_tstring.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/157055/{{ revealButtonHref }}
https://github.com/python/cpython/pull/157055/files#diff-8ec218d660fe515f0fe91bca85302204ea791b710cbba8985aaa30f2ff5fafdb
https://github.com/python/cpython/pull/157055/files#diff-8ec218d660fe515f0fe91bca85302204ea791b710cbba8985aaa30f2ff5fafdb
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L85
Makefile.pre.inhttps://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
View file https://github.com/pablogsal/cpython/blob/9016f1dfc04204e37451ef1d728618bee98179ae/Makefile.pre.in
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/157055/{{ revealButtonHref }}
https://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
https://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
https://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
https://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
https://github.com/python/cpython/pull/157055/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767c
Please reload this pagehttps://github.com/python/cpython/pull/157055/files
Please reload this pagehttps://github.com/python/cpython/pull/157055/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.