René's URL Explorer Experiment


Title: Preserve tokenizer-only compiler warning order by youknowdot · Pull Request #9316 · RustPython/RustPython · GitHub

Open Graph Title: Preserve tokenizer-only compiler warning order by youknowdot · Pull Request #9316 · RustPython/RustPython

X Title: Preserve tokenizer-only compiler warning order by youknowdot · Pull Request #9316 · RustPython/RustPython

Description: Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one tokenizer-origin escape and no parser-origin escape. Numeric warnings may occur in the same batch. Keep tokenizer-error recovery disabled: an escaped-brace warning promoted to an error must stop before a later numeric warning. This restores missing warnings for ordinary compile("t'\\{1}'", ..., "eval") and fixes ordering/error selection when an f/t-string escaped-brace warning precedes a numeric warning. Parser-only input keeps the existing #9296 path. No warning messages, exception metadata/args, raw-string handling, or callback conversion are changed. Deliberately excluded Mixed parser/tokenizer escape events. Template body decoding and nested ordinary strings have different timing; sorting every warning by source position would be incorrect. Invalid source, including invalid eval input that is valid as a module. single, interactive/prompt, AST-only and BlockExpr paths. Lexical-error recovery such as b'\z' + 0o8 under the error filter. Existing differences in those cases remain. Tokenizer error offsets/text/args also remain outside this patch. Validation Tested base: 0b0a3607648b5b01b3adfc1c06f3bec1d87f8a13. Candidate: 11c81757ef68e77c7aecddf104937de4beaf9926. Compared actual sealed RustPython binaries with final CPython 3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). The historical baseline is eb79fef279d646aedf5fca6598d330b54a401138; its warning implementation matches this PR's main base. Source-sensitive native crates were rebuilt for the candidate rather than reusing another worktree's VM artifact. Focused 18 observations: warning/error-selection matches improved from 8/18 to 13/18; complete raw matches, including exception fields/args/repr, improved from 1/18 to 4/18. Exactly five outputs changed, all as intended. Existing 24 observations: selection matches improved from 18/24 to 20/24; complete raw matches improved from 10/24 to 11/24. Only the two f-string tokenizer-before-numeric cases changed. No raw or selection regression in either set. Mixed-origin, invalid-source, Single, AST-only and invalid-octal controls retained their baseline outputs. Three existing canonical test_fstring methods passed: test_backslashes_in_string_part, test_fstring_backslash_before_double_bracket, and test_fstring_backslash_before_double_bracket_warns_once. CLI build with capi; scoped VM threading, nonthreading, no-default-feature and test-target checks; strict all-targets VM Clippy passed. These are scoped checks, not a full test-suite claim. Concurrent changes Source-only composition was reviewed with #9304 and #9314; a combined runtime build was not performed. When resolving #9304, preserve the explicit tokenizer_recovery: !tokenizer_only rather than its earlier unconditional true. The f/t-body diagnostic change adds a field to the escape event, so the origin-only patterns need .. when those changes are combined. AI assistance OpenAI Codex assisted with the implementation, source investigation, validation and PR description. Summary by CodeRabbit Bug Fixes Improved warning reporting in tokenizer-only execution modes. Escape-related warnings originating during tokenization are now reported as a collected batch without invoking the parser-related fallback or tokenizer recovery. Existing fallback behavior is unchanged for invalid syntax and for warnings originating from both tokenization and parsing. This preserves the established handling of those cases while making tokenizer-only warning reports more consistent. Conflict resolution validation (2026-10-23) Merged main afe5d9030a2c2722642699b00c71f24301805826 into this branch without rewriting its history. Preserved #9304's Single input boundary and recovery behavior, while retaining tokenizer-only Exec/Eval eligibility and tokenizer_recovery: !tokenizer_only. The PR diff remains one Rust file, +17/-4; no library or test changes were added. Validated tree 1062d4076b918cc9e238ec1fce7ac88d48b036cc, committed unchanged as 78a96fde5a904ba48b7c45d45324ebd0c8554423: Rust 1.99.1 VM library check, strict VM library Clippy, and default debug CLI build passed. Normal commit hooks passed without changing the tested tree. All three selected, unmodified canonical CPython test_fstring methods passed: test_backslashes_in_string_part, test_fstring_backslash_before_double_bracket, and test_fstring_backslash_before_double_bracket_warns_once. A fresh finite set of 32 compile observations compared the built candidate with final CPython 3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). Warning/error selection matched in 29/32, including all 12 eligible tokenizer-only observations. Complete raw outputs matched in 12/32. The three selection differences were the explicitly excluded invalid-eval/module input under both filters and invalid-octal input under the error filter. Both Single boundary controls matched selection under both filters. These are scoped checks, not a full-suite result. No selected-main baseline binary was rebuilt, so the counts are candidate-versus-CPython parity, not measured before/after improvements. Existing exception metadata/args differences remain outside this patch. The canonical tests used their unmodified source file with the candidate's real standard-library/support modules; no fakes or test-marker changes were used. Source-only mergeability was checked against newer main 5dcef1aa6d53c72315a44296146c768120841953; the merge was clean. No combined runtime build with that newer main was performed.

Open Graph Description: Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one to...

X Description: Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one to...

Opengraph URL: https://github.com/RustPython/RustPython/pull/9316

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:f6cde38d-7d76-644a-1d2a-442e9fafb78e
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD12A:4D9F2:154AC99:1C4914A:6B01FE4A
html-safe-nonce357b84f1291569f5e9941ebef915873599f3792f3cba9a7c3c028e1431dab0fa
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMTJBOjREOUYyOjE1NEFDOTk6MUM0OTE0QTo2QjAxRkU0QSIsInZpc2l0b3JfaWQiOiI4NzA1NTE5NTI5NTk5NTY1Mzg2IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacc4439cdfc99f94f222d5ac5aff9dd3ff6c592865610db5c8a312ed97a8de0a31
hovercard-subject-tagpull_request:4968701751
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9316/files
twitter:imagehttps://avatars.githubusercontent.com/u/336763669?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/336763669?s=400&v=4
og:image:altSummary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one to...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git
octolytics-dimension-user_id39710557
octolytics-dimension-user_loginRustPython
octolytics-dimension-repository_id135201145
octolytics-dimension-repository_nwoRustPython/RustPython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id135201145
octolytics-dimension-repository_network_root_nwoRustPython/RustPython
turbo-body-classeslogged-out env-production page-responsive
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/RustPython/RustPython/pull/9316/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FRustPython%2FRustPython%2Fpull%2F9316%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FRustPython%2FRustPython%2Fpull%2F9316%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=RustPython%2FRustPython
Reloadhttps://github.com/RustPython/RustPython/pull/9316/files
Reloadhttps://github.com/RustPython/RustPython/pull/9316/files
Reloadhttps://github.com/RustPython/RustPython/pull/9316/files
RustPythonhttps://github.com/RustPython
RustPythonhttps://github.com/RustPython/RustPython
Codehttps://github.com/RustPython/RustPython
Issues195 (195)https://github.com/RustPython/RustPython/issues
Pull requests133 (133)https://github.com/RustPython/RustPython/pulls
Discussionshttps://github.com/RustPython/RustPython/discussions
Actionshttps://github.com/RustPython/RustPython/actions
Projectshttps://github.com/RustPython/RustPython/projects
Wikihttps://github.com/RustPython/RustPython/wiki
Security and qualityhttps://github.com/RustPython/RustPython/security
Insightshttps://github.com/RustPython/RustPython/insights
Sign up for GitHub https://github.com/signup?return_to=%2FRustPython%2FRustPython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2FRustPython%2FRustPython%2Fissues%2Fnew%2Fchoose
youknowonehttps://github.com/youknowone
RustPython:mainhttps://github.com/RustPython/RustPython/tree/main
youknowdot:split/tokenizer-only-warning-order-20261022https://github.com/youknowdot/RustPython/tree/split/tokenizer-only-warning-order-20261022
Conversation 5 https://github.com/RustPython/RustPython/pull/9316
Commits 2 https://github.com/RustPython/RustPython/pull/9316/commits
Checks 19 https://github.com/RustPython/RustPython/pull/9316/checks
Files changed https://github.com/RustPython/RustPython/pull/9316/files
Please reload this pagehttps://github.com/RustPython/RustPython/pull/9316/files
Preserve tokenizer-only compiler warning order https://github.com/RustPython/RustPython/pull/9316/files#top
Show all changes 2 commits https://github.com/RustPython/RustPython/pull/9316/files
11c8175 Preserve tokenizer-only compiler warning order youknowdot Oct 22, 2026 https://github.com/RustPython/RustPython/pull/9316/commits/11c81757ef68e77c7aecddf104937de4beaf9926
78a96fd Merge main and preserve tokenizer-only warning ordering youknowdot Oct 23, 2026 https://github.com/RustPython/RustPython/pull/9316/commits/78a96fde5a904ba48b7c45d45324ebd0c8554423
Clear filters https://github.com/RustPython/RustPython/pull/9316/files
Please reload this pagehttps://github.com/RustPython/RustPython/pull/9316/files
Please reload this pagehttps://github.com/RustPython/RustPython/pull/9316/files
crates/vm/src/vm/compile.rshttps://github.com/RustPython/RustPython/pull/9316/files#diff-e7f70f57a7775d5472ffe0e4fe2f28f1c386fad78bef2ba2c46fd7f2224fa89e
View file https://github.com/youknowdot/RustPython/blob/78a96fde5a904ba48b7c45d45324ebd0c8554423/crates/vm/src/vm/compile.rs
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/RustPython/RustPython/pull/9316/{{ revealButtonHref }}
https://github.com/RustPython/RustPython/pull/9316/files#diff-e7f70f57a7775d5472ffe0e4fe2f28f1c386fad78bef2ba2c46fd7f2224fa89e
https://github.com/RustPython/RustPython/pull/9316/files#diff-e7f70f57a7775d5472ffe0e4fe2f28f1c386fad78bef2ba2c46fd7f2224fa89e
chatgpt-codex-connectorhttps://github.com/apps/chatgpt-codex-connector
Oct 23, 2026https://github.com/RustPython/RustPython/pull/9316/files#r4357888493
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
https://camo.githubusercontent.com/0854bfc38139c0d2b58a9d5d71f5ad92840cf0c17144621e462b474041793354/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f50322d79656c6c6f773f7374796c653d666c6174
Please reload this pagehttps://github.com/RustPython/RustPython/pull/9316/files
https://github.com/RustPython/RustPython/pull/9316/files#diff-e7f70f57a7775d5472ffe0e4fe2f28f1c386fad78bef2ba2c46fd7f2224fa89e
https://github.com/RustPython/RustPython/pull/9316/files#diff-e7f70f57a7775d5472ffe0e4fe2f28f1c386fad78bef2ba2c46fd7f2224fa89e
Please reload this pagehttps://github.com/RustPython/RustPython/pull/9316/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.