Title: Preserve tokenizer-only compiler warning order by youknowdot · Pull Request #9316 · RustPython/RustPython · GitHub
Open Graph Title: Preserve tokenizer-only compiler warning order by youknowdot · Pull Request #9316 · RustPython/RustPython
X Title: Preserve tokenizer-only compiler warning order by youknowdot · Pull Request #9316 · RustPython/RustPython
Description: Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one tokenizer-origin escape and no parser-origin escape. Numeric warnings may occur in the same batch. Keep tokenizer-error recovery disabled: an escaped-brace warning promoted to an error must stop before a later numeric warning. This restores missing warnings for ordinary compile("t'\\{1}'", ..., "eval") and fixes ordering/error selection when an f/t-string escaped-brace warning precedes a numeric warning. Parser-only input keeps the existing #9296 path. No warning messages, exception metadata/args, raw-string handling, or callback conversion are changed. Deliberately excluded Mixed parser/tokenizer escape events. Template body decoding and nested ordinary strings have different timing; sorting every warning by source position would be incorrect. Invalid source, including invalid eval input that is valid as a module. single, interactive/prompt, AST-only and BlockExpr paths. Lexical-error recovery such as b'\z' + 0o8 under the error filter. Existing differences in those cases remain. Tokenizer error offsets/text/args also remain outside this patch. Validation Tested base: 0b0a3607648b5b01b3adfc1c06f3bec1d87f8a13. Candidate: 11c81757ef68e77c7aecddf104937de4beaf9926. Compared actual sealed RustPython binaries with final CPython 3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). The historical baseline is eb79fef279d646aedf5fca6598d330b54a401138; its warning implementation matches this PR's main base. Source-sensitive native crates were rebuilt for the candidate rather than reusing another worktree's VM artifact. Focused 18 observations: warning/error-selection matches improved from 8/18 to 13/18; complete raw matches, including exception fields/args/repr, improved from 1/18 to 4/18. Exactly five outputs changed, all as intended. Existing 24 observations: selection matches improved from 18/24 to 20/24; complete raw matches improved from 10/24 to 11/24. Only the two f-string tokenizer-before-numeric cases changed. No raw or selection regression in either set. Mixed-origin, invalid-source, Single, AST-only and invalid-octal controls retained their baseline outputs. Three existing canonical test_fstring methods passed: test_backslashes_in_string_part, test_fstring_backslash_before_double_bracket, and test_fstring_backslash_before_double_bracket_warns_once. CLI build with capi; scoped VM threading, nonthreading, no-default-feature and test-target checks; strict all-targets VM Clippy passed. These are scoped checks, not a full test-suite claim. Concurrent changes Source-only composition was reviewed with #9304 and #9314; a combined runtime build was not performed. When resolving #9304, preserve the explicit tokenizer_recovery: !tokenizer_only rather than its earlier unconditional true. The f/t-body diagnostic change adds a field to the escape event, so the origin-only patterns need .. when those changes are combined. AI assistance OpenAI Codex assisted with the implementation, source investigation, validation and PR description. Summary by CodeRabbit Bug Fixes Improved warning reporting in tokenizer-only execution modes. Escape-related warnings originating during tokenization are now reported as a collected batch without invoking the parser-related fallback or tokenizer recovery. Existing fallback behavior is unchanged for invalid syntax and for warnings originating from both tokenization and parsing. This preserves the established handling of those cases while making tokenizer-only warning reports more consistent. Conflict resolution validation (2026-10-23) Merged main afe5d9030a2c2722642699b00c71f24301805826 into this branch without rewriting its history. Preserved #9304's Single input boundary and recovery behavior, while retaining tokenizer-only Exec/Eval eligibility and tokenizer_recovery: !tokenizer_only. The PR diff remains one Rust file, +17/-4; no library or test changes were added. Validated tree 1062d4076b918cc9e238ec1fce7ac88d48b036cc, committed unchanged as 78a96fde5a904ba48b7c45d45324ebd0c8554423: Rust 1.99.1 VM library check, strict VM library Clippy, and default debug CLI build passed. Normal commit hooks passed without changing the tested tree. All three selected, unmodified canonical CPython test_fstring methods passed: test_backslashes_in_string_part, test_fstring_backslash_before_double_bracket, and test_fstring_backslash_before_double_bracket_warns_once. A fresh finite set of 32 compile observations compared the built candidate with final CPython 3.15.0 (d220409a13b77de7a6c30abe16ea90db01528edc). Warning/error selection matched in 29/32, including all 12 eligible tokenizer-only observations. Complete raw outputs matched in 12/32. The three selection differences were the explicitly excluded invalid-eval/module input under both filters and invalid-octal input under the error filter. Both Single boundary controls matched selection under both filters. These are scoped checks, not a full-suite result. No selected-main baseline binary was rebuilt, so the counts are candidate-versus-CPython parity, not measured before/after improvements. Existing exception metadata/args differences remain outside this patch. The canonical tests used their unmodified source file with the candidate's real standard-library/support modules; no fakes or test-marker changes were used. Source-only mergeability was checked against newer main 5dcef1aa6d53c72315a44296146c768120841953; the merge was clean. No combined runtime build with that newer main was performed.
Open Graph Description: Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one to...
X Description: Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one to...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9316
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:f6cde38d-7d76-644a-1d2a-442e9fafb78e |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D12A:4D9F2:154AC99:1C4914A:6B01FE4A |
| html-safe-nonce | 357b84f1291569f5e9941ebef915873599f3792f3cba9a7c3c028e1431dab0fa |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMTJBOjREOUYyOjE1NEFDOTk6MUM0OTE0QTo2QjAxRkU0QSIsInZpc2l0b3JfaWQiOiI4NzA1NTE5NTI5NTk5NTY1Mzg2IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | c4439cdfc99f94f222d5ac5aff9dd3ff6c592865610db5c8a312ed97a8de0a31 |
| hovercard-subject-tag | pull_request:4968701751 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9316/files |
| twitter:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| og:image:alt | Summary Follow up #9296 for valid exec/eval input whose escape warnings all originate in the tokenizer. Reuse the existing collected warning batch, in source order, when it contains at least one to... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width