Title: Fix strict tokenizer lexical error propagation by youknowone · Pull Request #9935 · RustPython/RustPython · GitHub
Open Graph Title: Fix strict tokenizer lexical error propagation by youknowone · Pull Request #9935 · RustPython/RustPython
X Title: Fix strict tokenizer lexical error propagation by youknowone · Pull Request #9935 · RustPython/RustPython
Description: Summary Propagate lexical tokenizer errors without treating parser grammar recovery as a lexical failure. Preserve extra-token tolerance, diagnostic arguments/positions, and token/error ordering for malformed numbers, brackets, strings, interpolations, and NUL input. Remove the existing test_invalid_syntax expectedFailure marker. No new repository tests or changes to canonical assertions. Verification Linux x86_64, Rust 1.100.0, CPython 3.15.2 reference; main bf68333 baseline. test_tokenize passes (132 tests run). 85 private differential inputs in both modes: all 170 complete records match, including token tuples and exception class/arguments/text/start/end positions. Workspace all-targets Clippy passes, excluding wasm, venvlauncher, and CAPI. Mandatory commit hooks pass; prettier skipped because no applicable files. Full non-CAPI Rust workspace tests and doctests pass: 1,619 passed, 21 ignored, 0 failed. The ignored cases are two existing process-isolation tests and 19 doctests. CAPI remains deferred per the maintainer's instruction; Windows/WASI and CI are not yet verified. Quiet alternating baseline/candidate debug microbenchmarks over four rounds showed median changes of −3.4% for many-line source, −4.6% for long-line source, and −8.1% for f-strings. These local regression checks are not release-performance guarantees. Remaining limits This is not complete tokenizer parity. Ruff splits invalid Unicode identifiers such as a€b, so strict mode may emit the valid prefix before the correct diagnostic, and extra-token boundaries differ. A private lexer workaround is deliberately not added. A continuation ending in bare CR still has an EOF line/offset difference; LF/CRLF probes match. Existing eager readline/callback ordering and numeric-keyword SyntaxWarning/filter behavior remain outside this change. AI assistance Implementation, differential probes, and independent review used OpenAI Codex. The runtime does not expose an exact model version, so the commit records Assisted-by: Codex:model-unspecified rather than inventing one.
Open Graph Description: Summary Propagate lexical tokenizer errors without treating parser grammar recovery as a lexical failure. Preserve extra-token tolerance, diagnostic arguments/positions, and token/error ordering f...
X Description: Summary Propagate lexical tokenizer errors without treating parser grammar recovery as a lexical failure. Preserve extra-token tolerance, diagnostic arguments/positions, and token/error ordering f...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9935
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:89076d2b-e690-e11e-51fa-a472c516c62d |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | B4AA:CE438:C2A6CC:104E1B8:6B01D7E9 |
| html-safe-nonce | 275721c1919eed46ad66087fdaa7c1f69598fba7a00521c28d3d01ef03d96aac |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNEFBOkNFNDM4OkMyQTZDQzoxMDRFMUI4OjZCMDFEN0U5IiwidmlzaXRvcl9pZCI6Ijc3NDAxNzEyMDcyNzA2MjUwNSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 8ca34c125db55b446f5d6f218a7261c185dcac25358419dd614bd9bfc7ed6aad |
| hovercard-subject-tag | pull_request:5369280790 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9935/files |
| twitter:image | https://avatars.githubusercontent.com/u/69878?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/69878?s=400&v=4 |
| og:image:alt | Summary Propagate lexical tokenizer errors without treating parser grammar recovery as a lexical failure. Preserve extra-token tolerance, diagnostic arguments/positions, and token/error ordering f... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width