Title: Fix source decoding diagnostics and normalize decoded newlines by youknowdot · Pull Request #9429 · RustPython/RustPython · GitHub
Open Graph Title: Fix source decoding diagnostics and normalize decoded newlines by youknowdot · Pull Request #9429 · RustPython/RustPython
X Title: Fix source decoding diagnostics and normalize decoded newlines by youknowdot · Pull Request #9429 · RustPython/RustPython
Description: Summary Preserve the coding-cookie line for BOM conflicts, recognize CR line boundaries, and continue scanning after invalid coding candidates. Convert codec initialization failures to SyntaxError while preserving the underlying reason, following CPython's tokenizer initialization error handling. Normalize successfully decoded source before parsing so literal values, warning ranges, and diagnostic text use the same universal-newline input. Add the unchanged CPython 3.15 test_source_encoding suite using scripts/update_lib migrate from d220409a13b77de7a6c30abe16ea90db01528edc. The native changes are confined to the source decoder and its encoding-location record. They do not replace the compiler/parser diagnostic or type-comment work in #8954. Validation Focused CLI/VM cargo check and strict Clippy (-D warnings) passed. Repository hooks passed, including cspell; 228 updater unit tests passed. All 93 ordinary canonical cases passed, without new skips or assertion changes. The separate canonical Linux CPU validator passed all 489 fixed subprocess cases within its 300-second wall-time limit. A bounded CPython/RustPython comparison agrees on CR/CRLF literal values, AST locations, and syntax-error location attributes for compile(str) and compile(bytes). Scope and remaining differences This is not a claim of complete tokenizer compatibility. Successful decoder consumers are normalized together; direct string paths that bypass the decoder are not changed. CPython's pre-codec newline conversion and file-stream-relative codec error offsets remain separate work. Generic SyntaxError.args location tuples also remain different even where the location attributes match. No dependency pins or Ruff patches are changed. AI assistance OpenAI Codex assisted with the implementation, CPython source comparison, tests, and this pull request description; the model version is unavailable. Full workspace and C-API tests were not run locally; hosted CI remains pending. Summary by CodeRabbit Bug Fixes Improved recognition of source-file encoding declarations, including declarations on either of the first two lines and files with a UTF-8 byte-order mark. Source files using CRLF or CR line endings are now normalized consistently, helping prevent parsing issues. Encoding-related errors now provide more useful location and filename information.
Open Graph Description: Summary Preserve the coding-cookie line for BOM conflicts, recognize CR line boundaries, and continue scanning after invalid coding candidates. Convert codec initialization failures to SyntaxError...
X Description: Summary Preserve the coding-cookie line for BOM conflicts, recognize CR line boundaries, and continue scanning after invalid coding candidates. Convert codec initialization failures to SyntaxError...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9429
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:5770c15f-eb39-1000-e0ed-f0821997046e |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | B398:A99C2:3888485:47DB0F6:6B01DE8B |
| html-safe-nonce | 9d1ddf358a563a71e84046e8402de063dd460ad68f1464f52b3f453f26123445 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCMzk4OkE5OUMyOjM4ODg0ODU6NDdEQjBGNjo2QjAxREU4QiIsInZpc2l0b3JfaWQiOiI1OTc1ODIzMDEyNzQwNTg3MTQ4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 6181845cb778c0854e7e34f4b043bd1e9e68a68b3f89b3be36af99804625b01a |
| hovercard-subject-tag | pull_request:5050572374 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9429/files |
| twitter:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| og:image:alt | Summary Preserve the coding-cookie line for BOM conflicts, recognize CR line boundaries, and continue scanning after invalid coding candidates. Convert codec initialization failures to SyntaxError... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width