Title: Fix string casing against the pinned Unicode database by youknowdot · Pull Request #9824 · RustPython/RustPython · GitHub
Open Graph Title: Fix string casing against the pinned Unicode database by youknowdot · Pull Request #9824 · RustPython/RustPython
X Title: Fix string casing against the pinned Unicode database by youknowdot · Pull Request #9824 · RustPython/RustPython
Description: Summary Use the project's pinned ICU Unicode data consistently for str.lower, str.upper, str.istitle, and the lowercase/uppercase portions of capitalize, title and swapcase. Preserve the ASCII VM fast paths and WTF-8 surrogate handling. Rust 1.100 ships Unicode 18 data, while CPython 3.15 and our Unicode database use Unicode 17. Mixing Rust's char/str case operations with ICU predicates makes test_unicodedata.UnicodeMethodsTest.test_method_checksum fail on main, independently of #9816 and #9820. For example, '\u0277'.upper() must remain unchanged with Unicode 17, but Rust's newer tables map it to U+A7DD. The existing Unicode build script generates compact full-mapping and contextual-property tables from ICU. Runtime casing uses immutable table lookups, including one-to-many mappings, and CPython's lower_ucs4 final-sigma context rules. Lone surrogates remain context boundaries. There is no runtime cache or toolchain-version exception, and no Lib test expectations change. A direct ICU runtime implementation was rejected after release benchmarks showed substantial Latin-text overhead. The generated-table implementation retains an ASCII-prefix path and statically dispatched character output. The tiny table/output helpers are inlined to avoid the measured call overhead. CPython reference: Objects/unicodeobject.c, v3.15.0 (lower_ucs4, do_upper_or_lower, do_swapcase, unicode_istitle_impl). Validation Reproduced main's exact checksum mismatch: a6f9fa98bde3bc2c24ab040ad9fe67b073971bbe versus 8b2615a9fc627676cbc0b6fac0191177df97ef5f. Unicode crate: 26 unit/integration tests passed, including all scalar values against ICU for generated lower/upper mappings and all four casing predicates; ASCII-prefix sigma/combining-mark and lone-surrogate cases included. Release microbenchmarks of actual helpers against Rust std, short and repeated Latin, mixed ASCII/non-ASCII, Greek, Deseret and Turkish/full-expansion inputs. Final Latin/mixed lower was 0.876–0.950 times baseline time, upper 0.856–1.002 times baseline. Other tested inputs were faster. Against the previous main helper implementations, title/capitalize/swapcase took 0.209–0.654 times baseline time on Latin, Greek, mixed and full-expansion inputs. These are microbenchmarks, not a universal speedup claim. Pure ASCII interpreter dispatch is unchanged. Independent source review completed. Commit hooks passed. Rebuilt the interpreter on main 9f6a743 plus this change. All 0x110000 code points matched CPython 3.15.2 for lower/upper/title/capitalize/swapcase/casefold/istitle, with additional sigma, case-ignorable and surrogate-context cases. -m test test_unicodedata test_str: 212 tests passed, 19 skipped. cargo clippy -p rustpython-unicode --all-targets -- -D warnings: passed. Workspace tests were attempted but the local filesystem filled while linking the call_between_rust_and_python example (ld Bus error), before aggregate tests ran. This is not recorded as a passing local aggregate run; the subsequent hosted runs below provide aggregate coverage. Workspace strict Clippy (--all-targets, excluding wasm/venvlauncher/capi, -D warnings): passed. Separate CAPI strict Clippy (--all-targets -- -D warnings): passed. Hosted CI run 43373063655 completed successfully on head 2012fb1b53d5205b806d5098a83adfb6db977a23: native Rust and Python tests on Ubuntu/macOS/Windows, WASI Python tests, WASM package, Miri, lint and remaining scheduled checks passed. Ubuntu CAPI build/tests/doctests steps were individually verified as successful. Conditional matrix/parity jobs were skipped by the workflow. AI-assisted implementation and verification with OpenAI Codex, with independent source review. Exact model version is not exposed in this environment.
Open Graph Description: Summary Use the project's pinned ICU Unicode data consistently for str.lower, str.upper, str.istitle, and the lowercase/uppercase portions of capitalize, title and swapcase. Preserve the ASCII ...
X Description: Summary Use the project's pinned ICU Unicode data consistently for str.lower, str.upper, str.istitle, and the lowercase/uppercase portions of capitalize, title and swapcase. Preserve the AS...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9824
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:cb6fb861-e9d9-6f89-12ed-78cea7b0c0fe |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 8E78:785A9:362E4AF:44CCA05:6B01E6B2 |
| html-safe-nonce | 776eebc352e1332139800687b7aa5cfef5ac10a51fa31c90772d83bfcddf561d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RTc4Ojc4NUE5OjM2MkU0QUY6NDRDQ0EwNTo2QjAxRTZCMiIsInZpc2l0b3JfaWQiOiI5MTc0MTc1NDM0NTQyMjc4MzIyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 983d3330a0c1b70c8d93830930280c7c47cfae495aab215912cf9bc024cb0911 |
| hovercard-subject-tag | pull_request:5290670676 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9824/files |
| twitter:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/336763669?s=400&v=4 |
| og:image:alt | Summary Use the project's pinned ICU Unicode data consistently for str.lower, str.upper, str.istitle, and the lowercase/uppercase portions of capitalize, title and swapcase. Preserve the ASCII ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width