René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Complete rustpython-unicode isolation: case mapping, casing predicates, and sre case/space parity","articleBody":"Follow-up to #7560, continuing from #8211 (merged).\n\n## Background\n\n#8211 extracted `crates/unicode` (`rustpython-unicode`) and routed the\nunicodedata database, str classification predicates, casefold, identifier\npredicates, `\\N{}` name lookup, and the sre `\\d`/`\\w` tables through it.\n\nThe goal of #7560 — *one authoritative Unicode path*, usable outside the\nPython runtime — is not finished yet. Case conversion and casing predicates\nstill live inside `rustpython-vm` as direct icu4x calls, so any consumer\nother than the vm has to re-implement them (and will drift from CPython in\nexactly the ways the shared crate is meant to prevent: titlecase mappings\nfor `Lt` characters, final-sigma context, case-ignorable segmentation).\nThe sre engine also still carries two unverified case/space quirks marked\n`TODO: check with cpython`. This issue proposes the remaining work to make\nthe crate the single Unicode authority.\n\n## Current gaps (verified against main)\n\n### 1. Case conversion and casing predicates are vm-private\n\n`crates/vm` is the only crate besides `crates/unicode` that still depends on\n`icu_casemap` / `icu_locale` / `icu_properties` directly:\n\n- `vm/src/builtins/str.rs`:\n  - `title` / `capitalize` — `TitlecaseMapper::titlecase_segment` plus a\n    hand-written cased/case-ignorable segmentation loop\n    (`titlecase_string`, `titlecase_first`).\n  - `lower` (via `str::to_lowercase`), `capitalize`/`title`/`swapcase` —\n    context-sensitive final-sigma handling (`lowercase_or_sigma`,\n    `handle_capital_sigma` over `Cased` / `CaseIgnorable`).\n  - `istitle` — `GeneralCategoryGroup::TitlecaseLetter` +\n    `char::is_uppercase`/`is_lowercase`.\n- `vm/src/anystr.rs`:\n  - `is_cased\u003cVALID, INVALID\u003e` — the shared `islower`/`isupper` kernel,\n    generic over the icu `BinaryProperty` trait (`Lowercase` / `Uppercase`),\n    also consulting `TitlecaseLetter`.\n\nEverything above is Unicode *semantics* (context-sensitive case mapping,\nderived casing properties), not a string algorithm; per #7560's own\narchitecture (\"`crate::str` calls into rustpython-unicode for all\nUnicode-sensitive behavior\") it belongs in the shared crate. The crate's\n`case` module currently exposes only `casefold_str` / `casefold_wtf8`.\n\n### 2. sre_engine case/space helpers are unverified against CPython\n\n`crates/sre_engine/src/string.rs`:\n\n- `is_uni_space` — `SRE_UNI_IS_SPACE` is `Py_UNICODE_ISSPACE`, i.e. exactly\n  `classify::is_space`, but the current implementation is a hand-rolled BMP\n  `matches!` list marked `TODO: check with cpython`.\n- `lower_unicode` / `upper_unicode` — CPython's `sre_lower_unicode` /\n  `sre_upper_unicode` use the *simple* one-to-one mappings\n  (`Py_UNICODE_TOLOWER` / `TOUPPER`). The current implementation takes the\n  first character of the *full* mapping\n  (`to_lowercase().next()`), which diverges wherever a code point has a\n  full mapping but no simple one — e.g. `upper_unicode('ß')` returns `'S'`\n  while CPython keeps `'ß'` (U+00DF has no simple uppercase). Both are\n  marked `TODO: check with cpython`.\n\n## Proposal\n\n### A. Expand `unicode::case` (move, no behavior change)\n\nCode-point level:\n\n- `simple_lowercase` / `simple_uppercase` / `simple_titlecase` /\n  `simple_casefold` — direct wrappers over\n  `icu_casemap::CaseMapper`'s simple mappings (pure, borrowed compiled\n  data, no_std-clean). Needed by sre (B) and by any consumer implementing\n  per-code-point casing.\n- Casing predicates: `is_lowercase` (`Lowercase`), `is_uppercase`\n  (`Uppercase`), `is_titlecase` (`Lt`), `is_cased` (`Cased`),\n  `is_case_ignorable` (`Case_Ignorable`). These retire the icu\n  `BinaryProperty` generics in `vm/anystr.rs` and provide the\n  `islower`/`isupper`/`istitle` kernels.\n\nString level (`\u0026str` and `\u0026Wtf8`, following the `casefold_str` /\n`casefold_wtf8` precedent; lone surrogates pass through unchanged):\n\n- `lower_*` / `upper_*` — full mappings incl. final-sigma context.\n- `title_*` / `capitalize_*` / `swapcase_*` — move `titlecase_string`,\n  `titlecase_first`, `lowercase_or_sigma`, `handle_capital_sigma` from\n  `vm/builtins/str.rs` verbatim. The crate gains an `icu_locale` dependency\n  (root-locale `TitlecaseMapper`), which is no_std-compatible.\n\nThis is a pure relocation: byte-for-byte behavior, gated by `test_str`.\n\n### B. Fix sre_engine parity (behavior changes, separate commits)\n\n- `is_uni_space` → `classify::is_space`.\n- `lower_unicode` / `upper_unicode` → `case::simple_lowercase` /\n  `case::simple_uppercase`.\n- Each change is a documented CPython-correctness fix, verified by a full\n  scalar-range sweep (see Verification) — not folded into the move commit.\n\n### C. Drop icu from `rustpython-vm`\n\nAfter A, `vm/builtins/str.rs` and `vm/anystr.rs` consume `unicode::case`,\nand `icu_casemap` / `icu_locale` / `icu_properties` leave\n`crates/vm/Cargo.toml`. Acceptance: `crates/unicode` is the only workspace\nmember with a direct icu dependency.\n\n### D. (Optional) CodePoint-level convenience layer\n\n#7560 sketched a u32-first API; the crate today is `char`-first for\npredicates and `CodePoint`/`Wtf8` where non-scalars matter. If consumer\ncall sites show repeated `CodePoint -\u003e char` boilerplate, add thin\n`CodePoint` variants (predicates return `false`, mappings return identity\nfor lone surrogates). Decide from real usage, not up front.\n\n## Verification\n\n- Extend `crates/unicode/tests/differential.rs` and the committed reference\n  dataset with the new dimensions, reusing the existing version-skew\n  allow-list mechanism:\n  - casing predicates (`islower`-class, `isupper`-class, cased,\n    case-ignorable, `Lt`) swept over `0..0x110000`;\n  - simple case mappings swept against CPython — `_sre.getlower(ch, flags)`\n    covers the lowercase path directly; for upper/title, generate the\n    reference from the vendored `UnicodeData.txt` simple-mapping columns\n    and pin down CPython's exact table choice as part of the work.\n- Context-sensitive string-level behavior (final sigma, titlecase\n  segmentation) is covered by CPython's own `test_str` (`title`,\n  `capitalize`, `swapcase` cases) plus a small generated corpus if gaps\n  show up.\n- Regression gate unchanged: `test_unicodedata`, `test_str`, `test_re`,\n  `test_pkgutil`, `extra_tests/snippets/stdlib_unicode_shared.py` —\n  identical or better only. `cargo build -p rustpython-unicode --target\n  thumbv7em-none-eabi` stays green.\n\n## Out of scope\n\n- Python str object behavior and non-case string algorithms (unchanged\n  non-goals of #7560).\n- Locale-dependent casing (Turkish/Azeri dotted-I etc.) — CPython's `str`\n  does not perform it either; the root locale is the specified behavior.\n","author":{"url":"https://github.com/youknowone","@type":"Person","name":"youknowone"},"datePublished":"2026-07-08T04:18:11.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/8236/RustPython/issues/8236"}