René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"rustpython-unicode","articleBody":"rustpython-unicode should be a shared crate that provides CPython-compatible Unicode\nsemantics and Unicode data for Rust-based Python implementations and tools.\n\nIts purpose is not to implement Python string objects or high-level string methods. Instead,\nit should provide the Unicode foundation that Python runtimes need: character classification,\ncase mapping, normalization, identifier rules, regex character-class predicates, and\nunicodedata-style access to Unicode database information.\n\nThis makes it a natural shared dependency between RustPython and Pyre, while keeping actual\nstring operations in crate::str or the host runtime.\n\nDesign Goals\n\n- Match CPython behavior at the Unicode data and semantics level, not just in a few visible\n  edge cases.\n- Use a single version-pinned Unicode data source per targeted CPython release.\n- Expose low-level APIs that can be reused by str, re, parser, compiler, and unicodedata.\n- Support non-scalar code points where Python behavior requires it, so the core API should be\n  u32-based rather than char-based.\n- Be usable outside RustPython, especially by other Python-related Rust projects.\n\nScope\n\nrustpython-unicode should include:\n\n- Unicode character classification used by Python:\n    - isalpha\n    - isalnum\n    - isdecimal\n    - isdigit\n    - isnumeric\n    - isspace\n    - isprintable\n    - casing predicates if needed\n- Identifier-related predicates:\n    - is_xid_start\n    - is_xid_continue\n    - Python identifier helpers\n- Regex-oriented Unicode predicates:\n    - Unicode \\w\n    - Unicode \\d\n    - Unicode \\s\n    - any other CPython regex character classes that depend on Unicode tables\n- Case conversion and casing-related data:\n    - lowercase\n    - uppercase\n    - titlecase\n    - casefold\n    - full mappings where Python requires them\n- Unicode normalization support needed by Python:\n    - NFC\n    - NFD\n    - NFKC\n    - NFKD\n    - is_normalized\n- unicodedata-style database access:\n    - general category\n    - bidirectional class\n    - combining class\n    - east asian width\n    - mirrored\n    - decomposition\n    - decimal/digit/numeric values\n    - name lookup\n    - character lookup by name\n    - Unicode age/version checks if needed for CPython compatibility layers\n- Versioned Unicode tables aligned with CPython.\n\nNon-Goals\n\nrustpython-unicode should not include:\n\n- Python str object behavior\n- slicing, searching, splitting, joining, formatting, padding, or other string algorithms\n- Python object model concerns\n- interpreter-specific wrappers\n\nThose belong in crate::str or the embedding runtime.\n\nArchitecture\n\nA good split would be:\n\n- rustpython-unicode\n    - owns Unicode tables and Unicode semantics\n    - exposes u32-based predicates and mappings\n    - exposes unicodedata-style query APIs\n- crate::str\n    - owns Python string methods and higher-level string algorithms\n    - calls into rustpython-unicode for all Unicode-sensitive behavior\n- regex engine\n    - calls into rustpython-unicode for Unicode character classes\n- unicodedata module\n    - becomes a thin wrapper over rustpython-unicode\n\nThis keeps one authoritative Unicode path across the runtime.\n\nSuggested Public API Direction\n\npub mod classify {\n    pub fn is_alpha(cp: u32) -\u003e bool;\n    pub fn is_alnum(cp: u32) -\u003e bool;\n    pub fn is_decimal(cp: u32) -\u003e bool;\n    pub fn is_digit(cp: u32) -\u003e bool;\n    pub fn is_numeric(cp: u32) -\u003e bool;\n    pub fn is_space(cp: u32) -\u003e bool;\n    pub fn is_printable(cp: u32) -\u003e bool;\n}\n\npub mod identifier {\n    pub fn is_xid_start(cp: u32) -\u003e bool;\n    pub fn is_xid_continue(cp: u32) -\u003e bool;\n    pub fn is_python_identifier_start(cp: u32) -\u003e bool;\n    pub fn is_python_identifier_continue(cp: u32) -\u003e bool;\n}\n\npub mod regex {\n    pub fn is_word(cp: u32) -\u003e bool;\n    pub fn is_digit(cp: u32) -\u003e bool;\n    pub fn is_space(cp: u32) -\u003e bool;\n}\n\npub mod case {\n    pub fn to_lowercase(cp: u32) -\u003e CaseMapping;\n    pub fn to_uppercase(cp: u32) -\u003e CaseMapping;\n    pub fn to_titlecase(cp: u32) -\u003e CaseMapping;\n    pub fn casefold(cp: u32) -\u003e CaseMapping;\n}\n\npub mod normalize {\n    pub fn nfc\u003cI: IntoIterator\u003cItem = u32\u003e\u003e(input: I) -\u003e Normalized;\n    pub fn nfd\u003cI: IntoIterator\u003cItem = u32\u003e\u003e(input: I) -\u003e Normalized;\n    pub fn nfkc\u003cI: IntoIterator\u003cItem = u32\u003e\u003e(input: I) -\u003e Normalized;\n    pub fn nfkd\u003cI: IntoIterator\u003cItem = u32\u003e\u003e(input: I) -\u003e Normalized;\n    pub fn is_normalized_nfc\u003cI: IntoIterator\u003cItem = u32\u003e\u003e(input: I) -\u003e bool;\n}\n\npub mod data {\n    pub fn category(cp: u32) -\u003e GeneralCategory;\n    pub fn bidirectional(cp: u32) -\u003e BidiClass;\n    pub fn combining(cp: u32) -\u003e u8;\n    pub fn east_asian_width(cp: u32) -\u003e EastAsianWidth;\n    pub fn mirrored(cp: u32) -\u003e bool;\n    pub fn decomposition(cp: u32) -\u003e Option\u003cDecomposition\u003e;\n    pub fn decimal(cp: u32) -\u003e Option\u003cu8\u003e;\n    pub fn digit(cp: u32) -\u003e Option\u003cu8\u003e;\n    pub fn numeric(cp: u32) -\u003e Option\u003cNumericValue\u003e;\n    pub fn name(cp: u32) -\u003e Option\u003c\u0026'static str\u003e;\n    pub fn lookup(name: \u0026str) -\u003e Option\u003cu32\u003e;\n}\n\nThe exact types can change, but the important part is the boundary: low-level Unicode\nsemantics here, string algorithms elsewhere.\n\nCompatibility Model\n\nThe crate should define compatibility against a specific CPython line, for example:\n\n- CPython 3.14 Unicode semantics\n- version-pinned generated tables\n- explicit regeneration workflow when upgrading CPython\n\nThat matters because “Unicode-correct” is not enough here. The target is “CPython-\ncompatible.”\n\nWhy This Is Better Than Ad Hoc Fixes\n\n- str, re, parser, and unicodedata stop drifting apart.\n- There is one authoritative source for Unicode behavior.\n- Compatibility work becomes table-driven instead of patch-driven.\n- Future CPython upgrades become more mechanical and auditable.\n","author":{"url":"https://github.com/youknowone","@type":"Person","name":"youknowone"},"datePublished":"2026-04-05T07:10:53.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/7560/RustPython/issues/7560"}