Title: Fix most of `unicodedata`, unmask multiple tests, and remove `ucd` by joshuamegnauth54 · Pull Request #7947 · RustPython/RustPython · GitHub
Open Graph Title: Fix most of `unicodedata`, unmask multiple tests, and remove `ucd` by joshuamegnauth54 · Pull Request #7947 · RustPython/RustPython
X Title: Fix most of `unicodedata`, unmask multiple tests, and remove `ucd` by joshuamegnauth54 · Pull Request #7947 · RustPython/RustPython
Description: Python bundles an old version of Unicode for compatibility. RustPython tries to mimic supporting that old version by checking the version of individual chars. This is a problem for a few reasons. The first is that the age check adds an additional hit per each char lookup in Unicode data. The check is outdated because the unic-ucd-age crate is several versions behind the current Unicode version. The check rejects valid chars because of the version differences. The check is subtly wrong because it returns properties for Unicode 16.0.0 for Unicode 3.2.0 while checking against a Unicode 10.0.0 database. Unfortunately, there isn't a crate that can help us here. icu4x targets modern Unicode versions. Writing a data provider for icu4x for Unicode 3.2.0 is a lot of work for a legacy path. I opted to parse the Unicode 3.2.0 data myself but to skip icu4x (mostly) to instead write small lookup tables. As of this commit, Unicode names is still wrong for 3.2.0. Luckily, the crate RustPython uses is fast and robust for modern Unicode. Summary by CodeRabbit New Features Vendored Unicode 3.2.0 plus new “latest” compatibility datasets for broader coverage. Improvements Version-aware Unicode lookups (category, bidi, width, decomposition) with reduced allocations. Numeric APIs (digit/decimal/numeric) clarified to return normalized numeric values where applicable. Bug Fixes Applied normalization/decomposition corrections for historic compatibility. Documentation Added guidance on vendored Unicode data and update policy. Tests Updated Unicode tests to validate decompositions and versioned behavior.
Open Graph Description: Python bundles an old version of Unicode for compatibility. RustPython tries to mimic supporting that old version by checking the version of individual chars. This is a problem for a few reasons. T...
X Description: Python bundles an old version of Unicode for compatibility. RustPython tries to mimic supporting that old version by checking the version of individual chars. This is a problem for a few reasons. T...
Opengraph URL: https://github.com/RustPython/RustPython/pull/7947
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:805c7397-c918-b144-0548-f1ee415e8548 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D3A8:264DE9:65BC6:86C2F:6A610C8F |
| html-safe-nonce | c852f524f317c2f00f073a238bd48bd0cdd3454caaf25b5c1bac4e20a1ddadc0 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEM0E4OjI2NERFOTo2NUJDNjo4NkMyRjo2QTYxMEM4RiIsInZpc2l0b3JfaWQiOiI2OTM3ODMwMjc4OTc3MDk2ODQ3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | ebb66cf85e79e71bf5df6351e1668cfb799c280bb70540a426e53669a8d67886 |
| hovercard-subject-tag | pull_request:3726106193 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/7947/files |
| twitter:image | https://avatars.githubusercontent.com/u/48846352?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/48846352?s=400&v=4 |
| og:image:alt | Python bundles an old version of Unicode for compatibility. RustPython tries to mimic supporting that old version by checking the version of individual chars. This is a problem for a few reasons. T... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 86be97fd69ac233b46e43a5eed04ceac6498abe1efaf1af3b9573eb216019c05 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | f80280886562a52fe1f6b9cab9f9c1dc3b9a78a2 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width