Title: fix: Python-Rust combining char diff in isalnum by joshuamegnauth54 · Pull Request #7612 · RustPython/RustPython · GitHub
Open Graph Title: fix: Python-Rust combining char diff in isalnum by joshuamegnauth54 · Pull Request #7612 · RustPython/RustPython
X Title: fix: Python-Rust combining char diff in isalnum by joshuamegnauth54 · Pull Request #7612 · RustPython/RustPython
Description: Closes: #7518 Rust and Python differ on alphanumeric characters. Rust follows the Unicode standard closer than Python. This means that is_alphanumeric (char function in Rust) is different from isalnum (Python). To fix the discrepancy, RustPython needs to mimic Python by rejecting certain characters. Some classes of combining characters count as alphanumeric in Rust but not Python. Combining characters are accent marks that are combined with other characters to create a single grapheme. It's possible that this PR is not exhaustive. I fixed the combining character issue BUT I don't know the full range of discrepancies. This doesn't actually fix #7518, but it fixes a similar issue based on that report. Actually, CodeRabbit pointed me in the right direction so now it does fix #7518. assert!('\u{006e}'.is_alphanumeric()); assert!(!'\u{0303}'.is_alphanumeric()); assert!('\u{00f1}'.is_alphanumeric()); assert!('\u{0345}'.is_alphanumeric()); for raw in 0x0363..=0x036f { let c = char::from_u32(raw).unwrap(); assert!(c.is_alphanumeric()); } assert '\u006e'.isalnum() assert not '\u0303'.isalnum() assert '\u00f1'.isalnum() assert not '\u0345'.isalnum() assert not '\u0363'.isalnum() for raw in range(0x0363, 0x036f): assert not chr(raw).isalnum() ^Note the differences which are accounted for by the new tests. Summary by CodeRabbit Bug Fixes Corrected str.isalnum() so standalone Unicode combining marks are not treated as alphanumeric, while base letters remain valid. Tests Expanded tests for str.isalnum() covering combining-character cases. Added regex test ensuring standalone combining marks are not matched as word characters.
Open Graph Description: Closes: #7518 Rust and Python differ on alphanumeric characters. Rust follows the Unicode standard closer than Python. This means that is_alphanumeric (char function in Rust) is different from isal...
X Description: Closes: #7518 Rust and Python differ on alphanumeric characters. Rust follows the Unicode standard closer than Python. This means that is_alphanumeric (char function in Rust) is different from isal...
Opengraph URL: https://github.com/RustPython/RustPython/pull/7612
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:10705c52-c47c-e82a-c9bc-ee59579fd49e |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C126:37C180:34E5352:4921BE9:6A60DA6E |
| html-safe-nonce | 1a3bc8f5f4b3633d95ec1897216177a9d63301acfd8508f3ca146d4d9d5a2735 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMTI2OjM3QzE4MDozNEU1MzUyOjQ5MjFCRTk6NkE2MERBNkUiLCJ2aXNpdG9yX2lkIjoiMTU5NTA0NjA2NjkwNzA0NDQ2MiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 32b3aa0a3de02399d0035e7aa050301f0d7bb8db240433d3363dd3f672af4adf |
| hovercard-subject-tag | pull_request:3531209317 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/7612/files |
| twitter:image | https://avatars.githubusercontent.com/u/48846352?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/48846352?s=400&v=4 |
| og:image:alt | Closes: #7518 Rust and Python differ on alphanumeric characters. Rust follows the Unicode standard closer than Python. This means that is_alphanumeric (char function in Rust) is different from isal... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 8302bdfafed4cf34ac9c1d3593108fafa53bc3f86b893907e38fa6bdd74361ac |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | c3c2ff0c76b34772c83c9498a08cefee28f53da0 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width