Title: gh-136595: Normalize surrogate pairs in REPL input to fix UnicodeEnco… by vedant713 · Pull Request #136639 · python/cpython · GitHub
Open Graph Title: gh-136595: Normalize surrogate pairs in REPL input to fix UnicodeEnco… by vedant713 · Pull Request #136639 · python/cpython
X Title: gh-136595: Normalize surrogate pairs in REPL input to fix UnicodeEnco… by vedant713 · Pull Request #136639 · python/cpython
Description: The new REPL implementation (_pyrepl) crashes on Windows when the user inputs Unicode characters outside the Basic Multilingual Plane (≥ U+10000), such as emoji (e.g. 🐍). This happens because the Windows input layer provides surrogate pairs (UTF-16 code units) that _pyrepl attempts to process and tokenize directly, leading to unpaired surrogate handling issues. This commit introduces a normalize_surrogates() helper in Reader to explicitly normalize surrogate pairs by encoding to UTF-16 with 'surrogatepass' and decoding back. The get_unicode() method is patched to use this normalization so that any code consuming REPL input (e.g. syntax highlighting via tokenize) receives valid Unicode text. This resolves UnicodeEncodeError crashes in the REPL when typing emoji or other non-BMP characters on Windows. Fixes #136595 Issue: gh-136595
Open Graph Description: The new REPL implementation (_pyrepl) crashes on Windows when the user inputs Unicode characters outside the Basic Multilingual Plane (≥ U+10000), such as emoji (e.g. 🐍). This happens because the W...
X Description: The new REPL implementation (_pyrepl) crashes on Windows when the user inputs Unicode characters outside the Basic Multilingual Plane (≥ U+10000), such as emoji (e.g. 🐍). This happens because the W...
Opengraph URL: https://github.com/python/cpython/pull/136639
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:759521e1-8b1f-3255-7756-862a6efe25e7 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D570:D0C77:30DD0E1:4616CAB:6A569027 |
| html-safe-nonce | d1e4236054dbabdaffd664007957898a75656e02257e0e896f1a4153bfea2639 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTcwOkQwQzc3OjMwREQwRTE6NDYxNkNBQjo2QTU2OTAyNyIsInZpc2l0b3JfaWQiOiI3OTg2NzMyNTY2NTE4NTM4NjMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 219694e3b5f4b5ba1eea61a3333f6c57da444c0d378317c949b66cb17655d7ce |
| hovercard-subject-tag | pull_request:2662761183 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/136639/files |
| twitter:image | https://avatars.githubusercontent.com/u/66007382?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/66007382?s=400&v=4 |
| og:image:alt | The new REPL implementation (_pyrepl) crashes on Windows when the user inputs Unicode characters outside the Basic Multilingual Plane (≥ U+10000), such as emoji (e.g. 🐍). This happens because the W... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 8f21be8503ba8fe6154df415b855c03af52eae3f285bd6ae1fef041dc053da38 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 78af172a48a2056810bb627ffa684003e4f9a5ca |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width