René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Unicode characters ≥ 0x10000 cannot be inputted/behaves unusually at the REPL terminal.","articleBody":"# Bug report\n\n### Bug description:\n\nWhen the machine locale is set to UTF-8, when inputting a Unicode character ≥ 0x10000:\nIn CPython 3.13.5:\nhttps://github.com/user-attachments/assets/7777b063-76fe-4929-b854-cae7d61807d2\nIn Cpython 3.14.0b4:\n```\n\u003e\u003e\u003e Traceback (most recent call last):\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\readline.py\", line 394, in multiline_input\n    return reader.readline()\n           ~~~~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 748, in readline\n    self.handle1()\n    ~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 731, in handle1\n    self.do_cmd(cmd)\n    ~~~~~~~~~~~^^^^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 661, in do_cmd\n    self.refresh()\n    ~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 638, in refresh\n    self.screen = self.calc_screen()\n                  ~~~~~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\completing_reader.py\", line 261, in calc_screen\n    screen = super().calc_screen()\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 315, in calc_screen\n    colors = list(gen_colors(self.get_unicode()))\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\utils.py\", line 108, in gen_colors\n    for color in gen_colors_from_token_stream(gen, line_lengths):\n                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\utils.py\", line 168, in gen_colors_from_token_stream\n    for prev_token, token, next_token in token_window:\n                                         ^^^^^^^^^^^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\utils.py\", line 363, in prev_next_window\n    window = deque((None, next(iterator)), maxlen=3)\n                          ~~~~^^^^^^^^^^\n  File \"*\\Python\\Python314\\Lib\\tokenize.py\", line 582, in _generate_tokens_from_c_tokenizer\n    for info in it:\n                ^^\nUnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed\n\u003e\u003e\u003e Traceback (most recent call last):\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\readline.py\", line 394, in multiline_input\n    return reader.readline()\n           ~~~~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 748, in readline\n    self.handle1()\n    ~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 731, in handle1\n    self.do_cmd(cmd)\n    ~~~~~~~~~~~^^^^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 661, in do_cmd\n    self.refresh()\n    ~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 638, in refresh\n    self.screen = self.calc_screen()\n                  ~~~~~~~~~~~~~~~~^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\completing_reader.py\", line 261, in calc_screen\n    screen = super().calc_screen()\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\reader.py\", line 315, in calc_screen\n    colors = list(gen_colors(self.get_unicode()))\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\utils.py\", line 108, in gen_colors\n    for color in gen_colors_from_token_stream(gen, line_lengths):\n                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\utils.py\", line 168, in gen_colors_from_token_stream\n    for prev_token, token, next_token in token_window:\n                                         ^^^^^^^^^^^^\n  File \"*\\Python\\Python314\\Lib\\_pyrepl\\utils.py\", line 363, in prev_next_window\n    window = deque((None, next(iterator)), maxlen=3)\n                          ~~~~^^^^^^^^^^\n  File \"*\\Python\\Python314\\Lib\\tokenize.py\", line 582, in _generate_tokens_from_c_tokenizer\n    for info in it:\n                ^^\nUnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed\n```\nTwo surrogates were \"inputted\" and so two `UnicodeEncodeError`s.\n\n### CPython versions tested on:\n\n3.13, 3.14\n\n### Operating systems tested on:\n\nWindows\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-136639\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/haydenwong7bm","@type":"Person","name":"haydenwong7bm"},"datePublished":"2025-07-12T17:31:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/136595/cpython/issues/136595"}