Title: File tokenizer obscures encoding-related exceptions · Issue #151461 · python/cpython · GitHub
Open Graph Title: File tokenizer obscures encoding-related exceptions · Issue #151461 · python/cpython
X Title: File tokenizer obscures encoding-related exceptions · Issue #151461 · python/cpython
Description: Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem:
Open Graph Description: Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem:
X Description: Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem: <encoding>. Importing ...
Opengraph URL: https://github.com/python/cpython/issues/151461
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"File tokenizer obscures encoding-related exceptions","articleBody":"# Bug report\n\n### Bug description:\n\nWhen a source file declares an invalid or incompatible encoding, direct execution currently reports a generic `SyntaxError: encoding problem: \u003cencoding>`. Importing the same file, or executing it via `-m`, reports the underlying cause, such as unknown encoding: ... or the concrete codec decode error. This makes direct execution less informative and inconsistent with the import/runpy path.\n\n\n### Unknown encoding: current vs expected\n\n```diff\n❯ cat t.py\n# coding: dict-unpacking-at-home\nprint('Hi') \n\n❯ ./python.exe t.py\n-SyntaxError: encoding problem: dict-unpacking-at-home\n+ File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\n+SyntaxError: unknown encoding: dict-unpacking-at-home\n\n❯ cat t2.py\nimport t\n\n❯ ./python.exe t2.py\nTraceback (most recent call last):\n File \"/Users/bartosz.slawecki/Python/cpython/t2.py\", line 1, in \u003cmodule>\n import t\n File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: unknown encoding: dict-unpacking-at-home\n\n❯ ./python.exe -m t\nTraceback (most recent call last):\n File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 192, in _run_module_as_main\n mod_name, mod_spec, code = _get_module_details(mod_name, _Error)\n ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^\n File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 163, in _get_module_details\n code = loader.get_code(mod_name)\n File \"\u003cfrozen importlib._bootstrap_external>\", line 877, in get_code\n File \"\u003cfrozen importlib._bootstrap_external>\", line 806, in source_to_code\n File \"\u003cfrozen importlib._bootstrap>\", line 543, in _call_with_frames_removed\n File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: unknown encoding: dict-unpacking-at-home\n```\n\n### Encoding error: current vs expected\n\n```diff\n❯ cat t.py\n# coding: shift-jis\n\nlubię Pythona i jego team\n\n❯ ./python.exe t.py\n-SyntaxError: encoding problem: shift-jis\n+ File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\n+ # coding: shift-jis\n+SyntaxError: 'shift_jis' codec can't decode byte 0x99 in position 8: illegal multibyte sequence\n\n❯ cat t2.py\nimport t\n\n❯ ./python.exe t2.py\nTraceback (most recent call last):\n File \"/Users/bartosz.slawecki/Python/cpython/t2.py\", line 1, in \u003cmodule>\n import t\n File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: 'shift_jis' codec can't decode byte 0x99 in position 26: illegal multibyte sequence\n\n❯ ./python.exe -m t\nTraceback (most recent call last):\n File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 192, in _run_module_as_main\n mod_name, mod_spec, code = _get_module_details(mod_name, _Error)\n ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^\n File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 163, in _get_module_details\n code = loader.get_code(mod_name)\n File \"\u003cfrozen importlib._bootstrap_external>\", line 877, in get_code\n File \"\u003cfrozen importlib._bootstrap_external>\", line 806, in source_to_code\n File \"\u003cfrozen importlib._bootstrap>\", line 543, in _call_with_frames_removed\n File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: 'shift_jis' codec can't decode byte 0x99 in position 26: illegal multibyte sequence\n```\n\n## Investigation\n\nDirect execution uses the file tokenizer, which reads and decodes the file incrementally. Importing uses importlib to read the source into memory and then compiles the full string, so it goes through the string tokenizer.\n\nBoth tokenizers interpret [the declared source code encoding](https://docs.python.org/3/tutorial/interpreter.html#source-code-encoding) differently when invoked via `set_readline()` in this line of `_PyTokenizer_check_coding_spec`:\n\nhttps://github.com/python/cpython/blob/6679ac07d881f6e0ce30b7cc28b5671eafa20d9d/Parser/tokenizer/helpers.c#L420-L426\n\n- If `set_readline` is `fp_setreadl` (file tokenizer), it creates a file reader with `builtins.open` and uses that new reader to read the rest of the file past the encoding declaration line.\n\n Only `fp_setreadl` can allow `SyntaxError: encoding problem` from line 422 to bubble up, *unless* the erroneous bytes aren't translated by the new reader; either because of (1) no newline at the end of file, which deferred the translation in the incremental newline decoder, or (2) because the erroneous bytes are in a future chunk that was not yet read from the disk (around 8kB after the encoding spec line).\n\n- If `set_readline` is `buf_setreadl` (string tokenizer), it just sets `tok->enc` and always succeeds. The encoding error only happens later in `_PyTokenizer_translate_into_utf8` which bails out _before_ the parser is created; it goes through `_PyPegen_raise_tokenizer_init_error` in `_PyPegen_run_parser_from_string` to ensure that it is a `SyntaxError`. This explains why the vague exception never occurred when the code was imported / ran via runpy.\n\nIn the file tokenizer path, `_PyTokenizer_check_coding_spec()` overwrites the concrete exception raised by `set_readline()` with the generic `SyntaxError: encoding problem: \u003cencoding>`. The user is no longer given the information of whether there was codec lookup error or a file decoding error.\n\nThe linked PR fixes this by dropping the `encoding problem` exception entirely and replacing it with the `_PyPegen_raise_tokenizer_init_error(tok->filename)` call since all `set_readline` failure paths provide an active exception. This way we have a consistent experience in both tokenizers.\n\nEDIT: I moved the tokenizer-init error wrapper from pegen to tokenizer helpers. This used to live in pegen because only parser entry points needed it. The file tokenizer now needs the same normalization at the point where `set_readline()` fails, so the helper was moved to tokenizer helpers rather than making tokenizer code depend on pegen internals. I'm happy to adjust the layering if there is a better home for this helper.\n\ncc @lysnikolaou @pablogsal @serhiy-storchaka\n\n### CPython versions tested on:\n\n3.16, main\n\n### Operating systems tested on:\n\nmacOS\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-151462\n* gh-151470\n* gh-151474\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/johnslavik","@type":"Person","name":"johnslavik"},"datePublished":"2026-06-14T03:01:58Z","url":"https://github.com/python/cpython/issues/151461"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:f340efcc-5f00-f18c-586c-d402b4272bfd |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B934:1148D3:3AA73AD:4ACD528:6B0207CD |
| html-safe-nonce | b3ef6bda17c4ee09816a8b8c972830186fdf7ba333b6b5c610335cdb96b459dc |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCOTM0OjExNDhEMzozQUE3M0FEOjRBQ0Q1Mjg6NkIwMjA3Q0QiLCJ2aXNpdG9yX2lkIjoiODg1NjMwNTAyOTUxOTM3ODM4MiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 9e529cbfb69c98bf9554f7da6f2a86d4bb9bf929e613c274afeb425139af4209 |
| hovercard-subject-tag | issue:4657738089 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/151461/issue_layout |
| twitter:image | https://opengraph.githubassets.com/134d1db0857ae62651ffac2f489c4ff51f2b139562399f34c9a1d65fb14cbe02/python/cpython/issues/151461 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/134d1db0857ae62651ffac2f489c4ff51f2b139562399f34c9a1d65fb14cbe02/python/cpython/issues/151461 |
| og:image:alt | Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem: |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | johnslavik |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width