René's URL Explorer Experiment


Title: File tokenizer obscures encoding-related exceptions · Issue #151461 · python/cpython · GitHub

Open Graph Title: File tokenizer obscures encoding-related exceptions · Issue #151461 · python/cpython

X Title: File tokenizer obscures encoding-related exceptions · Issue #151461 · python/cpython

Description: Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem: . Importing the same file, or executing it via -m, reports...

Open Graph Description: Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem: . Importing the sa...

X Description: Bug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem: <encoding>. Importing ...

Opengraph URL: https://github.com/python/cpython/issues/151461

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"File tokenizer obscures encoding-related exceptions","articleBody":"# Bug report\n\n### Bug description:\n\nWhen a source file declares an invalid or incompatible encoding, direct execution currently reports a generic `SyntaxError: encoding problem: \u003cencoding>`. Importing the same file, or executing it via `-m`, reports the underlying cause, such as unknown encoding: ... or the concrete codec decode error. This makes direct execution less informative and inconsistent with the import/runpy path.\n\n\n### Unknown encoding: current vs expected\n\n```diff\n❯ cat t.py\n# coding: dict-unpacking-at-home\nprint('Hi') \n\n❯ ./python.exe t.py\n-SyntaxError: encoding problem: dict-unpacking-at-home\n+  File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\n+SyntaxError: unknown encoding: dict-unpacking-at-home\n\n❯ cat t2.py\nimport t\n\n❯ ./python.exe t2.py\nTraceback (most recent call last):\n  File \"/Users/bartosz.slawecki/Python/cpython/t2.py\", line 1, in \u003cmodule>\n    import t\n  File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: unknown encoding: dict-unpacking-at-home\n\n❯ ./python.exe -m t\nTraceback (most recent call last):\n  File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 192, in _run_module_as_main\n    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)\n                               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^\n  File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 163, in _get_module_details\n    code = loader.get_code(mod_name)\n  File \"\u003cfrozen importlib._bootstrap_external>\", line 877, in get_code\n  File \"\u003cfrozen importlib._bootstrap_external>\", line 806, in source_to_code\n  File \"\u003cfrozen importlib._bootstrap>\", line 543, in _call_with_frames_removed\n  File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: unknown encoding: dict-unpacking-at-home\n```\n\n### Encoding error: current vs expected\n\n```diff\n❯ cat t.py\n# coding: shift-jis\n\nlubię Pythona i jego team\n\n❯ ./python.exe t.py\n-SyntaxError: encoding problem: shift-jis\n+  File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\n+    # coding: shift-jis\n+SyntaxError: 'shift_jis' codec can't decode byte 0x99 in position 8: illegal multibyte sequence\n\n❯ cat t2.py\nimport t\n\n❯ ./python.exe t2.py\nTraceback (most recent call last):\n  File \"/Users/bartosz.slawecki/Python/cpython/t2.py\", line 1, in \u003cmodule>\n    import t\n  File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: 'shift_jis' codec can't decode byte 0x99 in position 26: illegal multibyte sequence\n\n❯ ./python.exe -m t\nTraceback (most recent call last):\n  File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 192, in _run_module_as_main\n    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)\n                               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^\n  File \"/Users/bartosz.slawecki/Python/cpython/Lib/runpy.py\", line 163, in _get_module_details\n    code = loader.get_code(mod_name)\n  File \"\u003cfrozen importlib._bootstrap_external>\", line 877, in get_code\n  File \"\u003cfrozen importlib._bootstrap_external>\", line 806, in source_to_code\n  File \"\u003cfrozen importlib._bootstrap>\", line 543, in _call_with_frames_removed\n  File \"/Users/bartosz.slawecki/Python/cpython/t.py\", line 0\nSyntaxError: 'shift_jis' codec can't decode byte 0x99 in position 26: illegal multibyte sequence\n```\n\n## Investigation\n\nDirect execution uses the file tokenizer, which reads and decodes the file incrementally. Importing uses importlib to read the source into memory and then compiles the full string, so it goes through the string tokenizer.\n\nBoth tokenizers interpret [the declared source code encoding](https://docs.python.org/3/tutorial/interpreter.html#source-code-encoding) differently when invoked via `set_readline()` in this line of `_PyTokenizer_check_coding_spec`:\n\nhttps://github.com/python/cpython/blob/6679ac07d881f6e0ce30b7cc28b5671eafa20d9d/Parser/tokenizer/helpers.c#L420-L426\n\n- If `set_readline` is `fp_setreadl` (file tokenizer), it creates a file reader with `builtins.open` and uses that new reader to read the rest of the file past the encoding declaration line.\n\n  Only `fp_setreadl` can allow `SyntaxError: encoding problem` from line 422 to bubble up, *unless* the erroneous bytes aren't translated by the new reader; either because of (1) no newline at the end of file, which deferred the translation in the incremental newline decoder, or (2) because the erroneous bytes are in a future chunk that was not yet read from the disk (around 8kB after the encoding spec line).\n\n- If `set_readline` is `buf_setreadl` (string tokenizer), it just sets `tok->enc` and always succeeds. The encoding error only happens later in `_PyTokenizer_translate_into_utf8` which bails out _before_ the parser is created; it goes through `_PyPegen_raise_tokenizer_init_error` in `_PyPegen_run_parser_from_string` to ensure that it is a `SyntaxError`. This explains why the vague exception never occurred when the code was imported / ran via runpy.\n\nIn the file tokenizer path, `_PyTokenizer_check_coding_spec()` overwrites the concrete exception raised by `set_readline()` with the generic `SyntaxError: encoding problem: \u003cencoding>`. The user is no longer given the information of whether there was codec lookup error or a file decoding error.\n\nThe linked PR fixes this by dropping the `encoding problem` exception entirely and replacing it with the `_PyPegen_raise_tokenizer_init_error(tok->filename)` call since all `set_readline` failure paths provide an active exception. This way we have a consistent experience in both tokenizers.\n\nEDIT: I moved the tokenizer-init error wrapper from pegen to tokenizer helpers. This used to live in pegen because only parser entry points needed it. The file tokenizer now needs the same normalization at the point where `set_readline()` fails, so the helper was moved to tokenizer helpers rather than making tokenizer code depend on pegen internals. I'm happy to adjust the layering if there is a better home for this helper.\n\ncc @lysnikolaou @pablogsal @serhiy-storchaka\n\n### CPython versions tested on:\n\n3.16, main\n\n### Operating systems tested on:\n\nmacOS\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-151462\n* gh-151470\n* gh-151474\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/johnslavik","@type":"Person","name":"johnslavik"},"datePublished":"2026-06-14T03:01:58Z","url":"https://github.com/python/cpython/issues/151461"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:f340efcc-5f00-f18c-586c-d402b4272bfd
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB934:1148D3:3AA73AD:4ACD528:6B0207CD
html-safe-nonceb3ef6bda17c4ee09816a8b8c972830186fdf7ba333b6b5c610335cdb96b459dc
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCOTM0OjExNDhEMzozQUE3M0FEOjRBQ0Q1Mjg6NkIwMjA3Q0QiLCJ2aXNpdG9yX2lkIjoiODg1NjMwNTAyOTUxOTM3ODM4MiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac9e529cbfb69c98bf9554f7da6f2a86d4bb9bf929e613c274afeb425139af4209
hovercard-subject-tagissue:4657738089
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/151461/issue_layout
twitter:imagehttps://opengraph.githubassets.com/134d1db0857ae62651ffac2f489c4ff51f2b139562399f34c9a1d65fb14cbe02/python/cpython/issues/151461
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/134d1db0857ae62651ffac2f489c4ff51f2b139562399f34c9a1d65fb14cbe02/python/cpython/issues/151461
og:image:altBug report Bug description: When a source file declares an invalid or incompatible encoding, direct execution currently reports a generic SyntaxError: encoding problem: . Importing the sa...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamejohnslavik
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/151461#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F151461
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F151461
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/151461
Reloadhttps://github.com/python/cpython/issues/151461
Reloadhttps://github.com/python/cpython/issues/151461
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/151461
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/151461
File tokenizer obscures encoding-related exceptionshttps://github.com/python/cpython/issues/151461#top
https://github.com/johnslavik
3.14bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.14%22
3.15bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.15%22
3.16new features, bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.16%22
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com/johnslavik
johnslavikhttps://github.com/johnslavik
on Jun 14, 2026https://github.com/python/cpython/issues/151461#issue-4657738089
the declared source code encodinghttps://docs.python.org/3/tutorial/interpreter.html#source-code-encoding
cpython/Parser/tokenizer/helpers.chttps://github.com/python/cpython/blob/6679ac07d881f6e0ce30b7cc28b5671eafa20d9d/Parser/tokenizer/helpers.c#L420-L426
6679ac0https://github.com/python/cpython/commit/6679ac07d881f6e0ce30b7cc28b5671eafa20d9d
@lysnikolaouhttps://github.com/lysnikolaou
@pablogsalhttps://github.com/pablogsal
@serhiy-storchakahttps://github.com/serhiy-storchaka
gh-151461: Fix encoding-related exception handling in file tokenizer #151462https://github.com/python/cpython/pull/151462
[3.15] gh-151461: Fix encoding-related exception handling in file tokenizer (GH-151462) #151470https://github.com/python/cpython/pull/151470
[3.14] gh-151461: Fix encoding-related exception handling in file tokenizer (GH-151462) #151474https://github.com/python/cpython/pull/151474
johnslavikhttps://github.com/johnslavik
on Jun 14, 2026https://github.com/python/cpython/issues/151461#event-26718974903
johnslavikhttps://github.com/johnslavik
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
3.14bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.14%22
3.15bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.15%22
3.16new features, bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.16%22
on Jun 14, 2026https://github.com/python/cpython/issues/151461#event-26718975153
serhiy-storchakahttps://github.com/serhiy-storchaka
on Jun 14, 2026https://github.com/python/cpython/issues/151461#event-26725345747
gh-151461: Fix encoding-related exception handling in file tokenizer (G…https://github.com/python/cpython/commit/f1a5f68e3761e010ccd4dda1342500c5ae40bbc4
gh-151461https://github.com/python/cpython/issues/151461
G…https://github.com/python/cpython/pull/151462
f1a5f68https://github.com/python/cpython/commit/f1a5f68e3761e010ccd4dda1342500c5ae40bbc4
[3.15] gh-151461: Fix encoding-related exception handling in file tok…https://github.com/python/cpython/commit/89e178d78cb297755ddb58b55e42e74ed31aa810
gh-151461https://github.com/python/cpython/issues/151461
89e178dhttps://github.com/python/cpython/commit/89e178d78cb297755ddb58b55e42e74ed31aa810
[3.14] gh-151461: Fix encoding-related exception handling in file tok…https://github.com/python/cpython/commit/3ad080ab7e742b2883ac4446d87d8cc22ff1483c
gh-151461https://github.com/python/cpython/issues/151461
3ad080ahttps://github.com/python/cpython/commit/3ad080ab7e742b2883ac4446d87d8cc22ff1483c
johnslavikhttps://github.com/johnslavik
completedhttps://github.com/python/cpython/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
on Jun 14, 2026https://github.com/python/cpython/issues/151461#event-26725914015
philthompson10https://github.com/philthompson10
on Jun 17, 2026https://github.com/python/cpython/issues/151461#event-26873465229
pythongh-151461: Fix encoding-related exception handling in file toke…https://github.com/philthompson10/cpython/commit/20525d35cf4cf0ff68693d43d6dda5723c476b5b
pythongh-151461https://github.com/python/cpython/issues/151461
20525d3https://github.com/philthompson10/cpython/commit/20525d35cf4cf0ff68693d43d6dda5723c476b5b
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/151461
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/151461
johnslavikhttps://github.com/johnslavik
3.14bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.14%22
3.15bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.15%22
3.16new features, bugs and security fixeshttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%223.16%22
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.