René's URL Explorer Experiment


Title: Wrong `SyntaxError.offset` for "Non-UTF-8 code starting with ..." when a non-ASCII character precedes the invalid byte · Issue #157378 · python/cpython · GitHub

Open Graph Title: Wrong `SyntaxError.offset` for "Non-UTF-8 code starting with ..." when a non-ASCII character precedes the invalid byte · Issue #157378 · python/cpython

X Title: Wrong `SyntaxError.offset` for "Non-UTF-8 code starting with ..." when a non-ASCII character precedes the invalid byte · Issue #157378 · python/cpython

Description: Bug report Bug description: SyntaxError.offset and SyntaxError.end_offset for the error "Non-UTF-8 code starting with ..." are too small when a valid multi-byte character precedes the invalid byte on the same line. Reproduction compile(b...

Open Graph Description: Bug report Bug description: SyntaxError.offset and SyntaxError.end_offset for the error "Non-UTF-8 code starting with ..." are too small when a valid multi-byte character precedes the invalid byte ...

X Description: Bug report Bug description: SyntaxError.offset and SyntaxError.end_offset for the error "Non-UTF-8 code starting with ..." are too small when a valid multi-byte character precedes the inv...

Opengraph URL: https://github.com/python/cpython/issues/157378

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Wrong `SyntaxError.offset` for \"Non-UTF-8 code starting with ...\" when a non-ASCII character precedes the invalid byte","articleBody":"# Bug report\n\n### Bug description:\n\n`SyntaxError.offset` and `SyntaxError.end_offset` for the error\n\"Non-UTF-8 code starting with ...\" are too small when a valid multi-byte\ncharacter precedes the invalid byte on the same line.\n\n## Reproduction\n\n```python\ncompile(b'\\xc3\\xa9X\\x80', '\u003cbug>', 'exec')\n```\n\nSame via a file:\n\n```console\n$ printf '\\xc3\\xa9X\\x80' > bug.py\n$ ./python bug.py\n```\n\nThe input is `\"éX\"` followed by the invalid byte `0x80`, with no encoding\ncookie. The invalid byte is the 3rd character of line 1.\n\n## Expected\n\n```console\n  File \"\u003cbug>\", line 1\n    éX\n      ^\nSyntaxError: Non-UTF-8 code starting with '\\x80' on line 1, but no encoding declared; ...\n```\n\n`offset == 3`, `end_offset == 3` (1-based character column).\n\n## Actual\n\n```console\n  File \"\u003cbug>\", line 1\n    éX\n     ^\nSyntaxError: Non-UTF-8 code starting with '\\x80' on line 1, but no encoding declared; ...\n```\n\n`offset == 2`, `end_offset == 2`.\n\nObserved values:\n\n| build                       | offset | end_offset |\n|-----------------------------|--------|------------|\n| main (575fe3914f4)          | 2      | 2          |\n| 3.14.7                      | 3      | 3          |\n| 3.15.0rc2+dev (e325fae3578) | 3      | 3          |\n\nMore inputs on main, all one column short per preceding multi-byte character:\n\n```python\ncompile(b'\\xc3\\xa9abc\\x80', ...)  # offset 4, expected 5\ncompile(b'\\t\\xc3\\xa9X\\x80', ...)  # offset 3, expected 4\ncompile(b'a\\xc3\\xa9b\\x80c', ...)  # offset 3, expected 4\n```\n\n## Cause\n\n`_PyTokenizer_ensure_utf8()` (Parser/tokenizer/helpers.c) computes a 1-based\ncharacter column and passes it to `_PyTokenizer_syntaxerror_known_range()`:\n\n    _PyTokenizer_syntaxerror_known_range(tok,\n            col_offset + 1, col_offset + 1, ...)\n\nSince 59a691361cb (gh-156894, PR #156901), `_syntaxerror_range()` converts its\n`col_offset`/`end_col_offset` arguments from bytes to characters with\n`byte_col_to_char_col()`, so the character column from `ensure_utf8()` is\nconverted a second time.\n\nThe same function is unchanged on current main (b1e7554ac1e).\n\n## Suggested fix\n\nPass byte columns from `ensure_utf8()`, e.g. `(int)(badchar - line_start) + 1`\nfor both arguments, and let `_syntaxerror_range()` do the byte-to-character\nconversion.\n\n## Tests\n\n`Lib/test/test_source_encoding.py` only checks the message text for this error;\nno test checks `offset`/`end_offset`, so this case is not covered.\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-157412\n* gh-157489\n* gh-158123\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/ashm-dev","@type":"Person","name":"ashm-dev"},"datePublished":"2026-09-12T20:12:35Z","url":"https://github.com/python/cpython/issues/157378"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:282ac904-b79f-a804-0767-8007687077e6
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idAAD0:1EFF:8C4C2E:BAF62C:6B01FA16
html-safe-nonce1e5d916a924f4b9c354648da5a2daa5bddcb2b8b48236f3032f97440c7303f59
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQUQwOjFFRkY6OEM0QzJFOkJBRjYyQzo2QjAxRkExNiIsInZpc2l0b3JfaWQiOiIzMjkzMjA1ODcwMDkxMDQxMzAyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac9d6c62734f9b2eb0be70561dee22e97a75cd31bcec21c4df7e8b610ca974abf0
hovercard-subject-tagissue:5435737918
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/157378/issue_layout
twitter:imagehttps://opengraph.githubassets.com/8043d6636dad4517c91fe2ae6bfe708452946db2f9b3cde6c6015528c6ea99a5/python/cpython/issues/157378
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/8043d6636dad4517c91fe2ae6bfe708452946db2f9b3cde6c6015528c6ea99a5/python/cpython/issues/157378
og:image:altBug report Bug description: SyntaxError.offset and SyntaxError.end_offset for the error "Non-UTF-8 code starting with ..." are too small when a valid multi-byte character precedes the invalid byte ...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameashm-dev
hostnamegithub.com
expected-hostnamegithub.com
None87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release423dbb54c5e7bfecc838eb387c1435eb3d0e760d
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/157378#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F157378
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F157378
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/157378
Reloadhttps://github.com/python/cpython/issues/157378
Reloadhttps://github.com/python/cpython/issues/157378
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/157378
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/157378
Wrong SyntaxError.offset for "Non-UTF-8 code starting with ..." when a non-ASCII character precedes the invalid bytehttps://github.com/python/cpython/issues/157378#top
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com/ashm-dev
ashm-devhttps://github.com/ashm-dev
on Sep 12, 2026https://github.com/python/cpython/issues/157378#issue-5435737918
575fe39https://github.com/python/cpython/commit/575fe3914f466e684f13aed1cdd68347f8746944
e325faehttps://github.com/python/cpython/commit/e325fae357821165d30fcca10ad4941d3c1e5201
59a6913https://github.com/python/cpython/commit/59a691361cb7b1d4fde3b92f98a61490381c62c9
gh-156894https://github.com/python/cpython/issues/156894
#156901https://github.com/python/cpython/pull/156901
b1e7554https://github.com/python/cpython/commit/b1e7554ac1e4268eddbb6d9dd7fad512c44d9256
gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error #157412https://github.com/python/cpython/pull/157412
[3.14] gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (GH-157412) #157489https://github.com/python/cpython/pull/157489
[3.15] gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (GH-157412) #158123https://github.com/python/cpython/pull/158123
ashm-devhttps://github.com/ashm-dev
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
on Sep 12, 2026https://github.com/python/cpython/issues/157378#event-31031993659
picnixzhttps://github.com/picnixz
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
on Sep 12, 2026https://github.com/python/cpython/issues/157378#event-31032066416
ashm-devhttps://github.com/ashm-dev
on Sep 14, 2026https://github.com/python/cpython/issues/157378#event-31090937257
Merge branch 'main' into pythongh-157378https://github.com/ashm-dev/cpython/commit/7af3a5b16800d54a52f01829f8cfbb2c34163dab
pythongh-157378https://github.com/python/cpython/issues/157378
7af3a5bhttps://github.com/ashm-dev/cpython/commit/7af3a5b16800d54a52f01829f8cfbb2c34163dab
pablogsalhttps://github.com/pablogsal
on Sep 14, 2026https://github.com/python/cpython/issues/157378#event-31098797983
gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (#157412)https://github.com/python/cpython/commit/e66bec09b722b8955d34f51f2c4b63793d59f9cf
gh-157378https://github.com/python/cpython/issues/157378
#157412https://github.com/python/cpython/pull/157412
e66bec0https://github.com/python/cpython/commit/e66bec09b722b8955d34f51f2c4b63793d59f9cf
[3.14] gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (G…https://github.com/python/cpython/commit/02d4c3b4347da57ba086f2ab7cdf6c0f56c9deeb
gh-157378https://github.com/python/cpython/issues/157378
G…https://github.com/python/cpython/pull/157412
02d4c3bhttps://github.com/python/cpython/commit/02d4c3b4347da57ba086f2ab7cdf6c0f56c9deeb
pablogsalhttps://github.com/pablogsal
on Sep 24, 2026https://github.com/python/cpython/issues/157378#event-31776833411
[3.15] gh-157378: Fix SyntaxError.offset for "Non-UTF-8 code" error (G…https://github.com/python/cpython/commit/55634044b61fec74af2114ff6390b7c84bd21bbf
gh-157378https://github.com/python/cpython/issues/157378
G…https://github.com/python/cpython/pull/157412
5563404https://github.com/python/cpython/commit/55634044b61fec74af2114ff6390b7c84bd21bbf
hugovkhttps://github.com/hugovk
completedhttps://github.com/python/cpython/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
on Sep 28, 2026https://github.com/python/cpython/issues/157378#event-31991873777
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/157378
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/157378
interpreter-core(Objects, Python, Grammar, and Parser dirs)https://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22interpreter-core%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.