René's URL Explorer Experiment


Title: untokenize() does not round-trip for code containing line breaks (`\` + `\n`) · Issue #125553 · python/cpython · GitHub

Open Graph Title: untokenize() does not round-trip for code containing line breaks (`\` + `\n`) · Issue #125553 · python/cpython

X Title: untokenize() does not round-trip for code containing line breaks (`\` + `\n`) · Issue #125553 · python/cpython

Description: Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.generate_tokens(io.StringIO(source_code).readline)) x = tokenize.untokeniz...

Open Graph Description: Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.generate_tokens(io.StringIO(source...

X Description: Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.gene...

Opengraph URL: https://github.com/python/cpython/issues/125553

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"untokenize() does not round-trip for code containing line breaks (`\\` + `\\n`)","articleBody":"# Bug report\n\n### Bug description:\n\nCode which contains line breaks is not round-trip invariant:\n```python\nimport tokenize, io\n\nsource_code = r\"\"\"\n1 + \\\n    2\n\"\"\"\n\ntokens = list(tokenize.generate_tokens(io.StringIO(source_code).readline))\nx = tokenize.untokenize(tokens)\nprint(x)\n# 1 +\\\n#     2\n```\n\nNotice that the space between `+` and `\\` is now missing. The current tokenizer code simply inserts a backslash when it encounters two subsequent tokens with a differeing row offset:\n\nhttps://github.com/python/cpython/blob/9c2bb7d551a695f35db953a671a2ddca89426bef/Lib/tokenize.py#L179-L182\n\nI think this should be fixed. The docstring of `tokenize.untokenize` says:\n> Round-trip invariant for full input:\n        Untokenized source will match input source exactly\n\nTo fix this, it will probably be necessary to inspect the raw line contents and count how much whitespace there is at the end of the line.\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-126010\n* gh-129153\n* gh-130579\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/tomasr8","@type":"Person","name":"tomasr8"},"datePublished":"2024-10-15T20:53:55Z","url":"https://github.com/python/cpython/issues/125553"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:1f9937aa-4284-b491-fa0e-7b0f9e00a0fc
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idC76C:D064:2AEE405:3A88264:6B00519A
html-safe-nonce0106cc6a37137a6ea5ba03199eb6d6920aef30bd294468c9655f5231ec5a19ee
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNzZDOkQwNjQ6MkFFRTQwNTozQTg4MjY0OjZCMDA1MTlBIiwidmlzaXRvcl9pZCI6IjgyOTI2NzkyNDE1MTkwOTIxMjIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmacb37a74fda2243a59d7821594c30498b93dd94a09afa1b45275156f6d7103e642
hovercard-subject-tagissue:2589895964
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/125553/issue_layout
twitter:imagehttps://opengraph.githubassets.com/c8c23ada7f6815da985667f25abe58f88e11b6d53960d1cdf085b3004b53e783/python/cpython/issues/125553
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/c8c23ada7f6815da985667f25abe58f88e11b6d53960d1cdf085b3004b53e783/python/cpython/issues/125553
og:image:altBug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.generate_tokens(io.StringIO(source...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernametomasr8
hostnamegithub.com
expected-hostnamegithub.com
Nonef420f4996e6b1ff1ccd6d7683f8134f4b5f0f8a50124c210dbfe7490e5644abb
turbo-cache-controlno-preview
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release26e00df917aba0f128135dc073ea0c5e33a70d40
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/python/cpython/issues/125553#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F125553
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fissues%2F125553
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/issues/125553
Reloadhttps://github.com/python/cpython/issues/125553
Reloadhttps://github.com/python/cpython/issues/125553
pythonhttps://github.com/python
cpythonhttps://github.com/python/cpython
Codehttps://github.com/python/cpython
Issues5k+ (5k+)https://github.com/python/cpython/issues
Pull requests2.4k (2.4k)https://github.com/python/cpython/pulls
Actionshttps://github.com/python/cpython/actions
Projectshttps://github.com/python/cpython/projects
Security and qualityhttps://github.com/python/cpython/security
Insightshttps://github.com/python/cpython/insights
https://github.com/python/cpython
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/125553
New issuehttps://github.com/login?return_to=https://github.com/python/cpython/issues/125553
untokenize() does not round-trip for code containing line breaks (\ + \n)https://github.com/python/cpython/issues/125553#top
stdlibStandard Library Python modules in the Lib/ directoryhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22stdlib%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com/tomasr8
tomasr8https://github.com/tomasr8
on Oct 15, 2024https://github.com/python/cpython/issues/125553#issue-2589895964
cpython/Lib/tokenize.pyhttps://github.com/python/cpython/blob/9c2bb7d551a695f35db953a671a2ddca89426bef/Lib/tokenize.py#L179-L182
9c2bb7dhttps://github.com/python/cpython/commit/9c2bb7d551a695f35db953a671a2ddca89426bef
gh-125553: Fix backslash continuation in untokenize #126010https://github.com/python/cpython/pull/126010
[3.13] gh-125553: Fix backslash continuation in untokenize (GH-126010) #129153https://github.com/python/cpython/pull/129153
[3.12] gh-125553: Fix backslash continuation in untokenize (GH-126010) #130579https://github.com/python/cpython/pull/130579
tomasr8https://github.com/tomasr8
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
on Oct 15, 2024https://github.com/python/cpython/issues/125553#event-14664300796
picnixzhttps://github.com/picnixz
stdlibStandard Library Python modules in the Lib/ directoryhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22stdlib%22
on Oct 26, 2024https://github.com/python/cpython/issues/125553#event-14906187078
pablogsalhttps://github.com/pablogsal
on Jan 21, 2025https://github.com/python/cpython/issues/125553#event-16019547176
gh-125553: Fix backslash continuation in `untokenize` (#126010)https://github.com/python/cpython/commit/7ad793e5dbdf07e51a71b70d20f3e6e3ab60244d
gh-125553https://github.com/python/cpython/issues/125553
#126010https://github.com/python/cpython/pull/126010
7ad793ehttps://github.com/python/cpython/commit/7ad793e5dbdf07e51a71b70d20f3e6e3ab60244d
pablogsalhttps://github.com/pablogsal
completedhttps://github.com/python/cpython/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
on Jan 21, 2025https://github.com/python/cpython/issues/125553#event-16019669642
miss-islingtonhttps://github.com/miss-islington
on Jan 21, 2025https://github.com/python/cpython/issues/125553#event-16019685278
pythongh-125553: Fix backslash continuation in `untokenize` (pythonGH…https://github.com/miss-islington/cpython/commit/dd6c75718135ce586b9aac38774d86c1b1974328
pythongh-125553https://github.com/python/cpython/issues/125553
pythonGH…https://github.com/python/cpython/pull/126010
dd6c757https://github.com/miss-islington/cpython/commit/dd6c75718135ce586b9aac38774d86c1b1974328
pablogsalhttps://github.com/pablogsal
on Jan 21, 2025https://github.com/python/cpython/issues/125553#event-16020193517
[3.13] gh-125553: Fix backslash continuation in `untokenize` (GH-126010…https://github.com/python/cpython/commit/3048dcd15a357fe54c51d2572dacc49a296d0fc9
gh-125553https://github.com/python/cpython/issues/125553
GH-126010https://github.com/python/cpython/pull/126010
3048dcdhttps://github.com/python/cpython/commit/3048dcd15a357fe54c51d2572dacc49a296d0fc9
hugovkhttps://github.com/hugovk
on Feb 26, 2025https://github.com/python/cpython/issues/125553#event-16479092020
[3.12] pythongh-125553: Fix backslash continuation in `untokenize` (p…https://github.com/hugovk/cpython/commit/9663fe8f44d4445a47c76ec1e9ccf21a40d87b20
pythongh-125553https://github.com/python/cpython/issues/125553
p…https://github.com/python/cpython/pull/126010
9663fe8https://github.com/hugovk/cpython/commit/9663fe8f44d4445a47c76ec1e9ccf21a40d87b20
pablogsalhttps://github.com/pablogsal
on Feb 27, 2025https://github.com/python/cpython/issues/125553#event-16503741527
[3.12] gh-125553: Fix backslash continuation in `untokenize` (GH-126010…https://github.com/python/cpython/commit/b8f2ff08f1a2cf5bd08c84e998527551197e9a6d
gh-125553https://github.com/python/cpython/issues/125553
GH-126010https://github.com/python/cpython/pull/126010
b8f2ff0https://github.com/python/cpython/commit/b8f2ff08f1a2cf5bd08c84e998527551197e9a6d
Zac-HDhttps://github.com/Zac-HD
on Mar 1, 2025https://github.com/python/cpython/issues/125553#event-16523632116
Fix test_clean_source for python 3.13.2https://github.com/HypothesisWorks/hypothesis/commit/c24224095e45f4b8000003341fe71411830ee563
c242240https://github.com/HypothesisWorks/hypothesis/commit/c24224095e45f4b8000003341fe71411830ee563
Fix test_clean_source for python 3.13.2https://github.com/HypothesisWorks/hypothesis/commit/d8dec21b4736cc8e248a40d2622a623f5376c6f2
d8dec21https://github.com/HypothesisWorks/hypothesis/commit/d8dec21b4736cc8e248a40d2622a623f5376c6f2
Sign up for freehttps://github.com/signup?return_to=https://github.com/python/cpython/issues/125553
Sign in to commenthttps://github.com/login?return_to=https://github.com/python/cpython/issues/125553
stdlibStandard Library Python modules in the Lib/ directoryhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22stdlib%22
topic-parserhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22topic-parser%22
type-bugAn unexpected behavior, bug, or errorhttps://github.com/python/cpython/issues?q=state%3Aopen%20label%3A%22type-bug%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.