Title: untokenize() does not round-trip for code containing line breaks (`\` + `\n`) · Issue #125553 · python/cpython · GitHub
Open Graph Title: untokenize() does not round-trip for code containing line breaks (`\` + `\n`) · Issue #125553 · python/cpython
X Title: untokenize() does not round-trip for code containing line breaks (`\` + `\n`) · Issue #125553 · python/cpython
Description: Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.generate_tokens(io.StringIO(source_code).readline)) x = tokenize.untokeniz...
Open Graph Description: Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.generate_tokens(io.StringIO(source...
X Description: Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.gene...
Opengraph URL: https://github.com/python/cpython/issues/125553
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"untokenize() does not round-trip for code containing line breaks (`\\` + `\\n`)","articleBody":"# Bug report\n\n### Bug description:\n\nCode which contains line breaks is not round-trip invariant:\n```python\nimport tokenize, io\n\nsource_code = r\"\"\"\n1 + \\\n 2\n\"\"\"\n\ntokens = list(tokenize.generate_tokens(io.StringIO(source_code).readline))\nx = tokenize.untokenize(tokens)\nprint(x)\n# 1 +\\\n# 2\n```\n\nNotice that the space between `+` and `\\` is now missing. The current tokenizer code simply inserts a backslash when it encounters two subsequent tokens with a differeing row offset:\n\nhttps://github.com/python/cpython/blob/9c2bb7d551a695f35db953a671a2ddca89426bef/Lib/tokenize.py#L179-L182\n\nI think this should be fixed. The docstring of `tokenize.untokenize` says:\n> Round-trip invariant for full input:\n Untokenized source will match input source exactly\n\nTo fix this, it will probably be necessary to inspect the raw line contents and count how much whitespace there is at the end of the line.\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-126010\n* gh-129153\n* gh-130579\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/tomasr8","@type":"Person","name":"tomasr8"},"datePublished":"2024-10-15T20:53:55Z","url":"https://github.com/python/cpython/issues/125553"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:1f9937aa-4284-b491-fa0e-7b0f9e00a0fc |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C76C:D064:2AEE405:3A88264:6B00519A |
| html-safe-nonce | 0106cc6a37137a6ea5ba03199eb6d6920aef30bd294468c9655f5231ec5a19ee |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNzZDOkQwNjQ6MkFFRTQwNTozQTg4MjY0OjZCMDA1MTlBIiwidmlzaXRvcl9pZCI6IjgyOTI2NzkyNDE1MTkwOTIxMjIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | b37a74fda2243a59d7821594c30498b93dd94a09afa1b45275156f6d7103e642 |
| hovercard-subject-tag | issue:2589895964 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/125553/issue_layout |
| twitter:image | https://opengraph.githubassets.com/c8c23ada7f6815da985667f25abe58f88e11b6d53960d1cdf085b3004b53e783/python/cpython/issues/125553 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/c8c23ada7f6815da985667f25abe58f88e11b6d53960d1cdf085b3004b53e783/python/cpython/issues/125553 |
| og:image:alt | Bug report Bug description: Code which contains line breaks is not round-trip invariant: import tokenize, io source_code = r""" 1 + \ 2 """ tokens = list(tokenize.generate_tokens(io.StringIO(source... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | tomasr8 |
| hostname | github.com |
| expected-hostname | github.com |
| None | f420f4996e6b1ff1ccd6d7683f8134f4b5f0f8a50124c210dbfe7490e5644abb |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 26e00df917aba0f128135dc073ea0c5e33a70d40 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width