René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"The lines in tokens from `tokenize.generate_tokens` incorrectly indicate multiple lines.","articleBody":"The line attribute in tokens returned by `tokenize.generate_tokens` incorrectly indicate multiple lines. The tokens should have an invariant that using the `.start` and `.end` attributes to index into the `.line` attribute will produce the `.string` attribute.\r\n\r\ntokbug.py:\r\n\r\n```python\r\nimport io\r\nimport sys\r\nimport tokenize\r\n\r\nSOURCE = r\"\"\"\r\na + \\\r\nb\r\n\"\"\"\r\n\r\nprint(sys.version)\r\nreadline = io.StringIO(SOURCE).readline\r\nfor tok in tokenize.generate_tokens(readline):\r\n    correct = (tok.string) == (tok.line[tok.start[1]: tok.end[1]])\r\n    print(tok, \"\" if correct else \"\u003c*****!!!\")\r\n```\r\n\r\nRun with 3.12.0a7:\r\n```\r\n% /usr/local/pyenv/pyenv/versions/3.12.0a7/bin/python tokbug.py\r\n3.12.0a7 (main, Apr  5 2023, 05:51:58) [Clang 14.0.3 (clang-1403.0.22.14.1)]\r\nTokenInfo(type=62 (NL), string='\\n', start=(1, 0), end=(1, 1), line='\\n')\r\nTokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\\\\n')\r\nTokenInfo(type=54 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\\\\n')\r\nTokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='b\\n')\r\nTokenInfo(type=4 (NEWLINE), string='\\n', start=(3, 1), end=(3, 2), line='b\\n')\r\nTokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')\r\n```\r\n\r\nRun with 3.12.0b1:\r\n```\r\n% /usr/local/pyenv/pyenv/versions/3.12.0b1/bin/python tokbug.py\r\n3.12.0b1 (main, May 23 2023, 16:19:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]\r\nTokenInfo(type=65 (NL), string='\\n', start=(1, 0), end=(1, 1), line='\\n')\r\nTokenInfo(type=1 (NAME), string='a', start=(2, 0), end=(2, 1), line='a + \\\\\\n')\r\nTokenInfo(type=55 (OP), string='+', start=(2, 2), end=(2, 3), line='a + \\\\\\n')\r\nTokenInfo(type=1 (NAME), string='b', start=(3, 0), end=(3, 1), line='a + \\\\\\nb\\n') \u003c*****!!!\r\nTokenInfo(type=4 (NEWLINE), string='\\n', start=(3, 1), end=(3, 2), line='a + \\\\\\nb\\n') \u003c*****!!!\r\nTokenInfo(type=0 (ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')\r\n```\r\n\r\nRelated to #104825? cc @pablogsal \n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-104975\n* gh-104982\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/nedbat","@type":"Person","name":"nedbat"},"datePublished":"2023-05-26T11:47:30.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/104972/cpython/issues/104972"}