René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`generate_tokens` tokenizes `$` differently with Python 3.12 than earlier","articleBody":"I tested Python 3.12 beta 1 with [Robot Framework](https://github.com/robotframework/robotframework) and noticed that `tokenize.generate_tokens()` handles expressions containing `$` differently than earlier. Earlier `$` yielded `ERRORTOKEN` but nowadays we get `OP`: \r\n\r\n```python\r\nPython 3.11.3 (main, Apr  5 2023, 14:15:06) [GCC 9.4.0] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n\u003e\u003e\u003e from tokenize import generate_tokens\r\n\u003e\u003e\u003e from io import StringIO\r\n\u003e\u003e\u003e next(generate_tokens(StringIO('$x').readline))\r\nTokenInfo(type=60 (ERRORTOKEN), string='$', start=(1, 0), end=(1, 1), line='$x')\r\n```\r\n\r\n```python\r\nPython 3.12.0b1 (main, May 22 2023, 23:31:26) [GCC 9.4.0] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n\u003e\u003e\u003e from tokenize import generate_tokens\r\n\u003e\u003e\u003e from io import StringIO\r\n\u003e\u003e\u003e next(generate_tokens(StringIO('$x').readline))\r\nTokenInfo(type=55 (OP), string='$', start=(1, 0), end=(1, 1), line='$x\\n')\r\n```\r\n\r\nWe support Python evaluation with special variables like `$var \u003e 1` in Robot Framework data and this change breaks our [tokenizing code](https://github.com/robotframework/robotframework/blob/34257fcee84604ecf7655edc0a11e415c23dab00/src/robot/variables/evaluation.py#L86). I didn't notice anything related in the release notes and decided to report this. If the change is intentional, we can easily update our code to handle also these semantics.\r\n\r\nNotice also that there's a small change with `TokenInfo.line` above. With Python 3.12 there's an additional `\\n` even though the original string didn't contain any newlines.","author":{"url":"https://github.com/pekkaklarck","@type":"Person","name":"pekkaklarck"},"datePublished":"2023-05-23T16:43:48.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/104802/cpython/issues/104802"}