René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Consider emitting buffered DEDENT tokens on the last line","articleBody":"In Python 3.12, porting the tokenizer to use the C tokenizer underneath to support PEP 701 has now a documented change in [docs.python.org/3.12/whatsnew/3.12.html#changes-in-the-python-api](https://docs.python.org/3.12/whatsnew/3.12.html#changes-in-the-python-api):\r\n\r\n\u003e Some final DEDENT tokens are now emitted within the bounds of the input. This means that for a file containing 3 lines, the old version of the tokenizer returned a DEDENT token in line 4 whilst the new version returns the token in line 3.\r\n\r\nApparently, this affects negatively some formatting tools (see https://github.com/PyCQA/pycodestyle/issues/1142). Let's consider what options do we have and see if we can fix this without adding a lot of maintenance burden to the C tokenizer or slowing down everything.\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-104980\n* gh-105000\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/pablogsal","@type":"Person","name":"pablogsal"},"datePublished":"2023-05-26T13:29:30.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/104976/cpython/issues/104976"}