René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Tokenizer produces different output on Windows on py312 for ends of files","articleBody":"# Bug report\r\n\r\nIf you copy and paste the following code into a `repro.py` file and run `python -m tokenize` on it on a Windows machine, the output is different on 3.12/3.13 compared to what it was on 3.11 (the file ends with a single newline):\r\n\r\n```py\r\nfoo = 'bar'\r\nspam = 'eggs'\r\n```\r\n\r\nOn Python 3.11, on Windows, the output is this:\r\n\r\n```\r\n\u003e python -m tokenize cpython/repro.py\r\n0,0-0,0:            ENCODING       'utf-8'\r\n1,0-1,3:            NAME           'foo'\r\n1,4-1,5:            OP             '='\r\n1,6-1,11:           STRING         \"'bar'\"\r\n1,11-1,13:          NEWLINE        '\\r\\n'\r\n2,0-2,4:            NAME           'spam'\r\n2,5-2,6:            OP             '='\r\n2,7-2,13:           STRING         \"'eggs'\"\r\n2,13-2,15:          NEWLINE        '\\r\\n'\r\n3,0-3,0:            ENDMARKER      ''\r\n```\r\n\r\nOn Python 3.13 (@ https://github.com/python/cpython/commit/6e62eb2e70a9f2a8099735989a58e8c8cfb4a2f2) on Windows, however, the output is this:\r\n\r\n```\r\n\u003e python -m tokenize repro.py\r\n0,0-0,0:            ENCODING       'utf-8'\r\n1,0-1,3:            NAME           'foo'\r\n1,4-1,5:            OP             '='\r\n1,6-1,11:           STRING         \"'bar'\"\r\n1,11-1,12:          NEWLINE        '\\n'\r\n2,0-2,4:            NAME           'spam'\r\n2,5-2,6:            OP             '='\r\n2,7-2,13:           STRING         \"'eggs'\"\r\n2,13-2,14:          NEWLINE        '\\n'\r\n3,0-3,1:            NL             '\\n'\r\n4,0-4,0:            ENDMARKER      ''\r\n```\r\n\r\nThere appear to be two changes here:\r\n- All the NEWLINE tokens now have `\\n` values, whereas on Python 3.11 they all had `\\r\\n` values\r\n- There is an additional NL token at the end, immediately before the ENDMARKER token.\r\n\r\nAs discussed in https://github.com/PyCQA/pycodestyle/issues/1142, this appears to be Windows-specific, and _may_ be the cause of a large number of spurious W391 errors from the pycodestyle linting tool. (W391 dictates that there should be one, and only one, newline at the end of a file.) The pycodestyle tool is included in flake8, meaning that test failures in pycodestyle can cause test failures for other flake8 plugins. (All tests for [the flake8-pyi plugin](https://github.com/PyCQA/flake8-pyi), for example, currently fail on Python 3.13 on Windows.)\r\n\r\n# Your environment\r\n\r\n```\r\nPython 3.13.0a0 (heads/main:6e62eb2e70, May 27 2023, 14:00:13) [MSC v.1932 64 bit (AMD64)] on win32\r\n```\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-105022\n* gh-105023\n* gh-105030\n* gh-105041\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/AlexWaygood","@type":"Person","name":"AlexWaygood"},"datePublished":"2023-05-27T13:14:10.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/105017/cpython/issues/105017"}