René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"(3.12+) tokenize exception type is wrong","articleBody":"\u003c!--\r\n  If you're new to Python and you're not sure whether what you're experiencing is a bug, the CPython issue tracker is not\r\n  the right place to seek help. Consider the following options instead:\r\n\r\n  - reading the Python tutorial: https://docs.python.org/3/tutorial/\r\n  - posting in the \"Users\" category on discuss.python.org: https://discuss.python.org/c/users/7\r\n  - emailing the Python-list mailing list: https://mail.python.org/mailman/listinfo/python-list\r\n  - searching our issue tracker (https://github.com/python/cpython/issues) to see if\r\n    your problem has already been reported\r\n--\u003e\r\n\r\n# Bug report\r\n\r\nthe document exception type for the `tokenize.*` functions is `TokenError` -- however it appears it now raises `SyntaxError`: https://docs.python.org/3.12/library/tokenize.html#tokenize.TokenError\r\n\r\n\u003e Raised when either a docstring or expression that may be split over several lines is not completed anywhere in the file, for example:\r\n\r\n```python\r\nimport io\r\nimport tokenize\r\n\r\nlist(tokenize.generate_tokens(io.StringIO('\"\"\"').readline))\r\n```\r\n\r\n```console\r\n$ python3.11 t.py\r\nTraceback (most recent call last):\r\n  File \"/home/asottile/workspace/switch-microcontroller/t.py\", line 4, in \u003cmodule\u003e\r\n    list(tokenize.generate_tokens(io.StringIO('\"\"\"').readline))\r\n  File \"/usr/lib/python3.11/tokenize.py\", line 465, in _tokenize\r\n    raise TokenError(\"EOF in multi-line string\", strstart)\r\ntokenize.TokenError: ('EOF in multi-line string', (1, 0))\r\n```\r\nand\r\n\r\n```console\r\n#  python3.13 t.py\r\nTraceback (most recent call last):\r\n  File \"/y/pyupgrade/t.py\", line 4, in \u003cmodule\u003e\r\n    list(tokenize.generate_tokens(io.StringIO('\"\"\"').readline))\r\n  File \"/usr/lib/python3.13/tokenize.py\", line 526, in _generate_tokens_from_c_tokenizer\r\n    for info in it:\r\n  File \"\u003cstring\u003e\", line 1\r\n    \"\"\"\r\n    ^\r\nSyntaxError: unterminated triple-quoted string literal (detected at line 1)\r\n```\r\n\r\n# Your environment\r\n\r\n\u003c!-- Include as many relevant details as possible about the environment you experienced the bug in --\u003e\r\n\r\n- CPython versions tested on: c7bf74bacd\r\n- Operating system and architecture: ubuntu 22.04 x86_64\r\n\r\n\u003c!--\r\nYou can freely edit this text. Remove any lines you believe are unnecessary.\r\n--\u003e\r\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-105399\n* gh-105439\n* gh-105466\n* gh-105472\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/asottile","@type":"Person","name":"asottile"},"datePublished":"2023-06-06T14:44:19.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/105390/cpython/issues/105390"}