René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Possible regression in Python3.12 tokenizer","articleBody":"# Bug report\r\n\r\nI have encountered a possible regression while I was testing pyrsistent with python3.12_beta1, see https://github.com/tobgu/pyrsistent/issues/275. Tests fail with following error:\r\n```\r\n/usr/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule\r\n    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)\r\n/usr/lib/python3.12/site-packages/_pytest/pathlib.py:564: in import_path\r\n    importlib.import_module(module_name)\r\n/usr/lib/python3.12/importlib/__init__.py:90: in import_module\r\n    return _bootstrap._gcd_import(name[level:], package, level)\r\n\u003cfrozen importlib._bootstrap\u003e:1293: in _gcd_import\r\n    ???\r\n\u003cfrozen importlib._bootstrap\u003e:1266: in _find_and_load\r\n    ???\r\n\u003cfrozen importlib._bootstrap\u003e:1237: in _find_and_load_unlocked\r\n    ???\r\n\u003cfrozen importlib._bootstrap\u003e:841: in _load_unlocked\r\n    ???\r\n/usr/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module\r\n    exec(co, module.__dict__)\r\ntests/hypothesis_vector_test.py:48: in \u003cmodule\u003e\r\n    PVectorAndLists = st.lists(st.builds(RefCountTracker)).map(\r\n/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py:347: in map\r\n    if is_identity_function(pack):\r\n/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:638: in is_identity_function\r\n    return bool(re.fullmatch(r\"lambda (\\w+): \\1\", get_pretty_function_description(f)))\r\n/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:432: in get_pretty_function_description\r\n    return extract_lambda_source(f)\r\n/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py:312: in extract_lambda_source\r\n    source = inspect.getsource(f)\r\n/usr/lib/python3.12/inspect.py:1274: in getsource\r\n    lines, lnum = getsourcelines(object)\r\n/usr/lib/python3.12/inspect.py:1266: in getsourcelines\r\n    return getblock(lines[lnum:]), lnum + 1\r\n/usr/lib/python3.12/inspect.py:1241: in getblock\r\n    for _token in tokens:\r\n/usr/lib/python3.12/tokenize.py:451: in _tokenize\r\n    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):\r\n/usr/lib/python3.12/tokenize.py:542: in _generate_tokens_from_c_tokenizer\r\n    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):\r\nE     File \"\u003cstring\u003e\", line 1\r\nE       lambda l: (l, pvector(l)))\r\nE                                ^\r\nE   SyntaxError: unmatched ')'\r\n```\r\n\r\nIt is triggered for this code (see https://github.com/tobgu/pyrsistent/blob/cc90f3e2b339653fde0df422aaf3ccdb3fc1225d/tests/hypothesis_vector_test.py#L48-L49 )\r\n```python\r\nPVectorAndLists = st.lists(st.builds(RefCountTracker)).map(\r\n    lambda l: (l, pvector(l)))\r\n```\r\nif I concatenate it to one line, everything works. I also tried to backport fix for https://github.com/python/cpython/issues/104866 as it sounds relevant, but it didn't solved the issue.\r\n\r\n# Your environment\r\n\r\n- Gentoo `~amd64`\r\n- Python 3.12.0b1\r\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-105021\n* gh-105032\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/arkamar","@type":"Person","name":"arkamar"},"datePublished":"2023-05-27T09:09:47.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":15},"url":"https://github.com/105013/cpython/issues/105013"}