René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"SyntaxError: unmatched ')' with whitespace in lambda?","articleBody":"# Bug report\r\n\r\nTesting on tip of 3.12 which includes the fix for #105013 (thanks!), I get the following difference in behaviour with Python 3.11 vs tip of Python 3.12:\r\n```python\r\nimport inspect\r\n\r\nfrom hypothesis.internal.reflection import extract_lambda_source\r\nfrom hypothesis.strategies import just\r\n\r\n#from hypothesis.strategies import just, one_of\r\n\r\n# This variant doesn't trigger a TypeError mid-callback, but both variants get the same final inspect error\r\n#one_of_nested_strategy_with_filter = one_of(\r\n#    just(0),\r\n#    just(1),\r\n#    one_of(just(2), just(3), one_of(just(4), just(5), one_of(just(6), just(7)))),\r\n#).filter(lambda x: x % 2 == 0)\r\n#x = get_pretty_function_description(one_of_nested_strategy_with_filter)\r\n#print(inspect.getsource(x))\r\n\r\n\r\none_of_nested_strategy_with_filter = (\r\n    just(0)\r\n).filter(lambda x: x % 2 == 0)\r\n\r\nx = extract_lambda_source(one_of_nested_strategy_with_filter)\r\n```\r\n\r\nWith Python 3.12, I get:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1241, in getblock\r\n    for _token in tokens:\r\n  File \"/usr/lib/python3.12/tokenize.py\", line 450, in _tokenize\r\n    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):\r\n  File \"/usr/lib/python3.12/tokenize.py\", line 537, in _generate_tokens_from_c_tokenizer\r\n    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):\r\n  File \"\u003cstring\u003e\", line 1\r\n    ).filter(lambda x: x % 2 == 0)\r\n    ^\r\nSyntaxError: unmatched ')'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/tmp/foo2.py\", line 10, in \u003cmodule\u003e\r\n    x = extract_lambda_source(one_of_nested_strategy_with_filter)\r\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py\", line 305, in extract_lambda_source\r\n    sig = inspect.signature(f)\r\n          ^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 3326, in signature\r\n    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 3070, in from_callable\r\n    return _signature_from_callable(obj, sigcls=cls,\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 2484, in _signature_from_callable\r\n    raise TypeError('{!r} is not a callable object'.format(obj))\r\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py\", line 39, in __repr__\r\n    suffix = \"\".join(\r\n             ^^^^^^^^\r\n  File \"/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py\", line 40, in \u003cgenexpr\u003e\r\n    f\".{name}({get_pretty_function_description(f)})\"\r\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py\", line 432, in get_pretty_function_description\r\n    return extract_lambda_source(f)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py\", line 312, in extract_lambda_source\r\n    source = inspect.getsource(f)\r\n             ^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1282, in getsource\r\n    lines, lnum = getsourcelines(object)\r\n                  ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1274, in getsourcelines\r\n    return getblock(lines[lnum:]), lnum + 1\r\n           ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1248, in getblock\r\n    _, *_token_info = _token\r\n                      ^^^^^^\r\nUnboundLocalError: cannot access local variable '_token' where it is not associated with a value\r\n```\r\n\r\nBut with Python 3.11, I get:\r\n```\r\nTraceback (most recent call last):\r\n  File \"/tmp/foo2.py\", line 10, in \u003cmodule\u003e\r\n    x = extract_lambda_source(one_of_nested_strategy_with_filter)\r\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/site-packages/hypothesis/internal/reflection.py\", line 305, in extract_lambda_source\r\n    sig = inspect.signature(f)\r\n          ^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/inspect.py\", line 3279, in signature\r\n    return Signature.from_callable(obj, follow_wrapped=follow_wrapped,\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/inspect.py\", line 3027, in from_callable\r\n    return _signature_from_callable(obj, sigcls=cls,\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.11/inspect.py\", line 2447, in _signature_from_callable\r\n    raise TypeError('{!r} is not a callable object'.format(obj))\r\nTypeError: just(0).filter(lambda x: \u003cunknown\u003e) is not a callable object\r\n```\r\n\r\nIf I change the program to drop the whitespace, it works in 3.12 too:\r\n```\r\nimport inspect\r\n\r\nfrom hypothesis.internal.reflection import extract_lambda_source\r\nfrom hypothesis.strategies import just\r\n\r\none_of_nested_strategy_with_filter = (just(0)).filter(lambda x: x % 2 == 0)\r\n\r\nx = extract_lambda_source(one_of_nested_strategy_with_filter)\r\n```\r\n\r\nI noticed this w/ a test failure in priority (the output is huge, so just a snippet here)\r\n```\r\nERROR collecting test/test_priority.py ____________________________________________________________________________________\r\n/usr/lib/python3.12/inspect.py:1241: in getblock\r\n    for _token in tokens:\r\n[...]\r\n/usr/lib/python3.12/tokenize.py:537: in _generate_tokens_from_c_tokenizer\r\n    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):\r\nE     File \"\u003cstring\u003e\", line 1\r\nE       ).map(lambda blocked: (blocked, active_readme_streams_from_filter(blocked)))\r\nE       ^\r\nE   SyntaxError: unmatched ')'\r\n        c_tokenizer = \u003cmodule '_tokenize' (built-in)\u003e\r\n        extra_tokens = True\r\n        source     = (').map(lambda blocked: (blocked, '\r\n 'active_readme_streams_from_filter(blocked)))\\n'\r\n[...]\r\nERROR test/test_priority.py - UnboundLocalError: cannot access local variable '_token' where it is not associated with a value\r\n```\r\n\r\nand a perhaps more useful test failure in hypothesis, which priority uses:\r\n```\r\ntest_one_of_flattens_filter_branches_2 ____________________________________________________________________________________\r\n[gw14] linux -- Python 3.12.0 /var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/bin/python3.12\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1241, in getblock\r\n    for _token in tokens:\r\n  File \"/usr/lib/python3.12/tokenize.py\", line 450, in _tokenize\r\n    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):\r\n  File \"/usr/lib/python3.12/tokenize.py\", line 537, in _generate_tokens_from_c_tokenizer\r\n    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):\r\n  File \"\u003cstring\u003e\", line 1\r\n    ).filter(lambda x: x % 2 == 0)\r\n    ^\r\nSyntaxError: unmatched ')'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python/tests/quality/test_discovery_ability.py\", line 101, in run_test\r\n    runner.run()\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py\", line 474, in run\r\n    self._run()\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py\", line 880, in _run\r\n    self.generate_new_examples()\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py\", line 684, in generate_new_examples\r\n    minimal_example = self.cached_test_function(\r\n                      ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py\", line 1065, in cached_test_function\r\n    self.test_function(data)\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py\", line 209, in test_function\r\n    self.__stoppable_test_function(data)\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/engine.py\", line 185, in __stoppable_test_function\r\n    self._test_function(data)\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python/tests/quality/test_discovery_ability.py\", line 79, in test_function\r\n    value = data.draw(specifier)\r\n            ^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py\", line 956, in draw\r\n    return strategy.do_draw(self)\r\n           ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py\", line 942, in do_draw\r\n    result = self.do_filtered_draw(data)\r\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py\", line 956, in do_filtered_draw\r\n    value = data.draw(self.filtered_strategy)\r\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py\", line 951, in draw\r\n    return strategy.do_draw(self)\r\n           ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py\", line 666, in do_draw\r\n    return data.draw(strategy)\r\n           ^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py\", line 951, in draw\r\n    return strategy.do_draw(self)\r\n           ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/strategies.py\", line 532, in do_draw\r\n    data.mark_invalid(f\"Aborted test because unable to satisfy {self!r}\")\r\n                                                               ^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py\", line 39, in __repr__\r\n    suffix = \"\".join(\r\n             ^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/strategies/_internal/misc.py\", line 40, in \u003cgenexpr\u003e\r\n    f\".{name}({get_pretty_function_description(f)})\"\r\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py\", line 432, in get_pretty_function_description\r\n    return extract_lambda_source(f)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/var/tmp/portage/dev-python/hypothesis-6.75.6/work/hypothesis-hypothesis-python-6.75.6/hypothesis-python-python3_12/install/usr/lib/python3.12/site-packages/hypothesis/internal/reflection.py\", line 312, in extract_lambda_source\r\n    source = inspect.getsource(f)\r\n             ^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1282, in getsource\r\n    lines, lnum = getsourcelines(object)\r\n                  ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1274, in getsourcelines\r\n    return getblock(lines[lnum:]), lnum + 1\r\n           ^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib/python3.12/inspect.py\", line 1248, in getblock\r\n    _, *_token_info = _token\r\n                      ^^^^^^\r\nUnboundLocalError: cannot access local variable '_token' where it is not associated with a value\r\n```\r\n\r\nSee also https://github.com/python/cpython/issues/105013.\r\n\r\n# Your environment\r\n\r\n- CPython versions tested on: 3.11.3, tip of 3.12\r\n- Operating system and architecture: Gentoo Linux, amd64\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-105061\n* gh-105120\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/thesamesam","@type":"Person","name":"thesamesam"},"datePublished":"2023-05-28T15:07:59.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/105042/cpython/issues/105042"}