René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`typing.Protocol` implementation means that `isinstance([], collections.abc.Mapping)` can sometimes evaluate to `True`","articleBody":"(In very specific circumstances)\r\n\r\n# Bug report\r\n\r\nDue to some changes that have been made to the implementation of `typing.Protocol` in Python 3.12, whether or not `isinstance([], collections.abc.Mapping)` evaluates to `True` or `False` now depends on precisely when and if garbage collection happens.\r\n\r\nMinimal repro:\r\n\r\n```pycon\r\n\u003e\u003e\u003e import gc, collections.abc, typing\r\n\u003e\u003e\u003e gc.disable()\r\n\u003e\u003e\u003e try:\r\n...     class Foo(collections.abc.Mapping, typing.Protocol): pass\r\n... except TypeError:\r\n...     pass\r\n...\r\n\u003e\u003e\u003e isinstance([], collections.abc.Mapping)\r\nTrue\r\n```\r\n\r\nWhy does this happen?! It's because `Foo` is an illegal class, so `TypeError` is raised during class initialisation here:\r\n\r\nhttps://github.com/python/cpython/blob/ce558e69d4087dd3653207de78345fbb8a2c7835/Lib/typing.py#L1905-L1912\r\n\r\n`TypeError` being raised here means that `Foo` never has `__protocol_attrs__` set on it. _But_, the error is raised _after_ class construction, meaning that if garbage collection happens at the wrong time, a reference to `Foo` is still retained in `collections.abc.Mapping.__subclasses__()`. This then creates problems in the `isinstance()` check, because of some behaviour in the `abc` module where it iterates through all of the subclasses of `collections.abc.Mapping` in order to determine whether an object is an instance of `collections.abc.Mapping`.\r\n\r\n```pycon\r\n\u003e\u003e\u003e import gc, collections.abc, typing\r\n\u003e\u003e\u003e gc.disable()\r\n\u003e\u003e\u003e try:\r\n...     class Foo(collections.abc.Mapping, typing.Protocol): pass\r\n... except TypeError:\r\n...     pass\r\n...\r\n\u003e\u003e\u003e x = collections.abc.Mapping.__subclasses__()\r\n\u003e\u003e\u003e x\r\n[\u003cclass 'collections.abc.MutableMapping'\u003e, \u003cclass '__main__.Foo'\u003e]\r\n\u003e\u003e\u003e x[-1].__protocol_attrs__\r\nset()\r\n```\r\n\r\nThe fix is to raise `TypeError` before the class has actually been created, rather than during class initialisation.\r\n\r\n# Why does this matter?\r\n\r\nThis might seem like an absurdly specific bug report, but it would be good to see it fixed. It was causing bizarre test failures in the `typing_extensions` backport. The issue was that we did this in one test method:\r\n\r\n```py\r\n    def test_protocols_bad_subscripts(self):\r\n        T = TypeVar('T')\r\n        S = TypeVar('S')\r\n        with self.assertRaises(TypeError):\r\n            class P(typing.Mapping[T, S], Protocol[T]): pass\r\n```\r\n\r\n...And that final `P` class _wasn't_ being cleaned up by the garbage collector immediately after that test having been run. That then caused an unrelated test elsewhere to fail due to the fact that `isinstance([], collections.abc.Mapping)` was evaluating to `True`, and it was _very_ hard to figure out why.\r\n\r\n\u003c!-- gh-linked-prs --\u003e\r\n### Linked PRs\r\n* gh-105281\r\n* gh-105318\n\u003c!-- /gh-linked-prs --\u003e\r\n","author":{"url":"https://github.com/AlexWaygood","@type":"Person","name":"AlexWaygood"},"datePublished":"2023-06-04T11:06:15.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/105280/cpython/issues/105280"}