René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support function decorators excellently","articleBody":"Decorators currently stress mypy's support for functional programming to the point that many decorators are impossible to type.  I'm intending this issue as a project plan and exploration of the issue of decorators and how to type them. It's a collection point for problems or incompletenesses that keep decorators from being fully supported, plans for solutions to those problems, and fully-implemented solutions to those problems.\r\n\r\n# Decorators that can only decorate a function of a fixed signature\r\n\r\nYou can define your decorator's signature explicitly, and it's completely supported:\r\n\r\n```python\r\nfrom typing import Callable, Tuple\r\n\r\ndef with_strlen(f: Callable[[int], str]) -\u003e Callable[[int], Tuple[str, int]]:\r\n    def ret(__i: int) -\u003e Tuple[str, int]:\r\n        r = f(__i)\r\n        return r, len(r)\r\n    return ret\r\n\r\n@with_strlen\r\ndef lol(x: int) -\u003e str:\r\n    return \"lol\"*x\r\n\r\nreveal_type(lol)  # E: Revealed type is 'def (builtins.int) -\u003e Tuple[builtins.str, builtins.int]\r\n ```\r\n Notes:\r\n* You can even use type variables to get some flexibility in argument and return types, but this falls over as soon as you don't know the exact number of arguments to expect.\r\n* Your decorated function's arguments can't be called by name.\r\n  - [ ] https://github.com/python/mypy/pull/2607 enables that\r\n\r\n# Decorators that do not change the signature of the function\r\n\r\nNearly fully supported. Here's how you do it:\r\n\r\n```python\r\nfrom typing import TypeVar, Callable, cast\r\n\r\nT = TypeVar('T')\r\n\r\ndef print_callcount(f: T) -\u003e T:\r\n    x = 0\r\n    def ret(*args, **kwargs):\r\n        nonlocal x\r\n        x += 1\r\n        print(\"%d calls so far\" % x)\r\n        return f(*args, **kwargs)\r\n\r\n    return cast(T, ret)\r\n\r\n@print_callcount\r\ndef lol(x: int) -\u003e str:\r\n    return \"lol\"*x\r\n\r\nreveal_type(lol)  # E: Revealed type is 'def (x: builtins.int) -\u003e builtins.str'\r\n```\r\n\r\nNotes: \r\n* Mypy trusts that you can call `f`.  You can set a bound on `T` to be `Callable`, but you don't need to for it to typecheck.\r\n* This business doesn't typecheck without the cast. That's a symptom of the fact that mypy doesn't understand the function nature of the argument to `print_callcount`, and there's no way to declare that argument as a `Callable` explicitly without losing the argument type of `lol` later. We'd like to minimize the places where casts are required. Doing so requires something along the lines of \"variadic argument variables\", discussed below.\r\n\r\n# Decorators that take arguments (\"second-order\"?)\r\n\r\nPlenty of decorators \"take arguments\" by *actually* being functions that return a decorator. For example, we'd like to be able to do this:\r\n\r\n```python\r\nfrom typing import Any, TypeVar, Callable, cast\r\n\r\nT = TypeVar('T')\r\n\r\ndef callback_callcount(cb: Callable[[int], None]) -\u003e Callable[[T], T]:\r\n    def outer(f: T) -\u003e T:\r\n        x = 0\r\n        def inner(*args, **kwargs):\r\n            nonlocal x\r\n            x += 1\r\n            cb(x)\r\n            return f(*args, **kwargs)\r\n        return cast(T, inner)\r\n    return outer\r\n\r\ndef print_int(x: int) -\u003e None:\r\n    print(x)\r\n\r\n@callback_callcount(print_int)\r\ndef lol(x: int) -\u003e str:\r\n    return \"lol\"*x\r\n\r\nreveal_type(lol)  # E: Revealed type is 'def (x: builtins.int) -\u003e builtins.str'\r\n```\r\nNotes:\r\n* This does not typecheck yet -- errors on calling the decorator, and `lol` ends up typed as `None`\r\n* Relevant issue: https://github.com/python/mypy/issues/1551\r\n  - [ ] https://github.com/python/mypy/issues/3028 fixes this.\r\n* Still has a non-ideal cast...\r\n\r\n# Mess with the return type or with arguments\r\n\r\nFor an arbitrary function you can't do this at all yet -- there isn't even a syntax.  Here's me making up some syntax for it.\r\n\r\n## Messing with the return type\r\n```python\r\nfrom typing import Any, Dict, Callable\r\n\r\nfrom mypy_extensions import SomeArguments\r\n\r\ndef reprify(f: Callable[[SomeArguments], Any]) -\u003e Callable[[SomeArguments], str]:\r\n    def ret(*args: SomeArguments.positional, **kwargs: SomeArguments.keyword):\r\n        return repr(f(*args, **kwargs))\r\n    return ret\r\n\r\n@reprify\r\ndef lol(x: int) -\u003e Dict[str, int]:\r\n    return {\"lol\": x}\r\n\r\nreveal_type(lol)  # E: Revealed type is 'def (x: builtins.int) -\u003e builtins.str'\r\n```\r\n\r\n## Messing with the arguments\r\n\r\n```python\r\nfrom typing import Any, Callable, TypeVar\r\n\r\nfrom mypy_extensions import SomeArguments\r\n\r\nR = TypeVar('R')\r\n\r\ndef supply_zero(f: Callable[[int, SomeArguments], R]) -\u003e Callable[[SomeArguments], R]:\r\n    def ret(*args: SomeArguments.positional, **kwargs: SomeArguments.keyword):\r\n        return f(0, *args, **kwargs)\r\n    return ret\r\n\r\n@supply_zero\r\ndef lol(x: int, y: str) -\u003e str:\r\n    return \"%d and %s\" % (x, y)\r\n\r\nreveal_type(lol)  # E: Revealed type is 'def (y: builtins.str) -\u003e builtins.str'\r\n```\r\n\r\nThe syntax here is fungible, but we would need a way to do approximately this thing -- capture the types and kinds of all a function's arguments in some kind of variation on a type variable. \r\n\r\nRelevant issues and discussions:\r\n* https://github.com/python/mypy/issues/1927\r\n* https://github.com/python/mypy/issues/3028\r\n* https://github.com/python/typing/issues/239\r\n* https://github.com/python/typing/issues/193\r\n\r\nVariadic type variables alone (https://github.com/python/typing/issues/193) get you some of the way there, but lose all keyword arguments of the decorated function. \r\n\r\nThings to do:\r\n* [ ] Implement variadic type variables (fill in PR when I have it)\r\n* [ ] Write up detailed proposal for semantics of argument variables\r\n  - [ ] ... and how they interact with `*args` and `**kwargs`\r\n  - [ ] ... and their relationship to variadic type variables and the *expand* operation\r\n  - [ ] ... and the semantics of an easy-to-use `SomeArguments`-style alias, so nobody has to actually engage with the details of the above when writing normal decorators.\r\n* [ ] Come to some kind of mypy-community consensus or near-consensus on that proposal. It'll be in `mypy_extensions` not `typing` at first -- this can be fodder for the future of PEP484, but while we're playing in such experimental land, not yet.\r\n* [ ] PR to implment the `SomeArguments` thing.\r\n","author":{"url":"https://github.com/sixolet","@type":"Person","name":"sixolet"},"datePublished":"2017-04-12T00:52:02.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":22},"url":"https://github.com/3157/mypy/issues/3157"}