Title: [DISCUSSION] Dataclasses · Issue #2698 · python-telegram-bot/python-telegram-bot · GitHub
Open Graph Title: [DISCUSSION] Dataclasses · Issue #2698 · python-telegram-bot/python-telegram-bot
X Title: [DISCUSSION] Dataclasses · Issue #2698 · python-telegram-bot/python-telegram-bot
Description: The idea to use dataclasses has been discussed a number of times (see #2056), more recently in #2491 and there is also a connection to #2607. So here I am opening an issue to write down some thoughts. This is rather for from being my fin...
Open Graph Description: The idea to use dataclasses has been discussed a number of times (see #2056), more recently in #2491 and there is also a connection to #2607. So here I am opening an issue to write down some though...
X Description: The idea to use dataclasses has been discussed a number of times (see #2056), more recently in #2491 and there is also a connection to #2607. So here I am opening an issue to write down some though...
Opengraph URL: https://github.com/python-telegram-bot/python-telegram-bot/issues/2698
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"[DISCUSSION] Dataclasses","articleBody":"The idea to use dataclasses has been discussed a number of times (see #2056), more recently in #2491 and there is also a connection to #2607. So here I am opening an issue to write down some thoughts.\r\n\r\nThis is rather for from being my final judgement on the matter and I'm aware that it will be a while until we properly discuss this. So more of a reminder for future-me.\r\n\r\n## Motivation\r\nWe have a lot of boilerplate code. especially the inits of classes like `Message` take up a lot of place, along with `__slots__` and `de/to_json`.\r\nThere are ways to hide those away by using 3rd pary libraries (that inspired `dataclasses`) or the built-in `dataclasses`. Here is what such approaches can possibly do for us:\r\n\r\n* removing boilerplate inits\r\n* removing boilerplate slots\r\n* Solving big parts of #2491 without any work at all\r\n* Making TelegramObjects immutable. I'm really not sure if we want that, but it's at least an interesting thought given that incoming updates represent a static information and changing them doesn't make sense at all.\r\n\r\n**Edits:** \r\n* inspired by #2747 dataclasses could also be used for `ext.Defaults` and `CallbackContext`, especially making them immutable\r\n* dataclasses would (as side effect) solve large parts of microsoft/pylance-release#1983\r\n* with dataclasses, we could get rid of the `__new__` workaround in `TelegramObject`: The reason for that was that we didn't want to have to call `super().__init__` in all the subclasses. if `TO` is a dataclass as well, then the auto-generated inits of the subclasses would automatically do that for us\r\n\r\n## A word on 3rd party libs\r\nA number of 3rd party libraries that offer similar functionality as `dataclasses`, most notably `attrs` and `pydantic` but also [some others](https://pydantic-docs.helpmanual.io/benchmarks/). I know that we try to keep our dependencies at a minimum and I'm fully backing that mentality. Still, let me list some pros & cons that I see of possible usage of `attrs` & `pydantic` - compared to using `dataclasses`\r\n\r\n* ➕ Automated handling of (nested!) to/de_json logic. `pydantic` has this built-in, `attrs` can do this together with the `cattrs` library\r\n* ➕/➖ Data validation. `pydantic` is heavily inspired by that. This means that e.g. `self.text = text` is replaced by `self.text = str(text)` in `pydantic`, but apparently also [slows it down](https://stefan.sofa-rockers.org/2020/05/29/attrs-dataclasses-pydantic/)\r\n* ➕ `attrs` comes with support for slots, which dataclass only adds in py3.10. IISC `pydantic` [does not](https://jackmckew.dev/dataclasses-vs-attrs-vs-pydantic.html)automatically \r\n* ➖ IISC neither `pydantic` nor `attrs` can do both extra arguments and slots\r\n* ➖IDE- & type-checker integration may be non-optimal (`pydantice` provides plugins for `mypy` and PyCharm)\r\n* ➖ They are dependencies. duh. If your own boilerplate is fasly, you can change it. If the dependency has a bug, then good night.\r\n* ➕/➖ Handling of extra arguments - this is actually really important, also for #2607 and I'll go into detail on that later. TL;DR: `pydantic` can dismiss additional kwargs that are passed to init, e.g. `Chat(…, new_argument=42)` will work. `attrs` can't, IISC. `dataclass` can't either\r\n\r\nBut let's focus on the last bullet point. If we can't handle that properly, then there is no use for `attrs` or `dataclass` at all\r\n\r\n## Dismissal of additional kwargs\r\n\r\nCurrently there is no way to make `dataclass` add `**kwargs` to the auto-generated `__init__`.\r\nBut what if we approach this problem differently? Instead of passing the additional kwargs, we can filter them out before. Ofc this is more expensive, but\r\n\r\n* the chance of this happening is low (new arguments are not added too aften & we're usually lightning fast on API updates \\*cough\\*)\r\n* similar to the discussion #1924, by removing `**kwargs` we no longer just swallow typos in arguments etc. Admittedly, this benefit is mall, since classes like `Message`, `Chat` etc are usually not manually instantiated.\r\n\r\nSuch an approach should work with both `attrs` and `dataclass`\r\n\r\nOne thing to keep in mind is that we currently pass `bot` to __all__ classes, so a lot of them always have one additional kwarg. This should be easily solvable by different means. E.g. one could\r\n\r\n* add `bot` to all signatures\r\n* add a class variable `HAS_SHORTCUTS` that tells us whether or not we need to pass `bot`\r\n* add a method `TelegramObject.set_bot` to remove `bot` from all signatures\r\n\r\nSo here's a small comparison of the two approaches:\r\n\r\n```python\r\nimport inspect\r\nimport timeit\r\n\r\n\r\nclass InspectDeJson:\r\n _init_params = None\r\n\r\n def __init__(self, arg1, arg2, arg3, arg4, arg5, arg6, arg7):\r\n self.arg1 = arg1\r\n self.arg2 = arg2\r\n self.arg3 = arg3\r\n self.arg4 = arg4\r\n self.arg5 = arg5\r\n self.arg6 = arg6\r\n self.arg7 = arg7\r\n\r\n @classmethod\r\n def de_json(cls, data):\r\n if cls._init_params is None:\r\n signature = inspect.signature(cls)\r\n cls._init_params = signature.parameters.keys()\r\n\r\n # try-except is significantly faster in case we already have a correct argument set\r\n try:\r\n return cls(**data)\r\n except TypeError as exc:\r\n if str(exc).startswith('__init__() got an unexpected keyword argument'):\r\n return cls(**{k: v for k, v in data.items() if k in cls._init_params})\r\n else:\r\n raise exc\r\n\r\n\r\nclass KwargsDeJson:\r\n\r\n def __init__(self, arg1, arg2, arg3, arg4, arg5, arg6, arg7, **kwargs):\r\n self.arg1 = arg1\r\n self.arg2 = arg2\r\n self.arg3 = arg3\r\n self.arg4 = arg4\r\n self.arg5 = arg5\r\n self.arg6 = arg6\r\n self.arg7 = arg7\r\n\r\n @classmethod\r\n def de_json(cls, data):\r\n return cls(**data)\r\n\r\n\r\ncorrect_data = {\r\n 'arg1': 'arg1',\r\n 'arg2': 'arg2',\r\n 'arg3': 'arg3',\r\n 'arg4': 'arg4',\r\n 'arg5': 'arg5',\r\n 'arg6': 'arg6',\r\n 'arg7': 'arg7',\r\n}\r\noverfull_data = correct_data.copy()\r\noverfull_data.update({str(i): i for i in range(5)})\r\n\r\nnumber = int(5*10e6)\r\n\r\nprint('Passing correct data')\r\na = timeit.timeit('InspectDeJson.de_json(correct_data)', globals=globals(), number=number)\r\nb = timeit.timeit('KwargsDeJson.de_json(correct_data)', globals=globals(), number=number)\r\nprint(f'InspectDeJson took {a/number} seconds on average')\r\nprint(f'KwargsDeJson took {b/number} seconds on average')\r\nprint(f'InspectDeJson took {a*100/b} % of KwargsDeJsons runtime')\r\n\r\nprint('\\n' + 20*\"=\" + '\\n')\r\n\r\n\r\nprint('Passing too much data')\r\na = timeit.timeit('InspectDeJson.de_json(overfull_data)', globals=globals(), number=number)\r\nb = timeit.timeit('KwargsDeJson.de_json(overfull_data)', globals=globals(), number=number)\r\nprint(f'InspectDeJson took {a/number} seconds on average')\r\nprint(f'KwargsDeJson took {b/number} seconds on average')\r\nprint(f'InspectDeJson took {a*100/b} % of KwargsDeJsons runtime')\r\n```\r\n\r\nExample output on my machine:\r\n\r\n```python\r\nInspectDeJson took 9.67620756e-07 seconds on average\r\nKwargsDeJson took 9.222072120000002e-07 seconds on average\r\nInspectDeJson took 104.92444034367406 % of KwargsDeJsons runtime\r\n\r\n====================\r\n\r\nPassing too much data\r\nInspectDeJson took 4.28629354e-06 seconds on average\r\nKwargsDeJson took 1.8550962000000004e-06 seconds on average\r\nInspectDeJson took 231.05505471899508 % of KwargsDeJsons runtime\r\n```\r\n\r\nSo in the \"usual\" case the overhead is neglectible IMO and the usage of inspect-dark magic is not too extensive.\r\nIn the cases where we have unhandled data, the slow down is heavily notiable - but we're still talking 4 vs 2 __micro__ seconds and as explained above, this shouldn't happen too often.\r\n\r\n## Summary\r\nFor today, much punchlines are:\r\n\r\n* Dismissal of additional kwargs can be handled in a different way, that works reasonably well IMO\r\n* Using 3rd party lib could reduce some more boilerplate code, specifically `to/de_json`, but neither `pydantic` nor `attrs` has all the features that I would like. Neither does `dataclass`, but at least that's included batteries\r\n\r\n## PS:\r\n\r\nPyCharm [apparently](https://youtrack.jetbrains.com/issue/PY-16760) still has no proper support for `dataclasses` - not that this would stop us, just mentioning it …","author":{"url":"https://github.com/Bibo-Joshi","@type":"Person","name":"Bibo-Joshi"},"datePublished":"2021-10-04T19:54:25Z","url":"https://github.com/python-telegram-bot/python-telegram-bot/issues/2698"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:b9488d98-48ae-b2c0-7711-0fffca3024ae |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | DD98:27E5CD:46FB1F8:5A89FE6:6B023AE5 |
| html-safe-nonce | a1e0333db99820b858e66dad005c39d6290b9abb3cbaede57b370a49e27b8e2e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJERDk4OjI3RTVDRDo0NkZCMUY4OjVBODlGRTY6NkIwMjNBRTUiLCJ2aXNpdG9yX2lkIjoiMjgxMzE4MzYwOTA3NzM4MTg2MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 07e7509df47d37da7df273b14cd504c954e88b32c7c1e2d3839a479aef970caf |
| hovercard-subject-tag | issue:1015570906 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python-telegram-bot/python-telegram-bot/2698/issue_layout |
| twitter:image | https://opengraph.githubassets.com/0e9f2bcf3b73baac661af997a5bacb8e93d94f3fb668d1b35cdaa3d3e8a2359c/python-telegram-bot/python-telegram-bot/issues/2698 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/0e9f2bcf3b73baac661af997a5bacb8e93d94f3fb668d1b35cdaa3d3e8a2359c/python-telegram-bot/python-telegram-bot/issues/2698 |
| og:image:alt | The idea to use dataclasses has been discussed a number of times (see #2056), more recently in #2491 and there is also a connection to #2607. So here I am opening an issue to write down some though... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Bibo-Joshi |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python-telegram-bot/python-telegram-bot git https://github.com/python-telegram-bot/python-telegram-bot.git |
| octolytics-dimension-user_id | 16178365 |
| octolytics-dimension-user_login | python-telegram-bot |
| octolytics-dimension-repository_id | 38696925 |
| octolytics-dimension-repository_nwo | python-telegram-bot/python-telegram-bot |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 38696925 |
| octolytics-dimension-repository_network_root_nwo | python-telegram-bot/python-telegram-bot |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width