Title: ABC causing massive memory swell · Issue #94284 · python/cpython · GitHub
Open Graph Title: ABC causing massive memory swell · Issue #94284 · python/cpython
X Title: ABC causing massive memory swell · Issue #94284 · python/cpython
Description: Bug report from abc import ABC, ABCMeta from datetime import datetime abcclasses = set() normalclasses = set() for i in range(10000): abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {})) normalclasses.add(type("normal_"+str(i), (object,),...
Open Graph Description: Bug report from abc import ABC, ABCMeta from datetime import datetime abcclasses = set() normalclasses = set() for i in range(10000): abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {})) normalclass...
X Description: Bug report from abc import ABC, ABCMeta from datetime import datetime abcclasses = set() normalclasses = set() for i in range(10000): abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {})) n...
Opengraph URL: https://github.com/python/cpython/issues/94284
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"ABC causing massive memory swell","articleBody":"\u003c!--\r\n If you're new to Python and you're not sure whether what you're experiencing is a bug, the CPython issue tracker is not\r\n the right place to seek help. Consider the following options instead:\r\n\r\n - reading the Python tutorial: https://docs.python.org/3/tutorial/\r\n - posting in the \"Users\" category on discuss.python.org: https://discuss.python.org/c/users/7\r\n - emailing the Python-list mailing list: https://mail.python.org/mailman/listinfo/python-list\r\n - searching our issue tracker (https://github.com/python/cpython/issues) to see if\r\n your problem has already been reported\r\n-->\r\n\r\n**Bug report**\r\n```py\r\nfrom abc import ABC, ABCMeta\r\nfrom datetime import datetime\r\n\r\n\r\nabcclasses = set()\r\n\r\nnormalclasses = set()\r\n\r\nfor i in range(10000):\r\n abcclasses.add(ABCMeta(\"abc_\"+str(i), (ABC, ), {}))\r\n normalclasses.add(type(\"normal_\"+str(i), (object,), {}))\r\n\r\n\r\nif __name__ == '__main__':\r\n\r\n starttime = datetime.now()\r\n import os, psutil\r\n process = psutil.Process(os.getpid())\r\n mb = 1024 * 1024\r\n mem = last = process.memory_info().rss\r\n print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {\"━\" * int(mem / 8_000_000)}')\r\n for item in normalclasses:\r\n issubclass(item, ABC)\r\n\r\n mem = process.memory_info().rss\r\n print(f'{i + 1:>4d} {mem / mb:8.2f}MB {(mem - last) / mb:+8.2f}MB | {\"━\" * int(mem / 8_000_000)}')\r\n print(f\"This took {datetime.now()-starttime}\")\r\n```\r\n\r\n**Your environment**\r\n\r\n\u003c!-- Include as many relevant details as possible about the environment you experienced the bug in -->\r\n\r\nRunning this takes several minutes and consumes over 20 gbs of memory. \r\n\r\nI've checked it against python 3.8-3.10. The issue is just that ABC caches every single issubclass evaluation combined with it searching its entire subclass tree recursively causes it to grind to a halt if you have any significant number of subclass 10k sub classes of ABC and 10k normal classes being compared to the base ABC climbs to over 20 GBs. I am not really sure it is necessary to check the entire subclass tree. I presume it is because of concerns about register calls happening low in the class tree. \r\n\r\nIt would likely be far more efficient to chase upward the calls to register and place those in the parent classes then to search the entire class hierarchy. This can get really aggressive performance issues if you have any multiple inheritance structures going on as well as those will get checked multiple times. \r\n\r\nSee these issues. \r\nhttps://github.com/python/cpython/issues/92810\r\nhttps://github.com/samuelcolvin/pydantic/issues/3829\r\n\r\n","author":{"url":"https://github.com/zrothberg","@type":"Person","name":"zrothberg"},"datePublished":"2022-06-26T06:04:21Z","url":"https://github.com/python/cpython/issues/94284"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:463fa493-6e5b-ee75-7077-6c025ff48549 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | C77C:97D39:ABDF92:E332F5:6B01EEE2 |
| html-safe-nonce | 199535694a15d4dc985aae985fe33fca7c6cae103708912d22fbe62b1663d081 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDNzdDOjk3RDM5OkFCREY5MjpFMzMyRjU6NkIwMUVFRTIiLCJ2aXNpdG9yX2lkIjoiMTYzMjQzNDQxMDA2NDMwOTk4NiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 8b70cb0903f104c077d3c13e1e786216c7b8c05824eb1e5d2cdf4c5425473b39 |
| hovercard-subject-tag | issue:1284816843 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/94284/issue_layout |
| twitter:image | https://opengraph.githubassets.com/53d5a8806797f78666818aa5a41c456bac7ed31842dd6daf71885d9f0e37a92d/python/cpython/issues/94284 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/53d5a8806797f78666818aa5a41c456bac7ed31842dd6daf71885d9f0e37a92d/python/cpython/issues/94284 |
| og:image:alt | Bug report from abc import ABC, ABCMeta from datetime import datetime abcclasses = set() normalclasses = set() for i in range(10000): abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {})) normalclass... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | zrothberg |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width