Title: Feature: Reduce Mypy's Cache Size · Issue #15731 · python/mypy · GitHub
Open Graph Title: Feature: Reduce Mypy's Cache Size · Issue #15731 · python/mypy
X Title: Feature: Reduce Mypy's Cache Size · Issue #15731 · python/mypy
Description: Reducing Mypy's Cache Size This is a meta-issue talking about different ways to reduce Mypy's cache size. I've been working on a branch in my free time, though it is probably too big for a PR, so I thought it would be best to get everyon...
Open Graph Description: Reducing Mypy's Cache Size This is a meta-issue talking about different ways to reduce Mypy's cache size. I've been working on a branch in my free time, though it is probably too big for a PR, so I...
X Description: Reducing Mypy's Cache Size This is a meta-issue talking about different ways to reduce Mypy's cache size. I've been working on a branch in my free time, though it is probably too big fo...
Opengraph URL: https://github.com/python/mypy/issues/15731
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feature: Reduce Mypy's Cache Size","articleBody":"## Reducing Mypy's Cache Size\r\n\r\nThis is a meta-issue talking about different ways to reduce Mypy's cache size. I've been working on a [branch](https://github.com/dosisod/mypy/tree/shrink-cache) in my free time, though it is probably too big for a PR, so I thought it would be best to get everyone's opinion on which optimizations (if any) would be worth-while.\r\n\r\nIn short, I've reduced Mypy's filesystem cache by about 27% using a few different techniques.\r\n\r\nHere is a breakdown of each of the commits, what I did to reduce the cache size, and by how much. We probably don't need to include all of these techniques since a lot of them only marginally reduce the cache size. The cache I've been using as a comparison is Mypy's own cache when checking itself.\r\n\r\n## The Numbers\r\n\r\n| Commit | Total Size | Diff | Percent \\* | Optimization Technique |\r\n|-------------|------------|------:|--------:|------------------------|\r\n| `master` | 26.9MB | | | |\r\n| `763a94d5e` | 26.5MB | 0.4MB | 1.6% | Use ints instead of bools for certain truthy/falsey values |\r\n| `623266f47` | 24.2MB | 2.2MB | 8.4% | Don't store fields that are nullable |\r\n| `6cefbfb27` | 23.4MB | 0.8MB | 3.0% | Drop `builtins.` prefix for common types |\r\n| `27e9e0d56` | 23.4MB | 47KB | 0.2% | Don't cache `builtins.object` in MRO because everything derives from it |\r\n| `d2d0aa005` | 22.2MB | 1.2MB | 4.5% | Don't store empty list/tuples in TypeInfo classes |\r\n| `cce01a60f` | 22.1MB | 17KB | 0.1% | Don't store empty symbol tables |\r\n| `88b5b6a3d` | 21.3MB | 0.8MB | 3.0% | Don't store `arg_names` and `arg_kinds` for func defs with type info |\r\n| `4275d51b5` | 21.1MB | 0.2MB | 0.8% | Store LDEF/GDEF/MDEF as ints |\r\n| `0a4dfeab2` | 19.9MB | 1.2MB | 4.4% | Replace `.class` key with empty string |\r\n| `abf66951c` | 19.9MB | 0.1MB | 0.5% | Store `NoneType` node as `\"None\"` string literal |\r\n| `22b91d94d` | 19.6MB | 0.1MB | 0.6% | Optimize `def_extras` usage |\r\n| `98c4ce817` | 19.6MB | 5KB | 0.0% | Don't store `Instance` args if they're empty |\r\n| | | | | |\r\n| Total | | 7.0MB | 27.0% | |\r\n\r\n> \\* Percent is calculated based on the original cache size of the `master` branch\r\n\r\nThe best techniques based on total savings are:\r\n\r\n* Don't store fields that are nullable (8.4%)\r\n* Don't store empty list/tuples in TypeInfo classes (4.5%)\r\n* Replace `.class` key with empty string (4.4%)\r\n* Drop `builtins.` prefix for common types (3.0%)\r\n* Don't store `arg_names` and `arg_kinds` for func defs with type info (3.0%)\r\n\r\nAfter that point everything starts to drop off, though they still might be worth including.\r\n\r\n## Backwards Compatibility\r\n\r\nI made sure to check that my changes would not break backwards compatibility. These new techniques allow for loading of both old and new caches, but of course the new cache format will be used when cached files needs to be rebuild.\r\n\r\n## Why?\r\n\r\nI was poking around the cache folder to see why the cache was the size it was, and I noticed that there was a lot of optimizations that could be made. In theory, smaller caches are quicker to save and load, while taking up less space on the users computer. In CI systems where storage space is metered, having a smaller cache size will speed up CI workflows and use up less cloud storage.\r\n\r\nLet me know if this is something you guys would be interested in! If so I'll start splitting this into separate PR's. Thanks!","author":{"url":"https://github.com/dosisod","@type":"Person","name":"dosisod"},"datePublished":"2023-07-21T05:23:40Z","url":"https://github.com/python/mypy/issues/15731"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:c473ee12-61f2-6041-1905-ad50a95abeed |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | CBA6:22FAD4:45D7A9:5D9936:6B002C8B |
| html-safe-nonce | 84b265e664f5b4bd4a732a1659d00b1f72b50588409b781d673530df48400462 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQkE2OjIyRkFENDo0NUQ3QTk6NUQ5OTM2OjZCMDAyQzhCIiwidmlzaXRvcl9pZCI6IjQxNjk4OTc2OTk4MTE3OTgxNTUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 0eb61afb462cc12ff1b53f4907df15211d8bf242ca1c81c997f01278d7c54296 |
| hovercard-subject-tag | issue:1815173691 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/mypy/15731/issue_layout |
| twitter:image | https://opengraph.githubassets.com/7224fa26e60feb9e5fc8e9ecb546c5a7526012fca1c6f234b452fb168519b959/python/mypy/issues/15731 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/7224fa26e60feb9e5fc8e9ecb546c5a7526012fca1c6f234b452fb168519b959/python/mypy/issues/15731 |
| og:image:alt | Reducing Mypy's Cache Size This is a meta-issue talking about different ways to reduce Mypy's cache size. I've been working on a branch in my free time, though it is probably too big for a PR, so I... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | dosisod |
| hostname | github.com |
| expected-hostname | github.com |
| None | 7d0829bbde9e48b2486cb5dd6d147896eb01d32134627091f818958a935ef3f0 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/mypy git https://github.com/python/mypy.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 7053637 |
| octolytics-dimension-repository_nwo | python/mypy |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 7053637 |
| octolytics-dimension-repository_network_root_nwo | python/mypy |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | f3f3a43ca095cdea3abbb394112e971f9fdb94fa |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width