Title: Faster, parallel type checking · Issue #933 · python/mypy · GitHub
Open Graph Title: Faster, parallel type checking · Issue #933 · python/mypy
X Title: Faster, parallel type checking · Issue #933 · python/mypy
Description: If we have a sufficiently large program to type check we should be able to speed up type checking significantly by using multiple type checker processes that work in parallel. One potential way to implement this: Have a coordinator proce...
Open Graph Description: If we have a sufficiently large program to type check we should be able to speed up type checking significantly by using multiple type checker processes that work in parallel. One potential way to ...
X Description: If we have a sufficiently large program to type check we should be able to speed up type checking significantly by using multiple type checker processes that work in parallel. One potential way to ...
Opengraph URL: https://github.com/python/mypy/issues/933
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Faster, parallel type checking","articleBody":"If we have a sufficiently large program to type check we should be able to speed up type checking significantly by using multiple type checker processes that work in parallel.\n\nOne potential way to implement this:\n- Have a coordinator process that maintains a module dependency graph and keeps track of a work list of modules that haven't been type checked. Initially this will contain all modules in the program.\n- Have multiple worker processes that repeatedly get (sets of cyclically dependent) modules from the coordinator that have their dependencies processed, type check them and mark them as processed on the coordinator.\n\nFor this to really work we are going to need a way of efficiently communicating type check results for a module between processes (to avoid type checking shared dependencies multiple times). Having a JSON serialization format (see #932) would be sufficient.\n\nAdditionally we need a quick way of figuring out the dependency graph of all modules (or at least an approximation of it). We'll probably have to cache that between runs and update it incrementally, similar to the approach outlined in #932.\n\nSo how much would this help? Obviously this depends on the shape of the dependency graph. Under reasonable assumptions we should be able to hit an effective level of parallelism of at least 4 for larger programs, but I wouldn't be surprised if we could get even better than that. Cyclic module dependencies can add a limit to how far we can parallelize. We can probably estimate the achievable level of parallelism for a particular program by analyzing the module dependency graph.\n\nThis is probably only worth implementing after we have incremental type checking (#932), and we should preserve incrementalism -- i.e., we'd only type check modules modified since the last run and modules that depend on them.\n","author":{"url":"https://github.com/JukkaL","@type":"Person","name":"JukkaL"},"datePublished":"2015-10-15T06:30:19Z","url":"https://github.com/python/mypy/issues/933"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:68cd45d0-0142-1f25-1ccf-1b82b8f02d22 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 92FC:D064:20B796B:2C5D5FD:6B001CD6 |
| html-safe-nonce | 6ca4108ffd870f3675d0e996f9e7a09d7787f6c5984dde37c16259332697c752 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MkZDOkQwNjQ6MjBCNzk2QjoyQzVENUZEOjZCMDAxQ0Q2IiwidmlzaXRvcl9pZCI6IjEyMjk1MDI1NzcxNzU5NjA3OTAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 2548c2947017142f0e226dccf74019de993e9685c1ff2fc063374089ccd9889e |
| hovercard-subject-tag | issue:111553782 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/mypy/933/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f4e327032416717bc20e98409e8d8bb678b4da6c1fd0a66223d40811d6d16f72/python/mypy/issues/933 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f4e327032416717bc20e98409e8d8bb678b4da6c1fd0a66223d40811d6d16f72/python/mypy/issues/933 |
| og:image:alt | If we have a sufficiently large program to type check we should be able to speed up type checking significantly by using multiple type checker processes that work in parallel. One potential way to ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | JukkaL |
| hostname | github.com |
| expected-hostname | github.com |
| None | 4eb4cba0d69f045374ac15de9666d22b0961e1d555cc3824f36b7734c00c78e0 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/mypy git https://github.com/python/mypy.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 7053637 |
| octolytics-dimension-repository_nwo | python/mypy |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 7053637 |
| octolytics-dimension-repository_network_root_nwo | python/mypy |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 44c658b24abf00491fb7ae5f56d42c9f38cbeab2 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width