Title: gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. by zitterbewegung · Pull Request #103927 · python/cpython · GitHub
Open Graph Title: gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. by zitterbewegung · Pull Request #103927 · python/cpython
X Title: gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. by zitterbewegung · Pull Request #103927 · python/cpython
Description: Summary: This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel. After porting we can see the direct impact on a multicore system. Without this change: Running make test is 5 min 26 seconds With this change: Running make test takes 3 min 39 seconds The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant. Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests. Issue: gh-82054
Open Graph Description: Summary: This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests other...
X Description: Summary: This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests other...
Opengraph URL: https://github.com/python/cpython/pull/103927
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/checks(.:format) |
| route-controller | pull_requests |
| route-action | checks |
| fetch-nonce | v2:3df8fafd-1b9f-95ed-56d2-ad95a0e6f04a |
| current-catalog-service-hash | 87dc3bc62d9b466312751bfd5f889726f4f1337bdff4e8be7da7c93d6c00a25a |
| request-id | A9D0:CADE5:FE838A:1544768:6969A2FC |
| html-safe-nonce | 678ce09d57ec8703cc63562a5bfe76501c3ddf9c1aa3945aaccfd5e5a201c382 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBOUQwOkNBREU1OkZFODM4QToxNTQ0NzY4OjY5NjlBMkZDIiwidmlzaXRvcl9pZCI6IjY1NTg1NTU1ODk3OTU0ODg1MDgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 31484bb0423422ba9f8eed7da1877af314e71c8e445a6f94de2073d7e8e63e80 |
| hovercard-subject-tag | pull_request:1330589344 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,checks,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/103927/checks |
| twitter:image | https://avatars.githubusercontent.com/u/30265?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/30265?s=400&v=4 |
| og:image:alt | Summary: This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests other... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 24c4c97a2d520cb286b35e1a4c22d7a4df3c26a2fa28dd7cdf0e65db327b4de7 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width full-width-p-0 |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 124667f43168afb6c9c03b7c02eb5b1d2e1be3d9 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width