René's URL Explorer Experiment


Title: Speedup `regr_test.py` by running test cases concurrently by AlexWaygood · Pull Request #10714 · python/typeshed · GitHub

Open Graph Title: Speedup `regr_test.py` by running test cases concurrently by AlexWaygood · Pull Request #10714 · python/typeshed

X Title: Speedup `regr_test.py` by running test cases concurrently by AlexWaygood · Pull Request #10714 · python/typeshed

Description: As we add more and more test cases, regr_test.py is getting kinda slow. Other than mypy_primer, it's now the slowest CI job we have by a long way (and we don't run mypy_primer on all PRs -- for example, it's skipped on this PR!). The reason for the slowness is that we now have regression tests for 11 stubs packages. We run all regression tests on Python 3.8-12 inclusive, and we run them all on linux, darwin and win32. That adds up for a total of 165 subprocesses that are created for each run of the test when it's run with --all (the flag we use in CI). At some point we may want to consider sharding this test between GitHub Actions workers, similar to the way we run mypy_test.py and pyright in CI. (We can also possibly reconsider whether we need to, e.g., run all tests on darwin, linux and Windows). For now, though, we can speed things up a lot just by running the subprocess concurrently using a ProcessPoolExecutor. This cuts the execution time in CI roughly in half, from around 5-6 minutes to around 2-3 minutes. (N.B.: A ProcessPoolExecutor feels like a slightly blunt instrument here with a lot of overhead; I'm sure there are more efficient ways of spawning subprocesses concurrently. I got this to work reasonably quickly, however, and when I tried different approaches I quickly ran into race conditions. I think this is ~good enough for now.)

Open Graph Description: As we add more and more test cases, regr_test.py is getting kinda slow. Other than mypy_primer, it's now the slowest CI job we have by a long way (and we don't run mypy_primer on all PRs --...

X Description: As we add more and more test cases, regr_test.py is getting kinda slow. Other than mypy_primer, it's now the slowest CI job we have by a long way (and we don't run mypy_primer on al...

Opengraph URL: https://github.com/python/typeshed/pull/10714

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:531d5683-0146-f449-caec-4cb235250aaa
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idD9D4:144E47:2A15DE8:3732913:6B000C38
html-safe-nonce77f8c5c8584fe7b2d53f50672c8c295823579b37bb71da65fc4cdce22ab55d12
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEOUQ0OjE0NEU0NzoyQTE1REU4OjM3MzI5MTM6NkIwMDBDMzgiLCJ2aXNpdG9yX2lkIjoiNzA4Nzc4Njk2MTg4ODM0OTI0MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacf652f7e3f91ff29b7ae18d35192cd45681f013c60f36181f3d92173b586651ad
hovercard-subject-tagpull_request:1517442387
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/typeshed/pull/10714/files
twitter:imagehttps://avatars.githubusercontent.com/u/66076021?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/66076021?s=400&v=4
og:image:altAs we add more and more test cases, regr_test.py is getting kinda slow. Other than mypy_primer, it's now the slowest CI job we have by a long way (and we don't run mypy_primer on all PRs --...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None0a0d8686a27478df474dd7415776a7a080cde8838b3cd92ea7601187572e5729
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/typeshed git https://github.com/python/typeshed.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id31696383
octolytics-dimension-repository_nwopython/typeshed
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id31696383
octolytics-dimension-repository_network_root_nwopython/typeshed
turbo-body-classeslogged-out env-production page-responsive
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release16a11879678aaf97e2a42c33ee70830aa084fa37
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/typeshed/pull/10714/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Ftypeshed%2Fpull%2F10714%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Ftypeshed%2Fpull%2F10714%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Ftypeshed
Reloadhttps://github.com/python/typeshed/pull/10714/files
Reloadhttps://github.com/python/typeshed/pull/10714/files
Reloadhttps://github.com/python/typeshed/pull/10714/files
pythonhttps://github.com/python
typeshedhttps://github.com/python/typeshed
Codehttps://github.com/python/typeshed
Issues196 (196)https://github.com/python/typeshed/issues
Pull requests251 (251)https://github.com/python/typeshed/pulls
Actionshttps://github.com/python/typeshed/actions
Security and qualityhttps://github.com/python/typeshed/security
Insightshttps://github.com/python/typeshed/insights
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Ftypeshed%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Ftypeshed%2Fissues%2Fnew%2Fchoose
AlexWaygoodhttps://github.com/AlexWaygood
python:mainhttps://github.com/python/typeshed/tree/main
AlexWaygood:optimise-regrtesthttps://github.com/AlexWaygood/typeshed/tree/optimise-regrtest
Conversation 8 https://github.com/python/typeshed/pull/10714
Commits 7 https://github.com/python/typeshed/pull/10714/commits
Checks 0 https://github.com/python/typeshed/pull/10714/checks
Files changed https://github.com/python/typeshed/pull/10714/files
Please reload this pagehttps://github.com/python/typeshed/pull/10714/files
Speedup regr_test.py by running test cases concurrently https://github.com/python/typeshed/pull/10714/files#top
Show all changes 7 commits https://github.com/python/typeshed/pull/10714/files
753ca80 Speedup `regr_test.py` by running test cases concurrently AlexWaygood Sep 15, 2023 https://github.com/python/typeshed/pull/10714/commits/753ca80a4fa96b39161ba956f441c264f65297eb
6db4d90 Deliberately break some test cases so you can see what it looks like … AlexWaygood Sep 15, 2023 https://github.com/python/typeshed/pull/10714/commits/6db4d908d08b1f6e9e8c0385133412262dd4a39d
af193c3 [pre-commit.ci] auto fixes from pre-commit.com hooks pre-commit-ci[bot] Sep 15, 2023 https://github.com/python/typeshed/pull/10714/commits/af193c3b9f927894deff79b9b1c5c9bc48fc18bb
3aa9cb4 Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" AlexWaygood Sep 15, 2023 https://github.com/python/typeshed/pull/10714/commits/3aa9cb4cbf73674fdf2ab474f79e598ec7681b75
19438a5 Revert "Deliberately break some test cases so you can see what it loo… AlexWaygood Sep 15, 2023 https://github.com/python/typeshed/pull/10714/commits/19438a5563cdbd6445b35678a31dc2e450db5838
d53781f Use threading instead AlexWaygood Sep 23, 2023 https://github.com/python/typeshed/pull/10714/commits/d53781fd152b8428a51a693b350fd7f3bf975448
e863f98 don't need that any more AlexWaygood Sep 23, 2023 https://github.com/python/typeshed/pull/10714/commits/e863f98330f88b51e47e041e05a94340a5b93738
Clear filters https://github.com/python/typeshed/pull/10714/files
Please reload this pagehttps://github.com/python/typeshed/pull/10714/files
Please reload this pagehttps://github.com/python/typeshed/pull/10714/files
tests/regr_test.pyhttps://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
View file https://github.com/AlexWaygood/typeshed/blob/e863f98330f88b51e47e041e05a94340a5b93738/tests/regr_test.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/typeshed/pull/10714/{{ revealButtonHref }}
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
Avasamhttps://github.com/Avasam
Dec 29, 2024https://github.com/python/typeshed/pull/10714/files#r1899045425
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
@AlexWaygoodhttps://github.com/AlexWaygood
Please reload this pagehttps://github.com/python/typeshed/pull/10714/files
AlexWaygoodhttps://github.com/AlexWaygood
Dec 29, 2024https://github.com/python/typeshed/pull/10714/files#r1899045505
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/typeshed/pull/10714/files
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com/python/typeshed/pull/10714/files#diff-d5ea625983c1cf96baeefa5e6f0ebc0f9aa3e1825ccef453db31938b00f557a7
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.