René's URL Explorer Experiment


Title: bpo-29595: Expose max_queue_size in ThreadPoolExecutor by prayerslayer · Pull Request #143 · python/cpython · GitHub

Open Graph Title: bpo-29595: Expose max_queue_size in ThreadPoolExecutor by prayerslayer · Pull Request #143 · python/cpython

X Title: bpo-29595: Expose max_queue_size in ThreadPoolExecutor by prayerslayer · Pull Request #143 · python/cpython

Description: Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API calls; I had to move data from one S3 bucket to another (~150M objects). Contrary to what I expected the maximum size of the underlying queue doesn't have a non-zero value by default. Thus my process ended up consuming gigabytes of memory, because it put more items into the queue than the threads were able to work off: The queue just kept growing. (It ran on K8s and the pod was rightfully killed eventually.) Of course there ways to work around this. One could use more threads, to some extent. Or you could use your own queue with a defined maximum size. But I think it's more work for users of Python than necessary. So this pull request exposes a max_queue_size parameter for the ThreadPoolExecutor which will be forwarded to queue.Queue(). It defaults to 0, so backward-compatibility is ensured. I am happy to add tests if you'd give me further instructions. I would've done this already, but I'm largely unfamiliar with this project as well as the language 😅 I hope you find this as useful as I would and am looking forward to read your thoughts about it! https://bugs.python.org/issue29595

Open Graph Description: Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API ca...

X Description: Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS AP...

Opengraph URL: https://github.com/python/cpython/pull/143

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:5422a5e9-264e-1d37-b083-52addd4588ec
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idDDA8:3D95EB:4CC204:652074:696B4246
html-safe-nonce1aae0e5259af908a4e85a9c4fb8716416660ffa3c357dfd16ed8a6383bf4ae26
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEREE4OjNEOTVFQjo0Q0MyMDQ6NjUyMDc0OjY5NkI0MjQ2IiwidmlzaXRvcl9pZCI6IjQxOTUxNDQ2ODU3ODIyNTQxNTAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac6aef1f83611fb7d95cdf9d60ad43a58d7fea949424f9b068c255883f981931fe
hovercard-subject-tagpull_request:106799067
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/143/files
twitter:imagehttps://avatars.githubusercontent.com/u/878512?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/878512?s=400&v=4
og:image:altHi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API ca...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None5f99f7c1d70f01da5b93e5ca90303359738944d8ab470e396496262c66e60b8d
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release82560a55c6b2054555076f46e683151ee28a19bc
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/143/files#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F143%2Ffiles
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F143%2Ffiles
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/143/files
Reloadhttps://github.com/python/cpython/pull/143/files
Reloadhttps://github.com/python/cpython/pull/143/files
python https://github.com/python
cpythonhttps://github.com/python/cpython
Please reload this pagehttps://github.com/python/cpython/pull/143/files
Notifications https://github.com/login?return_to=%2Fpython%2Fcpython
Fork 33.9k https://github.com/login?return_to=%2Fpython%2Fcpython
Star 71.1k https://github.com/login?return_to=%2Fpython%2Fcpython
Code https://github.com/python/cpython
Issues 5k+ https://github.com/python/cpython/issues
Pull requests 2.1k https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects 31 https://github.com/python/cpython/projects
Security Uh oh! There was an error while loading. Please reload this page. https://github.com/python/cpython/security
Please reload this pagehttps://github.com/python/cpython/pull/143/files
Insights https://github.com/python/cpython/pulse
Code https://github.com/python/cpython
Issues https://github.com/python/cpython/issues
Pull requests https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects https://github.com/python/cpython/projects
Security https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
prayerslayerhttps://github.com/prayerslayer
python:masterhttps://github.com/python/cpython/tree/master
prayerslayer:expose-max-queue-size-in-threadpoolexecutorhttps://github.com/prayerslayer/cpython/tree/expose-max-queue-size-in-threadpoolexecutor
Conversation 46 https://github.com/python/cpython/pull/143
Commits 10 https://github.com/python/cpython/pull/143/commits
Checks 0 https://github.com/python/cpython/pull/143/checks
Files changed https://github.com/python/cpython/pull/143/files
Please reload this pagehttps://github.com/python/cpython/pull/143/files
bpo-29595: Expose max_queue_size in ThreadPoolExecutor https://github.com/python/cpython/pull/143/files#top
Show all changes 10 commits https://github.com/python/cpython/pull/143/files
66c0031 expose max_queue_size in threadpoolexecutor prayerslayer Feb 17, 2017 https://github.com/python/cpython/pull/143/commits/66c0031a08e6499486cc05db737ace0fb0a8de2d
693689c tests prayerslayer Feb 18, 2017 https://github.com/python/cpython/pull/143/commits/693689c74b7dcdfad2ea6d1a2dbdcee2160e9896
e76e9a7 wording prayerslayer Feb 18, 2017 https://github.com/python/cpython/pull/143/commits/e76e9a7a1b2b78d3374229bf139f0bbedce0e561
c2079a7 review fixes prayerslayer Feb 19, 2017 https://github.com/python/cpython/pull/143/commits/c2079a74f723c37d9b7b219d4343f3074422246c
6103533 update docs prayerslayer Feb 20, 2017 https://github.com/python/cpython/pull/143/commits/6103533b364c19c48f5d61eb1dbef8cfc71c9727
edcf3f9 fix whitespace prayerslayer Feb 27, 2017 https://github.com/python/cpython/pull/143/commits/edcf3f91871a4f2d1b589a61bea7f3feb178ba3a
cde3edd Merge branch 'master' into expose-max-queue-size-in-threadpoolexecutor Oct 30, 2017 https://github.com/python/cpython/pull/143/commits/cde3eddb2e27addd502ad34033d44102e8d0e967
f10a6f9 Document blocking behavior Oct 30, 2017 https://github.com/python/cpython/pull/143/commits/f10a6f9ed4ca2e9384d3d4e64d0da39724377ae0
2338991 Add highlevel test for threadpool queue size Oct 30, 2017 https://github.com/python/cpython/pull/143/commits/23389910bfffcdb83f6e8cfbe6adc044bd8a54c0
9d1a24b Add news entry Oct 30, 2017 https://github.com/python/cpython/pull/143/commits/9d1a24b837464323a0063a26a56e3b8383072860
Clear filters https://github.com/python/cpython/pull/143/files
Please reload this pagehttps://github.com/python/cpython/pull/143/files
Please reload this pagehttps://github.com/python/cpython/pull/143/files
concurrent.futures.rst https://github.com/python/cpython/pull/143/files#diff-eb5cd13d65d2f215d8dc0d95eb1a1341ff69b226ac56ec6099e39582edecc8cf
thread.py https://github.com/python/cpython/pull/143/files#diff-692c518556397eee9681b886c6c7e85af5defbd0759a01f0f39c355d48d1fd75
test_concurrent_futures.py https://github.com/python/cpython/pull/143/files#diff-4123e7430fd434e48220681097a0e5ddd00fe6f44ede7c096ee256704db0cb41
2017-10-30-22-18-28.bpo-29595.Zd7P54.rst https://github.com/python/cpython/pull/143/files#diff-4ccda9679024fcc4082cf18fc7f3cb06e4c6000ec1a99909371a62c3d7af0522
Doc/library/concurrent.futures.rsthttps://github.com/python/cpython/pull/143/files#diff-eb5cd13d65d2f215d8dc0d95eb1a1341ff69b226ac56ec6099e39582edecc8cf
View file https://github.com/prayerslayer/cpython/blob/9d1a24b837464323a0063a26a56e3b8383072860/Doc/library/concurrent.futures.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/143/{{ revealButtonHref }}
https://github.com/python/cpython/pull/143/files#diff-eb5cd13d65d2f215d8dc0d95eb1a1341ff69b226ac56ec6099e39582edecc8cf
pitrouhttps://github.com/pitrou
Oct 31, 2017https://github.com/python/cpython/pull/143/files#r147999718
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
https://github.com/python/cpython/pull/143/files#diff-eb5cd13d65d2f215d8dc0d95eb1a1341ff69b226ac56ec6099e39582edecc8cf
pitrouhttps://github.com/pitrou
Oct 31, 2017https://github.com/python/cpython/pull/143/files#r148000084
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
https://github.com/python/cpython/pull/143/files#diff-eb5cd13d65d2f215d8dc0d95eb1a1341ff69b226ac56ec6099e39582edecc8cf
https://github.com/python/cpython/pull/143/files#diff-eb5cd13d65d2f215d8dc0d95eb1a1341ff69b226ac56ec6099e39582edecc8cf
Lib/concurrent/futures/thread.pyhttps://github.com/python/cpython/pull/143/files#diff-692c518556397eee9681b886c6c7e85af5defbd0759a01f0f39c355d48d1fd75
View file https://github.com/prayerslayer/cpython/blob/9d1a24b837464323a0063a26a56e3b8383072860/Lib/concurrent/futures/thread.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/143/{{ revealButtonHref }}
https://github.com/python/cpython/pull/143/files#diff-692c518556397eee9681b886c6c7e85af5defbd0759a01f0f39c355d48d1fd75
https://github.com/python/cpython/pull/143/files#diff-692c518556397eee9681b886c6c7e85af5defbd0759a01f0f39c355d48d1fd75
DimitrisJimhttps://github.com/DimitrisJim
Feb 18, 2017https://github.com/python/cpython/pull/143/files#r101903897
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
prayerslayerhttps://github.com/prayerslayer
Feb 19, 2017https://github.com/python/cpython/pull/143/files#r101915492
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
https://github.com/python/cpython/pull/143/files#diff-692c518556397eee9681b886c6c7e85af5defbd0759a01f0f39c355d48d1fd75
Lib/test/test_concurrent_futures.pyhttps://github.com/python/cpython/pull/143/files#diff-4123e7430fd434e48220681097a0e5ddd00fe6f44ede7c096ee256704db0cb41
View file https://github.com/prayerslayer/cpython/blob/9d1a24b837464323a0063a26a56e3b8383072860/Lib/test/test_concurrent_futures.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/143/{{ revealButtonHref }}
https://github.com/python/cpython/pull/143/files#diff-4123e7430fd434e48220681097a0e5ddd00fe6f44ede7c096ee256704db0cb41
pitrouhttps://github.com/pitrou
Apr 1, 2017https://github.com/python/cpython/pull/143/files#r109280810
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
prayerslayerhttps://github.com/prayerslayer
Aug 23, 2017https://github.com/python/cpython/pull/143/files#r134702011
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
pitrouhttps://github.com/pitrou
Oct 31, 2017https://github.com/python/cpython/pull/143/files#r148000945
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
pitrouhttps://github.com/pitrou
Oct 31, 2017https://github.com/python/cpython/pull/143/files#r148000831
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/python/cpython/pull/143/files
https://github.com/python/cpython/pull/143/files#diff-4123e7430fd434e48220681097a0e5ddd00fe6f44ede7c096ee256704db0cb41
Misc/NEWS.d/next/Library/2017-10-30-22-18-28.bpo-29595.Zd7P54.rsthttps://github.com/python/cpython/pull/143/files#diff-4ccda9679024fcc4082cf18fc7f3cb06e4c6000ec1a99909371a62c3d7af0522
View file https://github.com/prayerslayer/cpython/blob/9d1a24b837464323a0063a26a56e3b8383072860/Misc/NEWS.d/next/Library/2017-10-30-22-18-28.bpo-29595.Zd7P54.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/143/{{ revealButtonHref }}
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.