Title: bpo-29595: Expose max_queue_size in ThreadPoolExecutor by prayerslayer · Pull Request #143 · python/cpython · GitHub
Open Graph Title: bpo-29595: Expose max_queue_size in ThreadPoolExecutor by prayerslayer · Pull Request #143 · python/cpython
X Title: bpo-29595: Expose max_queue_size in ThreadPoolExecutor by prayerslayer · Pull Request #143 · python/cpython
Description: Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API calls; I had to move data from one S3 bucket to another (~150M objects). Contrary to what I expected the maximum size of the underlying queue doesn't have a non-zero value by default. Thus my process ended up consuming gigabytes of memory, because it put more items into the queue than the threads were able to work off: The queue just kept growing. (It ran on K8s and the pod was rightfully killed eventually.) Of course there ways to work around this. One could use more threads, to some extent. Or you could use your own queue with a defined maximum size. But I think it's more work for users of Python than necessary. So this pull request exposes a max_queue_size parameter for the ThreadPoolExecutor which will be forwarded to queue.Queue(). It defaults to 0, so backward-compatibility is ensured. I am happy to add tests if you'd give me further instructions. I would've done this already, but I'm largely unfamiliar with this project as well as the language 😅 I hope you find this as useful as I would and am looking forward to read your thoughts about it! https://bugs.python.org/issue29595
Open Graph Description: Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API ca...
X Description: Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS AP...
Opengraph URL: https://github.com/python/cpython/pull/143
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/checks(.:format) |
| route-controller | pull_requests |
| route-action | checks |
| fetch-nonce | v2:7b5041de-64dd-3369-c90e-6d184a2ae514 |
| current-catalog-service-hash | 87dc3bc62d9b466312751bfd5f889726f4f1337bdff4e8be7da7c93d6c00a25a |
| request-id | A9C4:3410DA:164E3B:1CC023:696AD724 |
| html-safe-nonce | 37cc97957658cfb902ba63cd66ceeb4c994cfc58d52933d463be708a7cb62c9e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBOUM0OjM0MTBEQToxNjRFM0I6MUNDMDIzOjY5NkFENzI0IiwidmlzaXRvcl9pZCI6IjEzMTkyODQ0MzQ4MDM2NDgzNiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | c37d6c7ed240ad576f4c0eb8f2c1e1dd01d9c68d7575f3304de1a13c7bc7d057 |
| hovercard-subject-tag | pull_request:106799067 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,checks,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/143/checks |
| twitter:image | https://avatars.githubusercontent.com/u/878512?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/878512?s=400&v=4 |
| og:image:alt | Hi! Please forgive this blunt pull request, I wanted to open an issue first and ask, but couldn't. The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API ca... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | c785f4ce187e9e7331257791b36ddee01625bb8e292a9b4fe2c16d4c006abf5d |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width full-width-p-0 |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | c718a376fcf780eb22089171adb84a543f660bf7 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width