René's URL Explorer Experiment


Title: gh-64192: Make `imap()`/`imap_unordered()` in `multiprocessing.pool` actually lazy by obaltian · Pull Request #136871 · python/cpython · GitHub

Open Graph Title: gh-64192: Make `imap()`/`imap_unordered()` in `multiprocessing.pool` actually lazy by obaltian · Pull Request #136871 · python/cpython

X Title: gh-64192: Make `imap()`/`imap_unordered()` in `multiprocessing.pool` actually lazy by obaltian · Pull Request #136871 · python/cpython

Description: Context recap (#64192) Let's consider that we have an input iterable and N = len(iterable). Current multiprocessing.Pool.imap and multiprocessing.Pool.imap_unordered are O(N) in space (unecessarily expensive on large iterables, completely impossible to use on infinite iterables): The call results: Iterator = pool.imap(func, iterable) iterates over all the elements of the iterable, submitting N tasks to the pool (results are collected into a list of size N). Following calls to next(results) take the oldest result from the list (FIFO) (waiting for it if not available yet) and return it. Proposal: add an optional buffersize param With this proposal, the call results: Iterator = pool.imap(func, iterable, buffersize=b) will iterate only over the first b elements of iterable (acquiring buffersize semaphore while iterating), submitting b tasks to worker threads and then will return the results iterator. Calls to next(results) will release buffersize semaphore (allowing task_handler thread to get the next input element from iterable to submit a new task to a worker thread) and then return the result. buffersize semaphores from iterators not exhausted yet are also being released on pool termination to avoid deadlocks. Benefits: The space complexity becomes O(b) When using a buffersize the client code takes back the control over the speed of iteration over the input iterable: after an initial spike of b calls to func to fill the buffer, the iteration over input iterable will follow the rate of the iteration over the results (controlled by the client), which is critical when func involves talking to services that you don't want to overload. Feature history buffersize support has been recently merged into concurrent.futures.Executor.map implementation (#125663) and many code/test/doc parts are based on ones from there to ensure consistency between modules. I want to thank authors of that PR for the references. Links: Docs preview Issue: gh-64192

Open Graph Description: Context recap (#64192) Let's consider that we have an input iterable and N = len(iterable). Current multiprocessing.Pool.imap and multiprocessing.Pool.imap_unordered are O(N) in space (unecessa...

X Description: Context recap (#64192) Let's consider that we have an input iterable and N = len(iterable). Current multiprocessing.Pool.imap and multiprocessing.Pool.imap_unordered are O(N) in space (unec...

Opengraph URL: https://github.com/python/cpython/pull/136871

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/commits/:range(.:format)
route-controllerpull_requests
route-actioncommits
fetch-noncev2:1feb0e6c-18d9-3102-c5d2-59907b5adb87
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idC7DE:A6578:73D9445:9E43115:69961872
html-safe-nonceed2a2a4817a6c6c5b91d845bcd17cad5f84df296b29fd64bc43ded0abbe91b97
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDN0RFOkE2NTc4OjczRDk0NDU6OUU0MzExNTo2OTk2MTg3MiIsInZpc2l0b3JfaWQiOiIzODczMDI4Mzk2MzUyNDExNzYyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacb20659b4687cd4ed0ef242abc06f9be76b0a81eb9fb9ce28ab86b565bdf3ecb6
hovercard-subject-tagpull_request:2679897101
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/commits
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
twitter:imagehttps://avatars.githubusercontent.com/u/49389276?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/49389276?s=400&v=4
og:image:altContext recap (#64192) Let's consider that we have an input iterable and N = len(iterable). Current multiprocessing.Pool.imap and multiprocessing.Pool.imap_unordered are O(N) in space (unecessa...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None7490a2fd98baca866acf503b81aa249491b42e5b4bab7f7a2801be81175c8845
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/python/cpython git https://github.com/python/cpython.git
octolytics-dimension-user_id1525981
octolytics-dimension-user_loginpython
octolytics-dimension-repository_id81598961
octolytics-dimension-repository_nwopython/cpython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id81598961
octolytics-dimension-repository_network_root_nwopython/cpython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release48a904441630875f5985de713035046dc65cc94c
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F136871%2Fcommits%2Fc941c16ddee10614a427e655d71b4974c8468c3d
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F136871%2Fcommits%2Fc941c16ddee10614a427e655d71b4974c8468c3d
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Fcommits&source=header-repo&source_repo=python%2Fcpython
Reloadhttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
Reloadhttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
Reloadhttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
python https://github.com/python
cpythonhttps://github.com/python/cpython
Please reload this pagehttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
Notifications https://github.com/login?return_to=%2Fpython%2Fcpython
Fork 34.1k https://github.com/login?return_to=%2Fpython%2Fcpython
Star 71.6k https://github.com/login?return_to=%2Fpython%2Fcpython
Code https://github.com/python/cpython
Issues 5k+ https://github.com/python/cpython/issues
Pull requests 2.1k https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects 31 https://github.com/python/cpython/projects
Security 0 https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Code https://github.com/python/cpython
Issues https://github.com/python/cpython/issues
Pull requests https://github.com/python/cpython/pulls
Actions https://github.com/python/cpython/actions
Projects https://github.com/python/cpython/projects
Security https://github.com/python/cpython/security
Insights https://github.com/python/cpython/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fpython%2Fcpython%2Fissues%2Fnew%2Fchoose
obaltianhttps://github.com/obaltian
python:mainhttps://github.com/python/cpython/tree/main
obaltian:feature/add-buffersize-to-multiprocessinghttps://github.com/obaltian/cpython/tree/feature/add-buffersize-to-multiprocessing
Conversation 27 https://github.com/python/cpython/pull/136871
Commits 23 https://github.com/python/cpython/pull/136871/commits
Checks 41 https://github.com/python/cpython/pull/136871/checks
Files changed https://github.com/python/cpython/pull/136871/files
Please reload this pagehttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
gh-64192: Make imap()/imap_unordered() in multiprocessing.pool actually lazy https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d#top
Show all changes 23 commits https://github.com/python/cpython/pull/136871/files
ccd0bbc draft: impl lazy input consumption in mp.Pool.imap(_unordered) Jul 20, 2025 https://github.com/python/cpython/pull/136871/commits/ccd0bbc67ab284fb2cf8db72cd1be4b8115b19c2
002ef46 Use semaphore to synchronize threads Jul 20, 2025 https://github.com/python/cpython/pull/136871/commits/002ef46d842dbbfea7098577cd60f6f35038cba7
6e0bc58 Update buffersize behavior to match concurrent.futures.Executor behavior Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/6e0bc58796fd7e5a82a3a3394489852d73402920
62b2b6a Release all `buffersize_lock` obj from the parent thread when terminate Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/62b2b6a9b50f3bdfb4a8930956ffa7c9afe2c29c
0b6ba41 Add 2 basic `ThreadPool.imap()` tests w/ and w/o buffersize Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/0b6ba419c226eea338ef56cc1e2d1f91199c883b
aade15e Fix accidental swap in imports Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/aade15e629ccbeae2d12818b0cf8cfc4e41b39bd
fb38a72 clear Pool._taskqueue_buffersize_semaphores safely Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/fb38a72056a2b1c1de4ee55e8063c85cadf8fed8
6ef488b Slightly optimize Pool._taskqueue_buffersize_semaphores terminate Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/6ef488b51446f5dd253962cb3f6ace0968649db9
1716725 Rename `Pool.imap()` buffersize-related tests Jul 21, 2025 https://github.com/python/cpython/pull/136871/commits/1716725d4af6a30a648ef27c02b5d523718f7866
9b43cd0 Fix typo in `IMapIterator.__init__()` Jul 22, 2025 https://github.com/python/cpython/pull/136871/commits/9b43cd086b38d3d39c9446ad8c0a07524c16d037
2d89341 Add tests for buffersize combinations with other kwargs Jul 22, 2025 https://github.com/python/cpython/pull/136871/commits/2d8934123e563379dc3b1ab578e62beae38c089f
9ab2705 Remove if-branch in `_terminate_pool` Jul 27, 2025 https://github.com/python/cpython/pull/136871/commits/9ab27055e3e5d71f8cd967baf0906bc1ecd5eab8
a955003 Add more edge-case tests for `imap` and `imap_unodered` Jul 27, 2025 https://github.com/python/cpython/pull/136871/commits/a95500340fe193e84a31c13168763737dbb7d4f7
80efd6e Split inf iterable test for `imap` and `imap_unordered` Jul 27, 2025 https://github.com/python/cpython/pull/136871/commits/80efd6e56ed45aed25211b15fca0ff591c32ac3c
83d6930 Add doc for `buffersize` argument of `imap` and `imap_unordered` Jul 27, 2025 https://github.com/python/cpython/pull/136871/commits/83d69306d1b6d0828c783e8072f4d4baf1ab4cd1
995ad8c add *versionadded* for `imap_unordered` Jul 28, 2025 https://github.com/python/cpython/pull/136871/commits/995ad8c298677e81bdb586ebefa7360fbdfe223e
3b6ad65 Remove ambiguity in `buffersize` description. Jul 28, 2025 https://github.com/python/cpython/pull/136871/commits/3b6ad65b71a93da3a770f6a4c79921aa14457a5c
c941c16 Set *versionadded* as next in docs Jul 28, 2025 https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
d09e891 Add whatsnew entry Jul 28, 2025 https://github.com/python/cpython/pull/136871/commits/d09e891d2f887691fc0543c4e437b05e26e943d8
9c6d89d Fix aggreed comments on code formatting/minor refactoring Jul 28, 2025 https://github.com/python/cpython/pull/136871/commits/9c6d89de5f66308882659fe6f614c3f66ad943e6
4550a01 Remove `imap` and `imap_unordered` body code duplication Jul 28, 2025 https://github.com/python/cpython/pull/136871/commits/4550a01bec9470991ce241a397ef5cf199aee3a1
77bde4d Merge branch 'main' into feature/add-buffersize-to-multiprocessing obaltian Aug 31, 2025 https://github.com/python/cpython/pull/136871/commits/77bde4d510388a513b255fb7f88a50529f1d3627
aec39fc Merge branch 'main' into feature/add-buffersize-to-multiprocessing obaltian Sep 3, 2025 https://github.com/python/cpython/pull/136871/commits/aec39fc45a8f07f38b3c9eb8fa4a0e22be70dec1
Clear filters https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
Please reload this pagehttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
Please reload this pagehttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
Prev https://github.com/python/cpython/pull/136871/commits/3b6ad65b71a93da3a770f6a4c79921aa14457a5c
Next https://github.com/python/cpython/pull/136871/commits/d09e891d2f887691fc0543c4e437b05e26e943d8
Please reload this pagehttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d
https://github.com/obaltian
obaltianhttps://github.com/python/cpython/commits?author=obaltian
https://github.com/python/cpython/blob/main/.github/CODEOWNERS#L527
Doc/library/multiprocessing.rsthttps://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d#diff-2cf07acf4f2d547e93e35c6a4ba2ee818e991b54e88da4a904ee0a49acb99eb3
View file https://github.com/python/cpython/blob/c941c16ddee10614a427e655d71b4974c8468c3d/Doc/library/multiprocessing.rst
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/python/cpython/pull/136871/commits/{{ revealButtonHref }}
https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d#diff-2cf07acf4f2d547e93e35c6a4ba2ee818e991b54e88da4a904ee0a49acb99eb3
https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d#diff-2cf07acf4f2d547e93e35c6a4ba2ee818e991b54e88da4a904ee0a49acb99eb3
https://github.com/python/cpython/pull/136871/commits/c941c16ddee10614a427e655d71b4974c8468c3d#diff-2cf07acf4f2d547e93e35c6a4ba2ee818e991b54e88da4a904ee0a49acb99eb3
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.