René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add concurrent.futures.InterpreterPoolExecutor","articleBody":"# Feature or enhancement\r\n\r\n### Proposal:\r\n\r\nWhile we wait on [PEP 734](https://peps.python.org/pep-0734/), there's nothing blocking us from adding a new executor for multiple interpreters.  Doing so would allow people to start trying them out.  (I wish I had thought of this for 3.13!)\r\n\r\nMy planned design:\r\n\r\n* mostly built on top of `ThreadPoolExecutor`\r\n* callables (and args) passed as initializer or to `submit()` or `map()` will be pickled (same as `ProcessPoolExecutor`)\r\n* initializer and `submit()` arg can be a script instead of a callable (a script for `map()` doesn't make sense since there are no args)\r\n* `__init__()` will take a new \"shared\" arg that corresponds to what `Interpreter.prepare_main()` takes in PEP 734, to allow each worker interpreter to share some data\r\n\r\nIn the future we could support a more efficient \"sharing\" scheme than pickle.  When (or if) PEP 734 is accepted, we can refactor the executor to use the `interpreters` module rather than using `_interpreters` directly.\r\n\r\nOpen questions:\r\n\r\n* reconstitute and raise uncaught exceptions from running in the interpreter (like we do for `ProcessPoolExecutor`)?\r\n* (optionally?) exec functions against __main__ instead of the their original module, i.e. use the body of the function as a script?\r\n\r\nCC @brianquinlan\r\n\r\n### Has this already been discussed elsewhere?\r\n\r\nThis is a minor feature, which does not need previous discussion elsewhere\r\n\r\n### Links to previous discussion of this feature:\r\n\r\n_No response_\r\n\r\n\u003c!-- gh-linked-prs --\u003e\r\n### Linked PRs\r\n* gh-124548\r\n* gh-125708\n* gh-134321\n* gh-140468\n* gh-140605\n\u003c!-- /gh-linked-prs --\u003e\r\n","author":{"url":"https://github.com/ericsnowcurrently","@type":"Person","name":"ericsnowcurrently"},"datePublished":"2024-09-27T18:19:25.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/124694/cpython/issues/124694"}