René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"subprocess can block all Python threads when using `vfork()` until the child process `exec()` succeeds or fails.","articleBody":"# Background\r\n\r\nWhen using the `subprocess` module on Linux, `vfork()` rather than `fork()` has been preferred when possible for a few Python releases as is generally offers MUCH higher performance, the larger the parent process, the more cpu time it saves on page table copying.\r\n\r\n# Problem\r\n\r\n`vfork()` - by design - does not return control to parent process until after the child process has successfully performed an `exec()` system call or has died.\r\n\r\nPython `subprocess` uses `_posixsubprocess.fork_exec()` extension module implementation to do handle async-signal-safe fork/vfork+exec code path (as that cannot safely be implemented in Python). This function does not release the GIL. Thus the GIL is held across the `vfork()` call. Meaning the parent process does not resume execution until the exec has succeeded, or failed and returned an error which our child writes to it's `errpipe_write` fd before exiting.\r\n\r\nThis can result in **all Python threads hanging** (ie: the GIL is held by a blocked thread) when the child `exec` system call takes a long time. Such as the binary existing on a potentially slow or high latency network filesystem.  (We witnessed this on a FUSE filesystem backed by remote (high latency) cloud storage with a large executable... but any slow or high latency filesystem containing an executable or anything interfering in the exec system call can cause the same problem).\r\n\r\n# Workarounds\r\n\r\nBuilding without `vfork` support in `_posixsubprocess` or otherwise passing an arg to subprocess API that happens to disable vfork because it is incompatible with the vfork-concept are workarounds. _(I won't recommend any of those because none of them are a good idea to pass when not needed for their primary purpose)_.\r\n\r\nAnother workaround viable in our specific case is to pre-read the executable before asking subprocess to launch it as that moves the high IO latency from the `exec` system call into a read without the GIL held beforehand as it pulls the executable into a fast-enough local storage cache.  (That being useful at all depends on your particular filesystem latency implementation situation)\r\n\r\n# Potential Solutions\r\n\r\nIf we could simply release the GIL around the `vfork()` call that would seem to be ideal... It's not clear to me that we can actually do that, I'm working out what a PR would look like or what blocks us from doing so safely.\r\n\r\nOffering yet another keyword argument to subprocess APIs to disable the use of APIs that could block the entire process upon `exec()` such as `vfork` is viable - but doesn't feel great given our existing long list of subprocess args.\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-104515\n* gh-104518\n* gh-104697\n* gh-104782\n* gh-104785\n* gh-104831\n* gh-104833\n* gh-104942\n* gh-104958\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/gpshead","@type":"Person","name":"gpshead"},"datePublished":"2023-05-10T23:41:38.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/104372/cpython/issues/104372"}