René's URL Explorer Experiment

Title: bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe by izbyshev · Pull Request #11671 · python/cpython · GitHub

Open Graph Title: bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe by izbyshev · Pull Request #11671 · python/cpython

X Title: bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe by izbyshev · Pull Request #11671 · python/cpython

Description: When used to run a new executable image, fork() is not a good choice for process creation, especially if the parent has a large working set: fork() needs to copy page tables, which is slow, and may fail on systems where overcommit is disabled, despite that the child is not going to touch most of its address space. Currently, subprocess is capable of using posix_spawn() instead, which normally provides much better performance. However, posix_spawn() does not support many of child setup operations exposed by subprocess.Popen(). Most notably, it's not possible to express close_fds=True, which happens to be the default, via posix_spawn(). As a result, most users can't benefit from faster process creation, at least not without changing their code. However, Linux provides vfork() system call, which creates a new process without copying the address space of the parent, and which is actually used by C libraries to efficiently implement posix_spawn(). Due to sharing of the address space and even the stack with the parent, extreme care is required to use vfork(). At least the following restrictions must hold: No signal handlers must execute in the child process. Otherwise, they might clobber memory shared with the parent, potentially confusing it. Any library function called after vfork() in the child must be async-signal-safe (as for fork()), but it must also not interact with any library state in a way that might break due to address space sharing and/or lack of any preparations performed by libraries on normal fork(). POSIX.1 permits to call only execve() and _exit(), and later revisions remove vfork() specification entirely. In practice, however, almost all operations needed by subprocess.Popen() can be safely implemented on Linux. Due to sharing of the stack with the parent, the child must be careful not to clobber local variables that are alive across vfork() call. Compilers are normally aware of this and take extra care with vfork() (and setjmp(), which has a similar problem). In case the parent is privileged, special attention must be paid to vfork() use, because sharing an address space across different privilege domains is insecure[1]. This patch adds support for using vfork() instead of fork() on Linux when it's possible to do safely given the above. In particular: vfork() is not used if credential switch is requested. The reverse case (simple subprocess.Popen() but another application thread switches credentials concurrently) is not possible for pure-Python apps because subprocess.Popen() and functions like os.setuid() are mutually excluded via GIL. We might also consider to add a way to opt-out of vfork() (and posix_spawn() on platforms where it might be implemented via vfork()) in a future PR. vfork() is not used if preexec_fn != None. With this change, subprocess will still use posix_spawn() if possible, but will fallback to vfork() on Linux in most cases, and, failing that, to fork(). [1] https://ewontfix.com/7 https://bugs.python.org/issue35823

Open Graph Description: When used to run a new executable image, fork() is not a good choice for process creation, especially if the parent has a large working set: fork() needs to copy page tables, which is slow, and may...

X Description: When used to run a new executable image, fork() is not a good choice for process creation, especially if the parent has a large working set: fork() needs to copy page tables, which is slow, and may...

Opengraph URL: https://github.com/python/cpython/pull/11671

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width