Title: In multiple threads applications, using fork results in occasional blocking rather than occasional hanging up. · Issue #111635 · python/cpython · GitHub
Open Graph Title: In multiple threads applications, using fork results in occasional blocking rather than occasional hanging up. · Issue #111635 · python/cpython
X Title: In multiple threads applications, using fork results in occasional blocking rather than occasional hanging up. · Issue #111635 · python/cpython
Description: Bug report Bug description: In multi-threaded applications, using fork results in blocking rather than hanging up. We have a multi-threaded application, and in order to interact with scripts, we use a self-developed component similar to ...
Open Graph Description: Bug report Bug description: In multi-threaded applications, using fork results in blocking rather than hanging up. We have a multi-threaded application, and in order to interact with scripts, we us...
X Description: Bug report Bug description: In multi-threaded applications, using fork results in blocking rather than hanging up. We have a multi-threaded application, and in order to interact with scripts, we us...
Opengraph URL: https://github.com/python/cpython/issues/111635
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"In multiple threads applications, using fork results in occasional blocking rather than occasional hanging up.","articleBody":"# Bug report\n\n### Bug description:\n\n# In multi-threaded applications, using fork results in blocking rather than hanging up.\n\nWe have a multi-threaded application, and in order to interact with scripts, we use a self-developed component similar to [[pexpect](https://github.com/pexpect/pexpect)](https://github.com/pexpect/pexpect). issue: https://github.com/pexpect/ptyprocess/issues/50, https://github.com/pexpect/pexpect/issues/369. The underlying implementation uses os.fork(). According to related issues, this can occasionally lead to unexpected deadlocks, as discussed in https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555. However, our scenario is quite interesting. In our case, the child process gets blocked, and it doesn’t return until after a timeout of 30 seconds. Additionally, this slowness manifests suddenly and persistently from a black box perspective. Normally, it should take less than 0.1 seconds, but in problematic environments, it can persist for more than 20 seconds, and even up to 30 seconds, eventually causing the child process to be killed due to a timeout. Alternatively, the process may execve to a root process, becoming unkillable and turning into a zombie process. The code and phenomena are as follows:\n\nCode:\n\nInitializing an expect instance for command execution:\n\n~~~python\ndef build(self, command, first_require=\"Username:\",\n second_require=\"Password:\", timeout=72000):\n self.logger.info('get cmd: %s', command)\n process = expect.Spawn(command)\n try:\n idx = process.expect([first_require], 30)\n if idx == 0:\n process.send_line(self.adm_user)\n else:\n self.logger.error(\"excute zenithctl output: %s\" % process.child_process_output)\n self.error_message = process.child_process_output\n self.close(process)\n return False\n ```\n return False\n except Exception:\n self.logger.info(traceback.format_exc())\n self.logger.info(\"run command failed\")\n return False\n finally:\n self.close(process)\n~~~\n\nExpect implementation, the main tasks performed by the child process include: Setting the window size, setting echo, closing file descriptors, and executing execve.\n\n```python\ndef _do_spawn(self):\n if not self.execute_file:\n raise ExpectException('Execute file is not selected')\n\n try:\n child_pid, child_fd = pty.fork()\n except OSError:\n raise ExpectException('Execute pty fork failed.')\n\n if child_pid == pty.CHILD:\n try:\n # Set window size\n try:\n set_win_size_method = getattr(termios, 'TIOCSWINSZ', -2146929561)\n windows_size = struct.pack('HHHH', 1920, 1080, 0, 0)\n fcntl.ioctl(sys.stdin.fileno(), set_win_size_method, windows_size)\n except IOError as io_error:\n if io_error.args[0] not in (errno.EINVAL, errno.ENOTTY):\n raise ExpectException('Set window size exception')\n\n # Set echo\n try:\n attr = termios.tcgetattr(sys.stdin.fileno())\n attr[3] = attr[3] & ~termios.ECHO\n termios.tcsetattr(sys.stdin.fileno(), termios.TCSANOW, attr)\n except termios.error as termi_err:\n if termi_err.args[0] == errno.EINVAL:\n raise IOError(termi_err.args[0], '%s: Set echo failed.' % termi_err.args[1])\n raise ExpectException('Set echo failed.')\n except IOError as io_error:\n if io_error.args[0] == errno.EINVAL:\n raise IOError(io_error.args[0], '%s: Set echo failed.' % io_error.args[1])\n raise ExpectException('Set echo failed.')\n\n # Close unnecessary open files\n max_file_no = resource.getrlimit(resource.RLIMIT_NOFILE)[0]\n os.closerange(3, max_file_no)\n signal.signal(signal.SIGHUP, signal.SIG_IGN)\n\n # Execute command line\n os.execv(self.execute_file, self.execute_command_args)\n except BaseException:\n # Exit child process in case of any exception\n sys.exit(1)\n else:\n # Handle parent process if needed\n self._process_parent_process(child_pid, child_fd)\n```\n\nThe `process.expect([first_require], 30)` method utilizes selectors and epoll for implementation, waiting for the child process to produce output.\n\n\n\nThe `close` function in `expect`.\n\n```python\ndef close(self):\n if self.is_child_process_closed:\n return\n \n os.close(self.child_fd)\n if self.sel:\n self.sel.close()\n\n time.sleep(self.close_child_process_delay)\n while True:\n # Check if the child process is still alive\n if self.child_process_terminated:\n break\n waitpid_options = 0 if self.child_process_read_eof else os.WNOHANG\n try:\n process_id, process_status = os.waitpid(self.child_pid, waitpid_options)\n if process_id == 0:\n continue\n if os.WIFEXITED(process_status):\n self.child_process_status = os.WEXITSTATUS(process_status)\n elif os.WIFSIGNALED(process_status):\n self.child_process_status = os.WTERMSIG(process_status)\n elif os.WIFSTOPPED(process_status):\n raise ExpectException('Process already been stopped.')\n self.child_process_terminated = True\n break\n except OSError as e:\n if e.errno == errno.ECHILD:\n raise ExpectException('Process already not exist.')\n else:\n raise\n break\n\n if not self.child_process_terminated:\n # Try to terminate the child process\n kill_signals = [signal.SIGHUP, signal.SIGCONT, signal.SIGINT, signal.SIGKILL]\n for _signal in kill_signals:\n try:\n os.kill(self.child_pid, _signal)\n except OSError as e:\n if e.errno != errno.ESRCH:\n time.sleep(self.terminate_child_process_delay)\n continue\n break\n\n self._set_closed_status()\n```\n\n\n\ncommand is :\n\n```shell\n/usr/bin/sudo -u xxx /bin/bash -c \"xxx\"\n```\n\nPhenomenon:\n\n- The logs for the exception branch \"execute zenithctl output:\" and the entrance log \"get cmd:\" are just over 30 seconds apart.\n- In the failure scenario, there is no record in the sudo logs.\n- Calling build twice in a ten-second interval, but seeing the output in the sudo logs within the same second. Additionally, the call time and the time recorded in the sudo logs are all over 10 seconds, even reaching 20 or 30 seconds.\n- In the sudo logs, some entries have TTY=unknown while others have TTY=pts/0.\n- The script that was supposed to be executed did not run.\n- The issue occurred in multiple live environments without any obvious pattern and couldn't be reproduced by us.\n- Retrying in the failing environment still results in failure.\n- Adjusting the execution time period still leads to failure.\n- Manually executing the command is very fast.\n- In the problematic scenario, system resources are normal, not exceptionally high.\n- Zombie processes appeared because, after 30 seconds, when we tried to kill the process, we found that the sudo process had already been started and the user did not have the permissions to handle it.\n- After restarting the service, the problem disappeared.\n- The first time the service was deployed, the file descriptor limit was 65536, and after restarting, it was 4096.\n\nProgress:\n\nWe are almost certain that the child process is blocked, not deadlocked; it's just slow, not perpetually stuck. The specific location is from after the fork in the child process to before the execve is finally executed. It’s unclear exactly where it is getting blocked. We have ruled out all system calls that could potentially block.\n\nPoints of confusion:\n\nUsing fork in a multithreading environment cannot explain why when calling build twice at a ten-second interval, the output appears in the sudo logs within the same second. Moreover, both the call time and the time recorded in the sudo logs exceed 10 seconds, sometimes even reaching 20 or 30 seconds.\n\nSuspicions:\n\nThe process is blocked after fork, waiting for some resources, and these resources are eventually acquired. What's puzzling is what unusual scenario triggers this situation.\n\n### CPython versions tested on:\n\n3.9\n\n### Operating systems tested on:\n\nLinux","author":{"url":"https://github.com/113xiaoji","@type":"Person","name":"113xiaoji"},"datePublished":"2023-11-02T08:19:41Z","url":"https://github.com/python/cpython/issues/111635"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:b261c5ed-f405-e2eb-05d5-29bbb02d992e |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 88EE:14700E:421AEF2:53D8845:6B0222E8 |
| html-safe-nonce | faf125270912bebd2a12e67fe3cd72914bd1b91583f0c252544683fdfdbab28d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4OEVFOjE0NzAwRTo0MjFBRUYyOjUzRDg4NDU6NkIwMjIyRTgiLCJ2aXNpdG9yX2lkIjoiNTgxODQxNTc2ODQ2NzAyMjU2OCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 882da1f95fe5bb35d69e3f10107673559df3f0b78d8b9e08deb41e54532df8f1 |
| hovercard-subject-tag | issue:1973734161 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/111635/issue_layout |
| twitter:image | https://opengraph.githubassets.com/14aa3b898dde9af8af882f46a031ef9aa46cd5cd32b5fb228725c99e84769236/python/cpython/issues/111635 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/14aa3b898dde9af8af882f46a031ef9aa46cd5cd32b5fb228725c99e84769236/python/cpython/issues/111635 |
| og:image:alt | Bug report Bug description: In multi-threaded applications, using fork results in blocking rather than hanging up. We have a multi-threaded application, and in order to interact with scripts, we us... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | 113xiaoji |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width