Title: multiprocessing.get_logger() logger deadlock on first call by subprocess to logger.info("...") due to internal logger.debug(...) call by multiprocessing.Queue._start_thread · Issue #91555 · python/cpython · GitHub
Open Graph Title: multiprocessing.get_logger() logger deadlock on first call by subprocess to logger.info("...") due to internal logger.debug(...) call by multiprocessing.Queue._start_thread · Issue #91555 · python/cpython
X Title: multiprocessing.get_logger() logger deadlock on first call by subprocess to logger.info("...") due to internal logger.debug(...) call by multiprocessing.Queue._start_thread · Issue #91555 · python/cpython
Description: Python: v3.10.2 Platform: Windows 10 Description: A deadlock occurs when a subprocess uses a logging.Logger returned multiprocessing.get_logger() where the initial logging level at the time of the first call to logger.info is level DEBUG...
Open Graph Description: Python: v3.10.2 Platform: Windows 10 Description: A deadlock occurs when a subprocess uses a logging.Logger returned multiprocessing.get_logger() where the initial logging level at the time of the ...
X Description: Python: v3.10.2 Platform: Windows 10 Description: A deadlock occurs when a subprocess uses a logging.Logger returned multiprocessing.get_logger() where the initial logging level at the time of the ...
Opengraph URL: https://github.com/python/cpython/issues/91555
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"multiprocessing.get_logger() logger deadlock on first call by subprocess to logger.info(\"...\") due to internal logger.debug(...) call by multiprocessing.Queue._start_thread","articleBody":"\u003cb>Python:\u003c/b> v3.10.2\r\n\u003cb>Platform:\u003c/b> Windows 10\r\n\r\n\u003cb>Description:\u003c/b> A deadlock occurs when a subprocess uses a logging.Logger returned multiprocessing.get_logger() where the initial logging level at the time of the first call to logger.info is level DEBUG or lower, which causes, during that same initial call to logger.info, an internal call to logger.debug(...) (at top of multiprocessing.Queue._start_thread), all of which causes the same thread to attempt to re-acquire an already held lock. \r\n\r\n\u003cb>Workaround:\u003c/b> Set logging level to INFO or higher (something above DEBUG) to prevent the logging.debug() statement at the top of Queue._start_thread from attempting to init/lock while init is already in progress.\r\n\r\nThe following example exhibits the issue when SHOW_THE_DEADLOCK==True. Set SHOW_THE_DEADLOCK==False to observe the workaround.\r\n\r\n\u003cpre>\r\nfrom concurrent.futures import ProcessPoolExecutor\r\nimport logging\r\nimport logging.handlers\r\nfrom multiprocessing import freeze_support\r\nimport multiprocessing\r\nimport sys\r\n\r\n\u003cb>SHOW_THE_DEADLOCK = True\u003c/b> # True to show the bug, False to show workaround.\r\n\r\ng_queue: multiprocessing.Queue = None\r\ndef global_init(logging_queue):\r\n global g_queue\r\n g_queue = logging_queue\r\n\r\ndef subprocess_logging_test():\r\n queue_handler = logging.handlers.QueueHandler(g_queue)\r\n l2 = multiprocessing.get_logger()\r\n l2.addHandler(queue_handler)\r\n\u003cb> if not SHOW_THE_DEADLOCK:\r\n l2.setLevel(\"INFO\") # default level is UNSET, if level is \u003c= DEBUG deadlock will occur, this prevents that.\r\n l2.info(\"write to log once at info and higher to perform thread init etc.\")\u003c/b>\r\n l2.setLevel(\"DEBUG\")\r\n l2.info(\"If initial level is DEBUG, deadlock here, else OK.\")\r\n l2.warning(\"If initial level is DEBUG, never reach this point, else OK.\")\r\n l2.debug(\"If initial level is DEBUG, never reach this point, else OK.\")\r\n l2.info(\"If initial level is DEBUG, never reach this point, else OK.\")\r\n\r\ndef main():\r\n global g_queue\r\n g_queue = multiprocessing.Queue(maxsize=99)\r\n handler = logging.StreamHandler(stream=sys.stdout)\r\n listener = logging.handlers.QueueListener(g_queue, handler)\r\n listener.start()\r\n with ProcessPoolExecutor(\r\n initializer=global_init,\r\n initargs=(g_queue,)\r\n ) as pexec:\r\n f = pexec.submit(subprocess_logging_test)\r\n f.result()\r\n listener.stop()\r\n\r\nif __name__ == '__main__':\r\n freeze_support()\r\n main()\r\n\u003c/pre>\r\n\r\nThe following is an annotated stack from the above example when SHOW_THE_DEADLOCK==True.\r\n\r\n\u003cpre>\r\nput (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\queues.py:92)\r\n \u003cb>with self._notempty: # \u003c----- dead lock\u003c/b>\r\n if self._thread is None:\r\n self._start_thread()\r\n self._buffer.append(obj)\r\n self._notempty.notify()\r\nput_nowait (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\queues.py:138)\r\nenqueue (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\handlers.py:1423)\r\nemit (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\handlers.py:1461)\r\nhandle (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:968)\r\ncallHandlers (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1696)\r\nhandle (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1634)\r\n_log (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1624)\r\nlog (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1547)\r\ndebug (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\util.py:50)\r\n_start_thread (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\queues.py:165)\r\n def _start_thread(self):\r\n \u003cb>debug('Queue._start_thread()') # \u003c----- at level DEBUG or lower, triggers logging call leading to deadlock.\u003c/b>\r\nput (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\queues.py:94)\r\nput_nowait (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\queues.py:138)\r\nenqueue (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\handlers.py:1423)\r\nemit (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\handlers.py:1461)\r\nhandle (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:968)\r\ncallHandlers (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1696)\r\nhandle (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1634)\r\n_log (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1624)\r\ninfo (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\logging\\__init__.py:1477)\r\nsubprocess_logging_test (c:\\PythonTesting\\logging_testing.py:25)\r\n_process_worker (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\concurrent\\futures\\process.py:243)\r\nrun (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\process.py:108)\r\n_bootstrap (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\process.py:315)\r\n_main (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\spawn.py:129)\r\nspawn_main (c:\\Users\\HappyPythonDev\\AppData\\Local\\Programs\\Python\\Python310\\Lib\\multiprocessing\\spawn.py:116)\r\n\u003cmodule> (\u003cstring>:1)\r\n\u003c/pre>\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-131812\n* gh-133898\n* gh-133899\n* gh-135858\n* gh-135910\n* gh-135911\n* gh-135954\n* gh-135988\n* gh-135989\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/AshleyT3","@type":"Person","name":"AshleyT3"},"datePublished":"2022-04-15T02:44:27Z","url":"https://github.com/python/cpython/issues/91555"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:44a2a7e4-358c-e66b-af2f-1dd8a22d3655 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 917A:380435:DD40FF:116DF22:6B02331E |
| html-safe-nonce | 478ed3355b8dfc54117c5a24c03320d3fd4b9e5a910a3be5d104a4e7555634a2 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MTdBOjM4MDQzNTpERDQwRkY6MTE2REYyMjo2QjAyMzMxRSIsInZpc2l0b3JfaWQiOiI3NjM2NDI1NjMzMzA3OTY0MTkwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 62ccc6e7fa552df885e5cd02ba2c8cac6a23b4c11a838f7781add6bb91de0e17 |
| hovercard-subject-tag | issue:1205194981 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/91555/issue_layout |
| twitter:image | https://opengraph.githubassets.com/65cec61e8d37628ddee6e054a9ee00b8a86eaff534d905b1f594dd31933de949/python/cpython/issues/91555 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/65cec61e8d37628ddee6e054a9ee00b8a86eaff534d905b1f594dd31933de949/python/cpython/issues/91555 |
| og:image:alt | Python: v3.10.2 Platform: Windows 10 Description: A deadlock occurs when a subprocess uses a logging.Logger returned multiprocessing.get_logger() where the initial logging level at the time of the ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | AshleyT3 |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width