Title: gh-140746: Fix Thread.start() hanging when the thread dies during bootstrap by Jihaysse · Pull Request #153776 · python/cpython · GitHub
Open Graph Title: gh-140746: Fix Thread.start() hanging when the thread dies during bootstrap by Jihaysse · Pull Request #153776 · python/cpython
X Title: gh-140746: Fix Thread.start() hanging when the thread dies during bootstrap by Jihaysse · Pull Request #153776 · python/cpython
Description: Problem Thread.start() waits for the new thread with a bare self._started.wait(). If the new thread dies before _bootstrap_inner() reaches self._started.set(); typically a MemoryError raised while the C-level thread_run() calls _bootstrap under memory pressure (the interpreter prints Exception ignored in thread started by ...), or a SystemExit which thread_run() clears silently → the event is never set and the parent blocks forever. The dead thread also lingers in threading._limbo, so it stays visible in threading.enumerate() while is_alive() is False. Any threaded server whose dispatcher thread spawns per-request threads (e.g. socketserver.ThreadingMixIn) freezes permanently the first time this fires. This was detected in production on Python 3.12: workers stopped serving after a burst of MemoryErrors, with kill -3 stack dumps showing the dispatcher thread blocked in Thread.start() → self._started.wait(), the request thread perpetually in its initial state, and no corresponding OS thread. Full investigation in gh-140746 (also reported by Patroni). Reproducer Deterministic: simulates the MemoryError that thread_run() swallows while calling _bootstrap (the same C code path, PyObject_Call() returning NULL): import threading class BrokenBootstrapThread(threading.Thread): # Simulates a MemoryError raised before _started.set(): thread_run() # catches it ("Exception ignored in thread started by ...") and the # thread exits without ever signalling startup. def _bootstrap(self): raise MemoryError t = BrokenBootstrapThread(target=lambda: None) t.start() # unpatched: hangs here forever print("started fine:", t.is_alive()) Unpatched (reproduced on 3.12.11 and main): prints the "Exception ignored" traceback, then start() blocks forever. With this PR: start() raises RuntimeError: thread failed to start and the thread is removed from _limbo. The organic reproducer from the issue (progressively lowering RLIMIT_DATA until a real MemoryError lands in the thread's bootstrap) also hangs unpatched and fails cleanly with this PR after ~1300 progressively-starved thread starts. Fix Add a C-level thread_started PyEvent to ThreadHandle, mirroring the existing thread_is_exiting: _bootstrap_inner() notifies it (via handle._set_started()) right after self._started.set(), so a waiter woken by the child always observes _started as set; thread_run() notifies it unconditionally on its exit path — covering a failed PyObject_Call of _bootstrap, the silent SystemExit case, and the _PyThreadState_MustExit() early exit during finalization. _PyEvent_Notify() is allocation-free C and a no-op on an already-set event, so this is safe on every exit path. Thread.start() waits on the C event (_wait_for_started(): GIL released, interruptible via Py_MakePendingCalls()) and then checks _started: set means the thread started normally (identical behavior to today); unset means the thread died before starting and never will run, so start() removes it from _limbo and raises RuntimeError("thread failed to start"). No timeout, no polling, no new handle states. Validation Environment: Ubuntu 24.04 x86-64 (Linux 6.17), gcc 13.3.0; main built both --with-pydebug and --disable-gil; stock 3.12.11 as the unpatched baseline. Both reproducers above: hang unpatched, clean RuntimeError + _limbo cleanup with this PR (both builds). End-to-end: a ThreadingTCPServer whose request thread dies in bootstrap freezes permanently unpatched; with this PR, handle_error() logs the RuntimeError and the server keeps serving. Test suites (both builds): test_threading, test_thread, test_threadedtempfile, test_thread_local_bytecode, test_socketserver -u all, test_concurrent_futures.test_thread_pool, test_queue — all pass. Three regression tests added: MemoryError in _bootstrap, MemoryError in _bootstrap_inner, and SystemExit in _bootstrap.
Open Graph Description: Problem Thread.start() waits for the new thread with a bare self._started.wait(). If the new thread dies before _bootstrap_inner() reaches self._started.set(); typically a MemoryError raised while ...
X Description: Problem Thread.start() waits for the new thread with a bare self._started.wait(). If the new thread dies before _bootstrap_inner() reaches self._started.set(); typically a MemoryError raised while ...
Opengraph URL: https://github.com/python/cpython/pull/153776
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:d0412870-5bfe-8d18-d3f9-06397f22cb89 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | BE9C:D4E59:119C3DB:161F425:6B023CB9 |
| html-safe-nonce | dae019450dce0622367c06d61cb8d8ad1d8c505acbbd0f62db66d29ac9d388d8 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRTlDOkQ0RTU5OjExOUMzREI6MTYxRjQyNTo2QjAyM0NCOSIsInZpc2l0b3JfaWQiOiIxNTE0NjI5MDM5NTcwNDQ0MDkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | f29fe3b6308e9c4572601fadd01ae382e5688be510eb8fe6499004a9a55c4075 |
| hovercard-subject-tag | pull_request:4062456916 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/153776/files |
| twitter:image | https://avatars.githubusercontent.com/u/64892838?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/64892838?s=400&v=4 |
| og:image:alt | Problem Thread.start() waits for the new thread with a bare self._started.wait(). If the new thread dies before _bootstrap_inner() reaches self._started.set(); typically a MemoryError raised while ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width