Title: Free-threading: stop-the-world from a callback under TYPE_LOCK can deadlock · Issue #158195 · python/cpython · GitHub
Open Graph Title: Free-threading: stop-the-world from a callback under TYPE_LOCK can deadlock · Issue #158195 · python/cpython
X Title: Free-threading: stop-the-world from a callback under TYPE_LOCK can deadlock · Issue #158195 · python/cpython
Description: Bug description: On free-threaded builds, the interpreter can deadlock when Python code running under an unpinned TYPE_LOCK stops the world, directly or through an automatic GC, while another thread holds TYPE_LOCK pinned and waits to st...
Open Graph Description: Bug description: On free-threaded builds, the interpreter can deadlock when Python code running under an unpinned TYPE_LOCK stops the world, directly or through an automatic GC, while another threa...
X Description: Bug description: On free-threaded builds, the interpreter can deadlock when Python code running under an unpinned TYPE_LOCK stops the world, directly or through an automatic GC, while another threa...
Opengraph URL: https://github.com/python/cpython/issues/158195
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Free-threading: stop-the-world from a callback under TYPE_LOCK can deadlock","articleBody":"### Bug description:\n\nOn free-threaded builds, the interpreter can deadlock when Python code running under an unpinned `TYPE_LOCK` stops the world, directly or through an automatic GC, while another thread holds `TYPE_LOCK` pinned and waits to stop the world. The callbacks are a metaclass `mro()` (from `type_ready()`) and a `str`-subclass class-dict key's `__eq__` (from `type_setattro()`).\n\nObserved on fresh free-threaded builds of main a000129 (which includes PR #152826), main 8c056c33 and 3.15 13b1659d (macOS arm64, Clang, 2026-09-25), and earlier on main 6757482, 3.15 2dafe2a and 3.14 1d8c9f8. The default build is not affected.\n\nThis is the same lock-order family as gh-158194 (`PyType_Freeze()`), gh-156910 and gh-155400, but a distinct bug. Those were fixed by pinning at C call sites, which cannot cover arbitrary Python code or the GC. The broader approach below might also cover gh-158194; gh-158194's pin does not fix this.\n\n#### Reproducer (main, 3.15 and 3.14.4+)\n\nTrimmed from the attached `repro.py` (`mro bug`):\n\n```python\nimport faulthandler, sys, threading, time\nprint(\"sys._is_gil_enabled() =\", sys._is_gil_enabled())\nif sys._is_gil_enabled() is not False:\n sys.exit(\"needs a free-threaded build with the GIL disabled\")\nfaulthandler.dump_traceback_later(20, exit=True) # dump stacks and exit if hung\n\ndef f(): pass\n\nclass Meta(type):\n def mro(cls): # called by type_ready() with TYPE_LOCK held, not pinned\n f.__qualname__ = \"f\" # stops the world (func_set_qualname)\n return type.mro(cls)\n\nstop, errors = threading.Event(), []\n\ndef guarded(fn):\n def run():\n try: fn()\n except BaseException as e: errors.append(e); raise\n return run\n\ndef setter():\n while not stop.is_set():\n Meta(\"K\", (), {})\n\ndef pinner(): # type_set_bases(): pinned TYPE_LOCK, then stop-the-world\n class A: pass\n class B: pass\n class X(A): pass\n while not stop.is_set():\n X.__bases__ = (B,); X.__bases__ = (A,)\n\ndef contender(): # plain stop-the-world; keeps stw->mutex contended\n def g(): pass\n while not stop.is_set():\n g.__qualname__ = \"g\"\n\nts = [threading.Thread(target=guarded(fn)) for fn in (setter, pinner, contender) for _ in range(2)]\nfor t in ts: t.start()\ntime.sleep(5); stop.set()\nfor t in ts: t.join()\nif errors: sys.exit(f\"FAIL: {errors[0]!r}\")\nprint(\"done\")\n```\n\n- **Actual:** the process hangs. faulthandler dumps the stacks after 20 s and exits with status 1.\n- **Expected:** `done` after about 5 s.\n\n`repro_realistic.py` (ABC subclasses + `functools.wraps` + GC) and `repro_dunder_pinner.py` hang on main and 3.15 only: 3.14 (from 3.14.4, which backported `type_lock_prevent_release()`) pins only the `__bases__`/MRO path.\n\n#### Native stacks of a hang\n\nStock main 8c056c33 (free-threaded debug, no patch), `repro.py mro bug`, captured with macOS `sample` (trimmed):\n\n```\nsetter type_ready_publish typeobject.c:9513 -> mro_internal typeobject.c:3641 -> [Python mro()]\n -> func_set_qualname funcobject.c:755 -> stop_the_world pystate.c:2475 -> _PyMutex_LockTimed\n -> _PyThreadState_Attach pystate.c:2305 -> _PyCriticalSection_Resume critical_section.c:152\n -> _PyMutex_LockTimed\npinner type_set_bases typeobject.c:2002 -> stop_the_world pystate.c:2475 -> _PyMutex_LockTimed\n```\n\nThese stacks are consistent with the lock-cycle diagnosis below; lock ownership is inferred from the code paths, not read from the samples.\n\n#### Mechanism\n\n1. T1's callback under unpinned `TYPE_LOCK` stops the world and parks on a busy `stw->mutex`. Parking detaches T1, which releases `TYPE_LOCK`.\n2. T2 takes `TYPE_LOCK` pinned (e.g. `__bases__` assignment) and parks on `stw->mutex`.\n3. `stw->mutex` is handed to T1, whose re-attach (`_PyCriticalSection_Resume()`) waits for `TYPE_LOCK`: ABBA.\n\n#### A proposed approach, for specialist review\n\n`fix-{main,3.15,3.14}.diff` change `Python/pystate.c` so that `stop_the_world()` never waits for a critical-section mutex while owning the stop-the-world locks: if the critical section cannot be resumed without blocking, it drops `stw->mutex` and the rwmutex, waits, and retries. Regression-tested candidate fix. Broader attachment, finalization, progress, and interaction concerns remain for specialist review:\n\n- Re-attaching with `tstate->critical_section` temporarily set to 0 around `_PyThreadState_Attach()`: every attachment outcome, including shutdown/finalization and free-threaded builds running with the GIL enabled.\n- Nested and two-mutex critical sections; pinned sections. Only the topmost critical section is restored, which matches what `_PyCriticalSection_Resume()` already does.\n- Global vs per-interpreter stop-the-world; progress under contention.\n- PR #152826 (gh-151518), merged on 2026-09-25, changes attach and the stop-the-world state machine. The bug still reproduces on main a000129, which includes it: `repro.py mro bug` hung 5/5 there with the same stack shape (`stop_the_world` now at `pystate.c:2523`). `fix-main.diff` applies cleanly to a000129 and completed 5/5, but the two changes still need review together; applying cleanly does not establish that they are compatible.\n- Maintainability: the try-resume could move into the critical-section implementation instead of duplicating its private representation in `pystate.c`.\n\nPerformance impact has not been measured. No failure was observed on the patched build tested (main 8c056c33 debug, exactly `fix-main.diff`): each configuration that hung on stock main debug ran 0/5 or 0/3 hung, and both regression tests passed 3/3.\n\n\u003cdetails>\n\u003csummary>Validation on fresh builds and environment\u003c/summary>\n\nFresh out-of-tree free-threaded builds of main 8c056c33 (stock, and with exactly the attached `fix-main.diff`, sha256 `844eee86…`) and 3.15 13b1659d (stock), run one at a time on 2026-09-25. A run counts as hung only when the process stopped using CPU and a native `sample` showed stacks consistent with the lock cycle above.\n\n| main debug | stock | patched |\n|---|---|---|\n| `repro.py mro bug` / `strkey bug` | 5/5 / 5/5 hung | 0/5 / 0/5 hung |\n| `repro.py mro gc`, `repro_realistic.py gc`, `repro_dunder_pinner.py mro bug` | 3/3 hung each | 0/3 hung each |\n| `ctl` modes | 0/4 | 0/4 |\n| regression tests | 3/3 hung (standalone) | 3/3 pass (in-tree) |\n\n`mro bug` also hung on stock main release (5/5) and stock 3.15 debug (3/3). On main a000129 (after PR #152826, debug), `repro.py mro bug` hung 5/5 stock and completed 5/5 with `fix-main.diff`; a native `sample` of one stock hang had the same shape as above. `test_free_threading`, `test_types` and `test_descr` pass when patched. Not built: patched main release, patched 3.15 and 3.14 (`fix-3.15.diff`, `fix-3.14.diff`).\n\n`python -VV` of the fresh stock builds:\n\n```\nPython 3.16.0a0 free-threading build (heads/val/main-base-dirty:8c056c3, Sep 25 2026, 08:58:37) [Clang 21.0.0 (clang-2100.3.34.2)]\nPython 3.15.0rc2+dev free-threading build (heads/val/315-base-dirty:13b1659, Sep 25 2026, 09:07:27) [Clang 21.0.0 (clang-2100.3.34.2)]\n```\n\nEarlier stock builds (used for the older results below):\n\n```\nPython 3.16.0a0 free-threading build (heads/main-dirty:6757482, Sep 22 2026, 20:29:11) [Clang 21.0.0 (clang-2100.3.34.2)]\nPython 3.15.0rc2+dev free-threading build (heads/3.15-dirty:2dafe2a, Sep 22 2026, 21:35:07) [Clang 21.0.0 (clang-2100.3.34.2)]\nPython 3.14.7+ free-threading build (heads/3.14-dirty:1d8c9f8, Sep 22 2026, 20:25:42) [Clang 21.0.0 (clang-2100.3.34.2)]\n```\n\nThe `-dirty` suffix comes from CPython's Makefile running `git describe --dirty` from the out-of-tree build directory, where git sees every tracked file as missing. All of these builds were made out-of-tree, and their source trees show no tracked changes; the fresh trees were verified clean after each build.\n\n\u003c/details>\n\n\u003cdetails>\n\u003csummary>Older results, code locations, patch details and open questions\u003c/summary>\n\n**Attached files.** Each reproducer prints `sys._is_gil_enabled()` and exits with status 2 unless it is `False`. A worker exception exits with status 3.\n\n- `repro.py [mro|strkey] [bug|gc|ctl] [secs]`: `bug` stops the world directly inside the callback, `gc` only allocates cycles (the automatic GC stops the world), and `ctl` is the control (no stop-the-world, GC disabled).\n- `repro_dunder_pinner.py`: the same, with `X.__repr__` assignment as the pinned partner (main and 3.15).\n- `repro_realistic.py [gc|bug]`: ABC subclass creation plus `functools.wraps` plus GC (main and 3.15).\n- `test_type_stw_deadlock.py`: a standalone copy of the two regression tests in the diffs.\n\n**Older stock-build results.** These are stock builds (no patch) of main @ 6757482, 3.15 @ 2dafe2a and 3.14 @ 1d8c9f8, run on 2026-09-25 with the reproducers as they were before the GIL check and the exception propagation were added (the stress logic is the same). CPU-time inactivity was used only to screen for a hang. A native `sample` was taken of the first hang in each cell; those samples had the same shape as the stacks above (with `stop_the_world` at `pystate.c:2454` on 6757482).\n\n| Build | `repro.py mro bug` | `repro.py strkey bug` | `repro_realistic.py gc` |\n|---|---|---|---|\n| main FT debug | 2/2 hung | 2/2 hung | 1/1 hung |\n| main FT release | 2/2 hung | 2/2 hung | 2/2 hung |\n| 3.15 FT debug | 2/2 hung | 2/2 hung | 2/2 hung |\n| 3.14 FT debug | 2/2 hung | 2/2 hung | 0/2 (expected: not pinned on 3.14) |\n\nControl: `repro.py mro ctl` completed on main release. The standalone regression tests hung on stock main release. In the earlier investigation (2026-09-23, same stock commits), the `gc` modes of `repro.py` hung on all three branches, and `repro_dunder_pinner.py` hung on main and 3.15.\n\n**Earlier investigation (2026-09-23; stock main 6757482 / 3.15 2dafe2a / 3.14 1d8c9f8; earlier revision of the patch).** Most of these patched builds were made before the attached diffs were finalized, so the exact source they were built from may differ.\n\n- All reproducer configurations that hang on stock builds completed on patched main debug and release, 3.15 debug and 3.14 debug.\n- The two new tests passed in 2.4–5.6 s.\n- `test_free_threading`, `test_threading`, `test_gc`, `test_descr`, `test_capi` and related suites passed on the patched debug builds.\n- After the diffs were finalized, patched main release was re-run: 2/2 `repro.py mro bug` and 3/3 regression-test runs completed.\n\n**Code locations.**\n\n| Step | main @ 6757482 | 3.15 @ 2dafe2a | 3.14 @ 1d8c9f8 |\n|---|---|---|---|\n| `mro_invoke()` calls Python `mro()` under unpinned `TYPE_LOCK` | `typeobject.c:3578` | `:3643` | `:3502` |\n| `type_setattro()` calls a str-subclass `__eq__` under `TYPE_LOCK` | `:6715` | `:6872` | `:6317` |\n| `stop_the_world()` blocks on `stw->mutex` | `pystate.c:2454` | `:2423` | `:2387` |\n| Attach resumes the critical section | `pystate.c:2305` → `critical_section.c:152` | `:2274` | `:2242` |\n| Pinned `TYPE_LOCK` then stop-the-world | `typeobject.c` 672, 1712, 1908, 1974, 3903, 6401 | 686, 1782, 1978, 2044, 3968, 6558 | `set_tp_mro` 647 only |\n\nThe unpinned sites are unchanged at the current tips (checked 2026-09-25). `stop_the_world()` was reworked by PR #152826, but the hang remains on main after it. On 3.14, `v3.14.3` has no `type_lock_prevent_release()` and `v3.14.4` does (the backport of gh-145685). This is the same hand-off-after-detach hazard described in gh-144513.\n\n**How the patch works.** A new `stw_lock()` replaces the direct `PyMutex_Lock(&stw->mutex)`. If the thread has a resumable critical section and the fast path fails, it:\n\n1. detaches, which suspends its critical sections;\n2. takes `stw->mutex` without detaching;\n3. re-attaches without resuming;\n4. tries to resume the critical section with `PyMutex_LockFast`;\n5. on failure, releases `stw->mutex` and the rwmutex, blocks in `_PyCriticalSection_Resume()` while owning nothing, and retries.\n\nA mutex pinned by `type_lock_prevent_release()` is not stored in the critical section, so pinned holders take the unchanged path. The rwmutex is released too, because holding it while waiting for `TYPE_LOCK` could still deadlock against a pending global stop-the-world such as `os.fork()`.\n\n**Alternatives considered:**\n\n- Call `mro()` outside `TYPE_LOCK`. This is incomplete: the `str`-key path and GC inside such code remain.\n- Never run Python code under `TYPE_LOCK`. This would be a large restructuring.\n- Disable hand-off for `stw->mutex`. This is not enough, because a consumed wake-up can still hang.\n- Pin at the callback sites. This is not possible across arbitrary Python code.\n\nAs noted above, the try-resume helpers could instead live in `critical_section.c`, for example as `_PyCriticalSection_TryResume()`, so that `pystate.c` does not depend on the critical section's private representation.\n\n**Open questions and gaps.**\n\n- Starvation on the retry path is not ruled out.\n- Linux, Windows, subinterpreters and a real `os.fork()` racing this path were not tested.\n- How often real programs combine a Python-level `mro()` or a `str`-subclass class-dict key with a concurrent pinned type update is unknown.\n- 3.15.0 final is expected on 2026-10-01 (PEP 790). gh-155400, from the same family, was marked release blocker. I leave any priority decision to the release manager.\n\n\u003c/details>\n\n#### How it was found\n\nAn audit of `stop_the_world()` callers for locks that are held across the wait.\n\n### CPython versions tested on:\n\n3.14, 3.15, CPython main branch\n\n### Operating systems tested on:\n\nmacOS\n\n#### Attachments\n\n[repro.py.txt](https://github.com/user-attachments/files/32672801/repro.py.txt)\n[repro_dunder_pinner.py.txt](https://github.com/user-attachments/files/32672803/repro_dunder_pinner.py.txt)\n[repro_realistic.py.txt](https://github.com/user-attachments/files/32672799/repro_realistic.py.txt)\n[test_type_stw_deadlock.py.txt](https://github.com/user-attachments/files/32672800/test_type_stw_deadlock.py.txt)\n[fix-main.diff.txt](https://github.com/user-attachments/files/32672798/fix-main.diff.txt)\n[fix-3.15.diff.txt](https://github.com/user-attachments/files/32672802/fix-3.15.diff.txt)\n[fix-3.14.diff.txt](https://github.com/user-attachments/files/32672797/fix-3.14.diff.txt)\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-158678\n* gh-159972\n* gh-160987\n* gh-161586\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/scasella","@type":"Person","name":"scasella"},"datePublished":"2026-09-25T23:58:31Z","url":"https://github.com/python/cpython/issues/158195"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:dd8026f5-1ac7-5c9e-73f7-7b501a2a6750 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | AD06:29AEE8:33E8698:4521286:6B01F99F |
| html-safe-nonce | 72bafcab83bcb145fda259d599eb9b8797aec2d59b7c0dd7ef8d5385507ad5dd |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBRDA2OjI5QUVFODozM0U4Njk4OjQ1MjEyODY6NkIwMUY5OUYiLCJ2aXNpdG9yX2lkIjoiMzE4MTk2OTU0MTQyOTMyODI4NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | a8be8ba60cc2d315617c36a0c3d440e8ce17ce546263bb41fc11584fd7a3d6fc |
| hovercard-subject-tag | issue:5589809946 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/158195/issue_layout |
| twitter:image | https://opengraph.githubassets.com/94ecb5a92525ece6a12d96e66ddc95ed4648b27ce1bbc1755cbba186f41976bf/python/cpython/issues/158195 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/94ecb5a92525ece6a12d96e66ddc95ed4648b27ce1bbc1755cbba186f41976bf/python/cpython/issues/158195 |
| og:image:alt | Bug description: On free-threaded builds, the interpreter can deadlock when Python code running under an unpinned TYPE_LOCK stops the world, directly or through an automatic GC, while another threa... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | scasella |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width